CN114138095A - Power consumption processing method and device for internet data center IDC and readable medium - Google Patents

Power consumption processing method and device for internet data center IDC and readable medium Download PDF

Info

Publication number
CN114138095A
CN114138095A CN202210109703.2A CN202210109703A CN114138095A CN 114138095 A CN114138095 A CN 114138095A CN 202210109703 A CN202210109703 A CN 202210109703A CN 114138095 A CN114138095 A CN 114138095A
Authority
CN
China
Prior art keywords
periodically executed
information
time
power consumption
processing time
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210109703.2A
Other languages
Chinese (zh)
Other versions
CN114138095B (en
Inventor
程明
毛宏举
李超
冯亦挥
曹雨洁
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba China Co Ltd
Original Assignee
Alibaba China Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba China Co Ltd filed Critical Alibaba China Co Ltd
Priority to CN202210109703.2A priority Critical patent/CN114138095B/en
Publication of CN114138095A publication Critical patent/CN114138095A/en
Application granted granted Critical
Publication of CN114138095B publication Critical patent/CN114138095B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/26Power supply means, e.g. regulation thereof
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/26Power supply means, e.g. regulation thereof
    • G06F1/28Supervision thereof, e.g. detecting power-supply failure by out of limits supervision
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/3058Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/34Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466Performance evaluation by tracing or monitoring
    • G06F11/3476Data logging
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Computer Hardware Design (AREA)
  • Computing Systems (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The embodiment of the application provides a power consumption processing method, equipment and a readable medium for IDC (Internet data center) so as to improve the accuracy of power consumption quantification of IDC. The method comprises the following steps: determining a periodically executed process; determining resource information of the periodically executed process according to the operation parameters of the periodically executed process; and estimating the power consumption information of the periodically executed process according to the resource information of the periodically executed process. The power consumption of each process can be accurately estimated, and a foundation is provided for the IDC to provide operation flexibility for the power system.

Description

Power consumption processing method and device for internet data center IDC and readable medium
Technical Field
The present application relates to the field of computer technologies, and in particular, to a power consumption processing method for an internet data center IDC, a terminal device, and a machine-readable medium.
Background
As a key infrastructure of the information age, the number and scale of Internet Data Centers (IDCs) have rapidly increased in the past decade, bringing about various opportunities and challenges to many fields such as society, economy, and technology. Meanwhile, the increasing energy demand of the IDC has gradually attracted attention from many parties.
The computing service provided by the IDC forms various computing tasks on the platform, and the computing tasks are application programs used for executing computing requests submitted by users. Some computing tasks may have some degree of execution flexibility in view of the different execution requirements and priorities of the tasks. Because a plurality of IDCs are equipped with renewable energy power generation resources, the flexibility transfer capability of the calculation task can consume more electric quantity in some time periods, thereby realizing the consumption of renewable energy. The data center scheduling system can reschedule the computation task processing time to make IDCs a potential power system demand response participant.
However, the existing theoretical model rarely quantifies and applies the flexibility of the IDC in combination with the actual operation condition, so the feasibility and accuracy of the model are greatly limited, and the potential of the IDC in providing the operation flexibility for the power system is influenced.
Disclosure of Invention
The embodiment of the application provides a power consumption processing method of IDC, so that the accuracy of power consumption quantification of IDC is improved.
Correspondingly, the embodiment of the application also provides electronic equipment and a machine readable medium, which are used for ensuring the realization and the application of the method.
In order to solve the above problem, an embodiment of the present application discloses a power consumption processing method for an internet data center IDC, where the method includes:
determining a periodically executed process;
determining resource information of the periodically executed process according to the operation parameters of the periodically executed process;
and estimating the power consumption information of the periodically executed process according to the resource information of the periodically executed process.
Optionally, the determining a process executed in a cycle includes: and acquiring the operation parameters of the process based on the operation log, and identifying the process executed periodically according to the operation parameters.
Optionally, the identifying a process executed periodically based on the running log includes:
inquiring log records of a process in an operation log, and acquiring operation parameters of the process, wherein the operation parameters comprise: signature information and start processing time;
screening a repeatedly executed process according to the signature information, and adding the repeatedly executed process to a first set;
and screening the periodically executed processes from the first set according to the starting processing time, and adding the periodically executed processes to a second set.
Optionally, the screening, according to the signature information, a repeatedly executed process added to the first set includes:
counting based on the signature information, and determining the number of times of the signature information;
judging whether the times of the signature information exceed a repetition threshold value;
and when the times of the signature information exceed a repetition threshold, adding the process corresponding to the signature information into a first set.
Optionally, the process executed periodically is screened from the first set according to the processing start time, and the process includes:
determining a time interval of starting processing time of two processes in the first set;
and calculating the change coefficient of each time interval, and determining the periodically executed process according to the change coefficient.
Optionally, the determining resource information of the periodically executed process according to the operation parameter of the periodically executed process includes:
determining corresponding processing time according to the starting processing time and the ending processing time;
determining average resource usage information from the resource usage information and the processing time, the resource usage information comprising: average CPU usage and/or average memory usage.
Optionally, the method further includes:
and determining the time parameter of the periodically executed process according to the dependency relationship of the periodically executed process.
Optionally, the determining the time parameter of the periodically executed process according to the dependency relationship of the periodically executed process includes:
determining a time parameter of a downstream sub-process to be a specified value when the execution of the process is independent of the process;
in the case that execution of a specified downstream process depends on the process, determining a time parameter thereof as a time difference between a start processing time of the specified downstream process and an end processing time of the process;
in the case that execution of more than one downstream process depends on the process, a target downstream process with the earliest starting processing time among the more than one downstream process is determined, and a time parameter of the target downstream process is determined as a time difference between the starting processing time of the target downstream process and the ending processing time of the process.
Optionally, the estimating, according to the resource information of the periodically executed process, power consumption information of the periodically executed process includes:
and inputting the resource information of the periodically executed process into a power consumption mapping model to obtain the estimated power consumption information of the periodically executed process.
Optionally, the method further includes: and adjusting the starting processing time of the periodically executed process according to the power consumption information and the time parameter of the periodically executed process.
Optionally, the adjusting the processing start time of the periodically executed process according to the power consumption information and the time parameter of the periodically executed process includes:
acquiring power supply information;
the power supply information and the power consumption information and the time parameter of the periodically executed process adjust a start processing time of the periodically executed process.
Optionally, the method further includes: and determining the flexibility parameter of the IDC according to the power consumption information and the time parameter of the process executed periodically.
The embodiment of the application also discloses an electronic device, which comprises: a processor; and a memory having executable code stored thereon that, when executed, causes the processor to perform a method as described in embodiments of the present application.
One or more machine-readable media having stored thereon executable code that, when executed, causes a processor to perform a method as described in embodiments of the present application are also disclosed.
Compared with the prior art, the embodiment of the application has the following advantages:
in the embodiment of the application, a periodic process can be determined, then the resource information of the periodically executed process is determined according to the operation parameters of the periodically executed process, and the power consumption information of the periodically executed process is estimated based on the resource information, so that the power consumption of each periodically executed process can be accurately estimated, and a basis is provided for the IDC to provide operation flexibility for the power system.
Drawings
Fig. 1 is a flowchart illustrating steps of an embodiment of a power consumption processing method of an IDC according to the present application;
2A, 2B, 2C are diagrams of examples of data dependencies of workflows of embodiments of the application;
FIG. 3 is a diagram illustrating an example of execution parameters of a process of a periodic batch computing task according to an embodiment of the present application;
FIG. 4 is a diagram illustrating an example of a coefficient of variation of a key performance parameter in an embodiment of the present application;
FIG. 5 is a schematic diagram of an example of periodic operation with data dependency according to an embodiment of the present application;
FIG. 6 is a schematic diagram of an example of power consumption for a weekday and weekend periodic operation in an embodiment of the application;
FIGS. 7A and 7B are diagrams illustrating examples of quantified results of power usage flexibility for a weekday and weekend periodic operation according to an embodiment of the present disclosure;
FIG. 8 is a schematic diagram of an example of IDC flexibility electrical loads for improving wind power consumption requirements according to an embodiment of the present application;
9A and 9B are schematic diagrams of flexible power load and flexible load ratio examples under different wind power plants according to an embodiment of the application;
fig. 10 is a flow chart of steps of another embodiment of a method for handling power consumption of an IDC of the present application;
fig. 11 is a schematic structural diagram of an apparatus according to an embodiment of the present application.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, the present application is described in further detail with reference to the accompanying drawings and the detailed description.
The embodiment of the application can be applied to the Internet data center IDC. Among all power consuming devices of the IDC, IT (Information Technology) devices generally generate the most power consumption by providing intensive computing services. And therefore need to be used reasonably in connection with the actual execution of the computing service and the power conditions.
The computing service provided by the IDC is processed by a computing task in the platform, the computing task is an application program executed on the computing platform, the purpose of the program is to solve computing requests submitted by users (such as data processing, data cleaning, data mining and the like), and the processing of the computing task is realized by a process. Some processes may have some degree of execution flexibility in view of the different execution requirements and priorities of the processes that compute tasks. According to the classification method widely used in various researches and productions, the process of the calculation task can be divided into two types, i.e., the process of an off-line task and the process of an on-line task. Among other things, online tasks are typically highly random and do not allow for delayed processing, such as web searching and online payment tasks. In contrast, an offline task is generally a batch computing task, and the execution process thereof consumes more computing resources, generally has longer processing time and more flexible processing time, and includes data mining and machine learning tasks.
The batch computation task is correspondingly realized by a batch process, which can be further divided into a process of an aperiodic batch computation task and a process of a periodic batch computation task. The aperiodic batch computing task refers to tasks submitted at any time, and comprises batch downloading tasks submitted by a user at any time, debugging tasks submitted by operators and the like. Generally, they do not have a fixed arrival time and also do not have an explicit deadline, and such computational tasks are less time-flexible. By contrast, a periodic computational task refers to a computational task that is processed periodically, such as periodic data gathering, state detection scanning, and so forth. Such computing tasks account for approximately 60% of batch processing computing tasks, may be considered one of the most promising sources of power consumption flexibility for IDC, and may optimally delay the processing process through a scheduling system of a data center cloud platform.
In the process of batch computing task execution, generally, a task scheduling system divides a batch computing task into several subtasks according to code logic, and each of the subtasks is responsible for completing a part of the whole computing task. After the segmentation, the processes of the subtasks without data dependency relationship can be processed in parallel, so that the overall computing efficiency is improved. In the embodiment of the application, the divided computing tasks are defined as subtasks, and the processes of the subtasks are used as main objects for evaluating the IDC power utilization flexibility. Wherein, during the running process, the calculation of the process of some subtasks depends on the result of the process of other subtasks, which results in the calculation sequence based on the data dependency relationship naturally generated during the process execution of the subtasks. In production, the sequence of processes of such a calculation task is defined as a workflow (workflow), by which execution steps and execution order are indicated.
The workflow framework specifies a start time and an acceptable expiration date (i.e., an end time) for each subtask process, which will typically begin executing as scheduled. However, if the process of the preceding subtask does not reach the start time of the process of the subsequent subtask after the process of the preceding subtask is completed, the process of the downstream subtask still needs to wait until the originally set start time to start the calculation. For periodic computing tasks, their workflow framework will automatically launch several times within its lifecycle. And the subtasks in each periodic computing task can be repeatedly executed for many times according to a plan specified by workflow, so that a 'periodic subtask' is formed. After information contained in workflow of the periodic computing task is mastered, the execution time and the latest completion time of the periodic subtask can be obtained, and based on the information, the time flexible execution potential of the periodic computing task in a production environment can be obtained.
The embodiment of the application can execute the power consumption processing of the IDC based on the periodical subtasks, and improves the processing flexibility.
Step 102, determining a periodically executed process.
Periodically executed processes may be determined, where the periodically executed processes may be determined based on a log of operations of the system, configuration information of the system, and the like. For example, a corresponding cycle identifier is added in the process of setting the periodic execution, or the process of the periodic execution is determined based on information such as parameters of the process, which may be specifically set according to requirements. In an optional embodiment, an operation parameter of a process may be obtained based on an operation log, and a process that is periodically executed may be identified according to the operation parameter, where the process is a process of a subtask of a computation task of the IDC.
The execution log is a log for recording the process execution process of the task, and generally, the execution footprints of all the processes are recorded in the execution log. The log typically contains the following three types of information: attribute information of the process itself (such as priority, signature information, etc. of each process), resource occupation information (such as CPU, memory, disk, and network I/O usage, etc.), and execution information (such as start and end processing time, running status, running server and cluster ID, etc.). The log record of each process may record the above attribute information, resource occupation information, and execution information. So that the process can be analyzed based on the log records. First, periodic processes may be identified based on log records.
The operation parameters of the process within the specified time range can be acquired from the operation log, wherein the signature information and the operation time are main parameters for judging whether the process has periodicity, and therefore the signature information, the start processing time, the end processing time and the like can be acquired as the operation parameters. Each computation task in the IDC scene has unique signature information, the signature information is shared by all subtasks, and correspondingly, the process of the subtask adopts the signature information. For example, the corresponding signature may be obtained through a hash equation (hash function) based on the source code of the computation task, and the corresponding scheduling information may compute the request based on the unique signature information and mark its computation content and computation logic. Since all subtasks of the same computation task will share the same signature, the subtasks with the same signature may be different subtasks of the same computation task or the same subtask that is periodically and repeatedly executed. An execution time for the process may be determined based on the start processing time and the end processing time, and a periodic process may be determined based on the execution time. In an optional embodiment, the obtaining the operation parameter of the process based on the operation log and identifying the process that is executed periodically according to the operation parameter includes: inquiring log records of a process in an operation log, and acquiring operation parameters of the process, wherein the operation parameters comprise: signature information and start processing time; screening a repeatedly executed process according to the signature information, and adding the repeatedly executed process to a first set; and screening the periodically executed processes from the first set according to the starting processing time, and adding the periodically executed processes to a second set.
On the basis of the above embodiment, screening repeatedly executed processes according to the signature information, and adding the processes into the first set includes: counting based on the signature information, and determining the number of times of the signature information; and judging whether the times of the signature information exceed a repetition threshold, and adding the process corresponding to the signature information into a first set when the times of the signature information exceed the repetition threshold.
In order to screen out processes of periodic subtasks from a data set of a hybrid computing task, processes that repeatedly appear in different time periods may be identified, and therefore signature information corresponding to each process may be counted, for example, the signature information is counted by a corresponding counter +1 every time the signature information appears, and the number of times that each signature information is counted may be obtained, and a corresponding repetition threshold may be set, for example, the repetition threshold is set to 1, or set to other values such as 2, 3, and the like according to requirements. When the number of times of signature information exceeds a repetition threshold, it is confirmed that the subtask corresponding to the signature information is a repeated subtask, a process corresponding to the signature information may be added to the first set, and each process corresponds to a corresponding log record.
After acquiring the repeatedly executed process, the periodically executed process may be detected based on the repeatedly executed process. Here, the periodically executed processes may be referred to as periodic processes, and the processes are periodically executed, so that a time difference of start processing times (or arrival time) thereof is substantially close to constant, and thus the periodically executed processes may be determined based on the start processing times of the processes. The process of screening periodic execution from the first set according to the starting processing time comprises: determining a time interval of starting processing time of two processes in the first set; and calculating the change coefficient of each time interval, and determining the periodically executed process according to the change coefficient. The processes corresponding to the same signature information may be sorted in the first set according to a time sequence, and then each process may be sorted according to the signature information, as shown in table 1 below. After processing, processing time interval information between processes with the same signature is calculated, and a variation coefficient CV value is calculated at intervals to measure whether the execution time interval of the processes under the signature is greatly changed or not, and if the execution time interval of the processes under the signature is greatly changed (namely the CV value is larger), the processes are not periodic processes.
TABLE 1
Figure 93481DEST_PATH_IMAGE001
In an alternative embodiment, the Coefficient of Variation (CV) may be used as a recognition criterion to measure the variance of the time to start the process. CV is an estimation parameter of relative discreteness in a data set, and one determination method is to divide a Median Absolute Deviation (MAD) by the median, as shown in formulas (1) and (2):
Figure 969033DEST_PATH_IMAGE002
(1)
Figure 732590DEST_PATH_IMAGE003
(2)
wherein the content of the first and second substances,
Figure 769816DEST_PATH_IMAGE004
the sample data is represented by a sample data,
Figure 251613DEST_PATH_IMAGE005
representing the median. The larger the CV value, the higher the degree of fluctuation around the median. In this case, the time difference of the start processing time, i.e., the inter-arrival time, is relatively constant, representing that the workload is periodic. A variation threshold may be preset to represent the degree of acceptance of the deviation. And comparing the calculated CV value with a variation threshold, and if the CV value meets the variation threshold, determining that the process is a periodically executed process. Periodically executing processes may thus be added to the second set and log records in the run log associated with each process in the second set.
And 104, determining resource information of the periodically executed process according to the operation parameters of the periodically executed process.
After deeply analyzing the characteristics of each data, some data are key parameters capable of reflecting the potential of power consumption and time flexibility, and these key parameters can be called resource information, and the resource information includes: run time parameters and time parameters, etc. The run-time parameter is a slack time, which refers to a time flexibility parameter for executing a sub-task. The running log usually records the start processing time, the end processing time and the total resource usage for each process, so that the specific usage of the subtasks needs to be analyzed.
In an optional embodiment, the determining resource information of the periodically executed process according to the operation parameter of the periodically executed process includes: determining corresponding processing time according to the starting processing time and the ending processing time; determining average resource usage information from the resource usage information and the processing time, the resource usage information comprising: average CPU usage and/or average memory usage. And determining the time parameters of the subtasks according to the dependency relationship.
Wherein, the processing time of the process can be obtained by the following formula (3). The amount of processing time may affect the time that the IDC participates in the demand response. Furthermore, the average usage of computing resources during execution is a factor in the dynamic power size of the server. In view of the currently prevalent and rapidly growing computing and storage requirements, the CPU and memory are considered as two major energy consumptions during the operation of the servers of the IDC. The calculation formulas of the average CPU and memory usage are shown in formula (4) and formula (5):
Figure 399698DEST_PATH_IMAGE006
(3)
Figure 283340DEST_PATH_IMAGE007
(4)
Figure 491468DEST_PATH_IMAGE008
(5)
wherein the time parameter for a process is related to its dependency. During execution, the computation of some processes depends on the results of other processes, which results in a sequence of computations based on data dependencies that naturally occur during the execution of the processes. The calculation sequence of such a calculation task is defined as a workflow (workflow), which indicates execution steps and execution order. In workflow, data dependencies between processes may be described as a Directed Acyclic Graph (DAG), as shown in fig. 2A, 2B, 2C. The workflow framework specifies a start processing time and an acceptable deadline for each process and then will start executing as scheduled. In this case, if the start time of the subsequent process is not reached after the previous process is completed, the downstream process still needs to wait until the start time set originally before starting the calculation. For periodic processes, their workflow framework will automatically launch several times within its lifecycle. And repeatedly executing the process of each periodic work task for many times according to a plan specified by the workflow. After information contained in workflow of the periodic process is mastered, the execution time and the latest completion time of the periodic process can be acquired, and based on the information, the time flexible execution potential of the periodic calculation task in the production environment can be acquired.
Three data dependencies between workflow sub-tasks as shown in fig. 2A, 2B, and 2C. A and E represent the start processing time and the end processing time, respectively, recorded in the execution log by each process. E-A represents the processing time T for each process. For each process, the acceptable delay time, i.e. the time flexibility, may also be used to determine the time parameter of the process. In consideration of the dependency of various data, the slack time of each process can be calculated in the following manner, wherein the slack time can be understood as a time range that can be flexibly adjusted, which can also be referred to as an execution time range.
1) In workflow1 (workflow 1), process SL2 depends on the results of its upstream process SL 1. Then the latest completion time of process SL1 is the start time of process SL2 (denoted as a 2). Therefore, the relaxation time of SL1 is calculated as A2-E1, as shown in FIG. 3.
2) In workflow2 (workflow 2), process SL1 and process SL2 are both computational tasks upstream of process SL 3. The process SL1 and the process SL2 are distributed to different virtual machines for processing during the execution process, so that the parallel calculation of the two processes can be realized. The execution deadline constraints of the process SL1 and the process SL2 depend on the start time of the process SL3 at the same time, and the influence of postponing any one of the processes on the whole calculation process is small. In this case, the acceptable latency for process SL1 is A3-E1, and the acceptable latency for SL2 is A3-E2.
3) In workflow3 (workflow 3), process SL1 is simultaneously an upstream computing task for process SL2 and process SL 3. In this case, the execution deadline of the process SL1 depends on the earliest start time in its subsequent processes. That is, the acceptable delay time for the process SL1 is calculated as min { A2, A3} -E1.
In an alternative embodiment, the dependency relationship of the periodically executed process may be obtained; determining a time parameter of a downstream sub-process to be a specified value when the execution of the process is independent of the process; in the case that execution of a specified downstream process depends on the process, determining a time parameter thereof as a time difference between a start processing time of the specified downstream process and an end processing time of the process; in the case that execution of more than one downstream process depends on the process, a target downstream process with the earliest starting processing time among the more than one downstream process is determined, and a time parameter of the target downstream process is determined as a time difference between the starting processing time of the target downstream process and the ending processing time of the process. The slack time for each process can be determined by equation (6):
Figure 460561DEST_PATH_IMAGE009
(6)
wherein the latest end time of the upstream process is limited by the start time of the downstream process starting earliest. If the process does not have a downstream process that depends on its outcome output, it may be considered to have no time to complete limitation, and therefore the flexible time parameter may be set to a specified value for the process. In some scenarios, where the execution of the downstream process described above is independent of the process, its execution is similar to an aperiodic computing task, where a process without the latest completion time constraint will be processed as soon as possible by the scheduling system, and its slack time is assigned to 0, i.e., without regard to its flexibility in time shifting.
The key execution parameter of each periodic process forms a tuple of resource information, which is composed of 6 key execution parameters and is represented as
Figure 648222DEST_PATH_IMAGE010
. From these data, it is possible to estimate the power consumed by each subtask and quantify the role of its temporal flexibility in the power system.
Therefore, the flexibility of power utilization of the data center considering the running condition of the periodic subtasks can be evaluated based on the running logs according to the calculation task scheduling principle in the workflow and the dependency relationship between the subtasks.
And 106, estimating the power consumption information of the periodically executed process according to the resource information of the periodically executed process.
The embodiment of the application can be provided with a power consumption mapping model, and the power consumption mapping model can be obtained based on power consumption information training of the server of the IDC. The memory use information and the CPU use information of the server can be obtained in advance, other resource use information such as temperature information and the like can also be obtained, and the power consumption information of the server can be used as sample data. Various machine learning and neural network models can be used to train the power consumption mapping model. The power consumption mapping model can be normalized first, and then the correlation degree calculation is carried out on the resource use information after the normalization processing, so as to determine the corresponding correlation degree information. The method comprises the steps that the correlation degree between resource data and power consumption can be determined through correlation degree analysis aiming at various resource usages such as the use condition of a memory, the use condition of a CPU (Central processing Unit), the temperature and the use condition of a network, and therefore resource information with high correlation degree with the power consumption is selected to analyze the power consumption condition. The correlation between the memory usage and the power consumption and the correlation between the CPU usage and the power consumption are determined through correlation analysis, so that the power consumption is analyzed by taking resource information such as the memory and the CPU as an example in the embodiment of the present application. The actual processing may also be determined based on other resource information, such as temperature, etc., depending on the requirements. After determining that the power consumption related resource information needs to be analyzed, normalized resource information, such as normalized memory usage, normalized CPU usage, and the like, may be input into the power consumption mapping model to obtain corresponding power consumption information. And determining a loss function based on the actual power consumption corresponding to the sample data, reversely transmitting the loss function to the power consumption mapping model, and adjusting parameters of the power consumption mapping model. The process is iterated for multiple times until the output result of the power consumption mapping model meets the evaluation condition, and a corresponding power consumption mapping model is obtained. In an alternative embodiment, the power consumption mapping model may use a 3-layer network structure, which can prevent overfitting while ensuring accuracy. And (4) carrying out data normalization by adopting a minimum-maximum normalization method. The data set was divided into a training set (80%) and a validation set (20%) by normalization. During the training process, the Mean Square Error (MSE) is set as a criterion to evaluate the performance of the training network to quantify the error between the output estimate and the actual output of the validation set. If the performance is acceptable, this network is used to estimate the dynamic power requirements of the periodic subtasks.
The resource information of the periodically executed process can be input into the power consumption mapping model, the estimated power consumption information is output through the processing of the power consumption mapping model, and the power consumption information corresponding to the periodically executed process is determined.
In an alternative embodiment, the power consumption mapping model utilizes CPU usage and memory usage as input vectors. Wherein the total power consumption of the server can be divided into a fixed part and a dynamic part. The fixed portion represents the power consumption when the server is idle, and is generally considered to be a constant value. While the dynamic portion represents the dynamic power requirements of computation, data writing and reading, which are highly correlated with CPU and memory usage. The dynamic power of the server may be determined based on the following equation (7) and used as an output vector in the network.
Figure 386371DEST_PATH_IMAGE011
(7)
The dynamic power demand of the periodic subtasks can thus be estimated based on the power consumption mapping model.
On the basis of the embodiment, after the dynamic power demand of the periodic subtasks is evaluated, the processing time of the periodic subtasks can be flexibly and dynamically adjusted according to the actual conditions of power generation, power network load and the like, so that the subtasks can be more reasonably scheduled, and the power condition is stabilized.
In an optional embodiment, the start processing time of the periodically executed process is adjusted according to the power consumption information and the time parameter of the periodically executed process. Based on the power consumption information and the time parameter of the periodically executed process, the executable flexible time of the periodically executed process can be determined, and the starting processing time of the periodically executed process can be adjusted by combining the information of the power consumption, the power condition and the like of the current IDC, so that resources are reasonably scheduled.
In some optional embodiments, the adjusting the processing start time of the periodically executed process according to the power consumption information and the time parameter of the periodically executed process includes: acquiring power supply information, the power supply information and power consumption information and time parameters of the periodically executed process, and adjusting the starting processing time of the periodically executed process. In the actual processing, power supply information such as power generation capacity information and power generation amount information corresponding to the power station can be determined in combination with the power supply situation, and the starting processing time of the periodically executed process can be adjusted in combination with the power consumption information and the time parameter of the periodically executed process. In an electric power system with unstable power generation such as wind power generation and solar power generation, the execution of a non-periodically executed process and a process of a calculation task with a high priority is preferentially ensured in the time when the power generation is insufficient and the periodically executed process can be flexibly adjusted, and the task of the periodically executed process is suspended in the time, so that the power consumption of the IDC can be reasonably scheduled by reasonably utilizing the power consumption.
Flexible operation of IDCs can play an important role in power system operation, such as intermittent resources with high penetration levels. And determining the flexibility parameter of the IDC according to the power consumption information and the time parameter of the process executed periodically. The flexibility parameter can adopt two indexes of flexible load capacity and flexible load ratio to evaluate the flexibility related to the operation of the power system. The flexible load capacity refers to the flexible load magnitude of all periodic processes in the kth time interval, and can be calculated by the following equation (8). The flexible demand rate, i.e., the ratio of the flexible load amount at the moment, can be calculated by the ratio of the flexible load to the original load, and can be calculated by the following equation (9). These flexibility parameters may be positive or negative numbers indicating that the IDC is capable of adjusting the load up or down, respectively, during this time interval. The interaction process between the IDC and the power system can be simulated with a model-based approach (e.g., optimization model) or a model-free approach (e.g., deep reinforcement learning). The above flexible indicators are quantified, for example, using a Mixed Integer Linear Programming (MILP) model.
Figure 765399DEST_PATH_IMAGE012
(8)
Figure 956209DEST_PATH_IMAGE013
(9)
After the flexibility quantitative evaluation processing is carried out, the flexibility of the power system provided by the IDC clusters can be clearly quantified, so that the data center is allowed to participate in the cooperative operation of the power system, and the promotion effects of promoting regional development, balancing energy resource supply and reducing carbon emission are achieved. And theoretical basis can be provided for the data center to participate in the power system demand side response, the auxiliary service market and the electric energy market in the next step.
In summary, the operation parameters of the processes can be obtained based on the operation log, so that the periodic processes are identified based on the operation parameters, the processes are processes of subtasks of the work task corresponding to the computing service of the IDC, then the resource information of the processes executed periodically is determined according to the operation parameters of the processes executed periodically, and the power consumption information of the processes executed periodically is estimated based on the resource information, so that the power consumption of each process can be accurately estimated, and the IDC provides a basis for providing operation flexibility for the power system.
On the basis of the above embodiment, the present embodiment applies the above process to the run log data set of batch computing tasks generated by 8272 servers of 3 batch clusters within one week. Only batch computing tasks are computed on the batch cluster. In this case, the execution log includes the execution records of both the periodic batch calculation tasks and the aperiodic batch calculation tasks. The progress of a periodic subtask (i.e., a periodic work periodic jobis) is identified from the mixed set of computing tasks. The results show that the periodic computing tasks account for approximately 55% of the batch tasks on these clusters, with 97% being on a daily basis. Therefore, the present embodiment mainly studies the case of having the operation with the cycle of day and the time flexibility with the flexibility quantization method. The periodic subtasks and the non-periodic subtasks are executed by corresponding processes, so that the related parameters of the periodic subtasks and the non-periodic subtasks are parameters of the corresponding processes.
The periodic process can be identified through the steps, and the resource information of the periodic process can be determined. As shown in fig. 3, the distribution of the number, CPU and memory usage of the service cluster in one day has a significant peak-to-valley difference, and a large part of the periodic tasks are processed in the early morning (from 0:00 to 8: 00). For example, one observable surge is at 0: 00. The number remained at a relatively low level for several hours after 8: 00. The usage distribution trends of computing resources are similar. The reason for this determination after deep analysis is that most of the periodic subtasks on these clusters are SQL query tasks, which are used to collect and count data on activities of the previous day, and usually need to be completed before 8 o' clock in order to be managed the next day. On the other hand, non-periodic batch tasks are processed as quickly as possible and they have a user-oriented feature with a higher priority in the preemption of the batch cluster's computing resources. Most non-periodic batch tasks are generated temporarily, submitted manually by application developers, such as temporary testing and temporary queries. Thus after 8 o' clock, more non-periodic batch tasks are submitted and distributed to the batch cluster, requiring processing as quickly as possible and occupying a large amount of computing resources. The number of periodic subtasks and the usage of computing resources remains low for these hours, taking into account the resources preempted by these non-periodic computing tasks. In addition, a dramatic increase in the number of tasks and resource consumption occurs at around 18:00 a day. This is because these tasks are submitted for the purpose of generating a running report for the current day. On the other hand, the submission number of the non-periodic subtasks is greatly reduced after 18:00, and occupied resources are released. As can be seen from a comparison of the execution parameters of fig. 3 for different time periods, the periodic subtasks do not necessarily strictly follow a constant execution pattern over different periods. This is because the randomness of the non-periodic subtasks with higher priority introduces uncertainty into the availability of computational resources for the periodic subtasks of lower priority. Therefore, the execution mode may fluctuate within a certain range.
Referring to FIG. 4, a variation of the process execution pattern of a periodic subtask is shown, including a distribution of CV values for 6 key execution parameters. The task tracking of the process is divided into two data sets of working days and weekends so as to compare the change conditions of key execution parameters in different dates. In FIG. 4, the shorter boxes indicate that the periodic subtask critical execution parameters change more intensively during this time period, while the longer boxes indicate that the distribution of the critical execution parameters is more dispersed.
It can be seen that the start and end treatment times have relatively large CV values on weekdays and weekends compared to other parameters. The reason is that the scheduling system may defer execution time of a lower priority periodic process in case of insufficient computing resources. Furthermore, the variations in processing time, resource usage and slack time of the periodic process appear relatively low. This means that whenever a process of a regular task starts processing, its execution logic and execution contents do not change much. Furthermore, the weekend execution pattern changes much less than weekdays, indicating that weekend process work execution follows the predetermined schedule more strictly, and process execution delays for periodic subtasks occur less because of the smaller number of non-periodic subtask processes with higher priority during the weekend, which has less impact on periodic work.
In the flexibility analysis for the IDC, time, power consumption, and flexibility of power usage can be analyzed. Wherein the flexibility of the process execution time of the periodic subtasks is analyzed. To illustrate the relationship of data dependency, execution order, and slack time, an example of a periodic subtask process is depicted in FIG. 5. In the figure, each circle represents the progress of one periodic subtask. The color refers to the average treatment time and the magnitude refers to the relaxation time. Arrows indicate data dependencies and execution order between tasks. Wherein the grey arrows indicate that the temporal flexibility of the upstream task is determined by the earliest downstream task. For example, the slack time of task 9 is determined by the task that starts earliest in its downstream tasks, i.e., task 12.
Over the slack time distribution of the processes of the periodic subtasks over weekdays and weekends, the slack time of the processes of the periodic subtasks decreases as their ending processing time approaches 24: 00. I.e., the periodic subtasks submitted on the same day and having a period of days are generally completed within the same day. In addition, there is a portion (66% weekday and 68% weekend) of the periodic subtasks with a deadline of 8: 00. This observation is consistent with the observation of fig. 8, that is, in the cluster under study, most of the periodic tasks are scheduled to be completed before 8:00 for management purposes.
For the estimation of power consumption, a data-driven power consumption mapping model can be established in the dimension of the periodic subtasks. The power consumption mapping model can be obtained by adopting neural network training. Figure 6 shows the results at 15 minute intervals during the weekday and weekend days. It can be seen that the power consumption curve and resource usage curve for the process of the periodic subtasks have similar trends, i.e., peak in the early morning and then continue to decline until a peak occurs at around 18: 00. Wherein there is a non-linear correlation between CPU and memory utilization and power consumption of 2:00 to 6: 00. This is because CPU and memory utilization affect server power usage differently for different types of computing tasks. If the periodic subtasks generated during this time need to write results to disk, more power is required on the hard disk on the server. Furthermore, possible interference between the execution of the various tasks may also result in a total resource usage that is not equal to the simple sum of all task resource consumptions. Due to the change in the user request, it can be seen that the power consumption situation on weekdays is slightly different from the weekend situation.
Flexibility with respect to power may be performed in conjunction with a power system operating environment, and fig. 7A, 7B depict flexible power loads for periodic subtasks to illustrate the time flexibility potential of power loads on these server clusters. Wherein the power load situation of the periodic subtasks is divided into 7 groups, thereby better showing the time flexibility of load transfer. As can be seen from fig. 7A and 7B, the delay time of about 68% of the power load is shorter than 15 minutes, and the power load with 11%, 9% and 8.9% flexibility can perform time shift within [1h,2h), [30min,1h) and [2h,4h), respectively. The power and time flexibility scores between weekdays and weekends are similar. In the embodiment, in the flexibility quantification link in the power system operation environment, the power consumption caused by the periodic subtasks in the working day is used as the original load for evaluation.
In a scenario that an IDC server cluster and a high wind permeability power system are operated cooperatively, the IDC adjusts a power load curve of the IDC by adjusting an execution mode of a periodic subtask under the limitation of computational resource burden and execution time limit, and can be determined by the following formula:
Figure 446096DEST_PATH_IMAGE014
(10)
Figure 304331DEST_PATH_IMAGE015
(11)
Figure 854261DEST_PATH_IMAGE016
(12)
wherein the content of the first and second substances,
Figure 532367DEST_PATH_IMAGE017
is a variable 0-1 indicating whether to transfer the task to within a period k, if
Figure 825945DEST_PATH_IMAGE018
It means that the periodic subtask j is shifted into the kth time period. Equation (11) constrains that the process for each periodic subtask should run only once per day and be completed before its latest execution time. Equation (12) ensures that the aggregate power load of all periodic subtasks over the k period should not exceed the power capacity of the server after readjustment. The model is tested by combining historical hour wind power generation of a certain year according to the embodiment of the application. The method can solve different operation target problems of the power system, such as enhancement of photovoltaic utilization rate, reduction of operation cost and the like.
FIG. 8 shows the results of the load shifting flexibility potential quantification of the periodic subtasks. The solid line represents the raw load curve of the periodic subtasks, and the other dashed lines represent some IDC power load curves in different wind scenarios. Therefore, compared with the original load represented by a solid line, the IDC can rearrange the processing mode of the periodic calculation task in the scene facing the wind power consumption, so that the wind power consumption proportion is improved along with the change of the wind power output. The grey area represents the flexible load space between the maximum upward load adjustment and the downward load adjustment, representing the time transfer capability per time interval. As can be seen, as time goes closer to 24:00, the narrower the gray area. It can be concluded that these periodic tasks will reduce the flexible power load over time.
Fig. 9A and 9B show IDC agility power load and agility power load ratio in different wind power scenarios, which correspond to fig. 8. As can be seen from the figure, the load transfer flexibility of the IDC mainly occurs between 2:00 and 9: 00. During this time, the upward load adjustment is about 130 kilowatts at maximum, and the downward load adjustment is about 125 kilowatts at maximum. Furthermore, the maximum FDR is about 60% and down about 20%. Experimental results prove that the IDC can be used for wind power output with high randomness, the flexible operation potential of the IDC is exerted, and the clean energy utilization rate of a power system and the IDC is improved in a load valley period.
In the embodiment of the application, the power consumption mapping can be accurately performed under the condition that the server configuration and the calculation tasks have heterogeneity, and the six server power consumption mapping models are tested by using the current data set to perform performance comparison. In the control group, the power demand of the servers of the three models is established as a mathematical model with CPU or memory utilization as key variables, and the other three models regard the number of executed computing tasks as key variables affecting the power demand. The accuracy of these models is measured by Mean Absolute Error (MAE) and Normalized Root Mean Square Error (NRMSE). According to experiments, the power mapping neural network is trained by reading historical data generated on various production servers, so that the power mapping relation suitable for the servers is obtained, the limitation that a static model cannot be applied to highly heterogeneous servers in a production environment can be well overcome, and the result can be more accurate.
The test results of the other three models of the embodiment of the application show that the method has higher accuracy. Since the calculation tasks are highly diversified and huge in actual operation production, there is no adaptability if the server power is estimated only by the number of execution calculation tasks. In contrast, the quantization method proposed by the present embodiment can obtain the consumed power according to the time-varying resource usage.
The application also designs a comparative example to illustrate the effect of considering data dependence in the method. If there is no downstream computing task as its latest completion time, the upstream periodic subtasks may be executed at any time during the post-arrival commit day. In this case, the latest end time of all periodic subtasks is set to 24: 00. The comparative case was tested in the same operating scenario of a high wind penetration power system, and the simulation results are shown in fig. 10.
It can be seen that the periodic subtasks are commonly shifted to morning and evening processing, and that the maximum upward flexibility load and the maximum downward flexibility load are 1250 kilowatts and 500 kilowatts, respectively. However, this case significantly amplifies the IDC flexibility potential, since most batch computation tasks have data dependency constraints, and periodic subtasks need to be scheduled under constraints that take into account the order of execution due to data dependency.
In summary, the embodiments of the present application provide a data-driven evaluation framework for IDC to investigate the aggregate power consumption flexibility of periodic batch processing computing tasks and to use a computing task data set in a real production environment to illustrate its effectiveness.
On the basis of the above embodiments, the present embodiment describes the evaluation of power consumption and the adjustment of power in conjunction with a specific power utilization example.
Referring to fig. 10, a flowchart illustrating steps of another embodiment of a power consumption processing method of an IDC of the present application is shown.
Step 1002, querying a log record of a process in an operation log, and acquiring operation parameters of the process, where the operation parameters include: signature information and start processing time.
And 1004, screening a repeatedly executed process according to the signature information, and adding the repeatedly executed process to the first set.
Optionally, the screening, according to the signature information, a repeatedly executed process added to the first set includes: counting based on the signature information, and determining the number of times of the signature information; judging whether the times of the signature information exceed a repetition threshold value; and when the times of the signature information exceed a repetition threshold, adding the process corresponding to the signature information into a first set.
Step 1006, according to the processing start time, selecting a periodically executed process from the first set, and adding the periodically executed process to a second set.
Optionally, the process executed periodically is screened from the first set according to the processing start time, and the process includes: determining a time interval of starting processing time of two processes in the first set; and calculating the change coefficient of each time interval, and determining the periodically executed process according to the change coefficient.
Step 1008 determines a corresponding processing time according to the start processing time and the end processing time.
Step 1010, determining average resource usage information according to the resource usage information and the processing time, where the resource usage information includes: average CPU usage and/or average memory usage.
Step 1012, determining the time parameter of the periodically executed process according to the dependency relationship of the periodically executed process.
Wherein, the determining the time parameter of the periodically executed process according to the dependency relationship of the periodically executed process includes: determining a time parameter of a downstream sub-process to be a specified value when the execution of the process is independent of the process; in the case that execution of a specified downstream process depends on the process, determining a time parameter thereof as a time difference between a start processing time of the specified downstream process and an end processing time of the process; in the case that execution of more than one downstream process depends on the process, a target downstream process with the earliest starting processing time among the more than one downstream process is determined, and a time parameter of the target downstream process is determined as a time difference between the starting processing time of the target downstream process and the ending processing time of the process.
Step 1014, inputting the resource information of the periodically executed process into a power consumption mapping model to obtain the estimated power consumption information of the periodically executed process.
Step 1016, determining the flexibility parameter of the IDC according to the power consumption information and the time parameter of the process executed periodically.
And adjusting the starting processing time of the periodically executed process according to the power consumption information and the time parameter of the periodically executed process. The adjusting the processing start time of the periodically executed process according to the power consumption information and the time parameter of the periodically executed process includes: acquiring power supply information; the power supply information and the power consumption information and the time parameter of the periodically executed process adjust a start processing time of the periodically executed process.
The method and the device for quantizing the IDC power utilization flexibility can realize data-driven IDC power utilization flexibility quantization, including identifying the process of the periodic subtasks, extracting the key operation mode, mapping power consumption and calculating task execution, and quantizing the power utilization flexibility under the operation environment of the power system. The power consumption and time flexibility of the computing tasks are estimated by using historical production data, thereby overcoming the limitations of the existing model on heterogeneous computing tasks and server configurations.
Flexibility quantification also takes into account the execution of the periodic computation tasks and the data dependencies between the subtasks. Arrival patterns and execution order constraints for periodic computing tasks are extracted from historical production data sets to assess power consumption flexibility. The real task scheduling and the dependency among the tasks are considered, and the feasibility and the accuracy of the method in the actual running environment are highlighted.
It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the embodiments are not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the embodiments. Further, those skilled in the art will also appreciate that the embodiments described in the specification are presently preferred and that no particular act is required of the embodiments of the application.
On the basis of the above embodiments, the present embodiment further provides a power consumption processing apparatus for an internet data center IDC, which is applied to an electronic device of a server device (cluster).
The system comprises a period identification module, a task execution module and a task execution module, wherein the period identification module is used for determining a period execution process, the operation parameters of the process can be obtained based on an operation log, and the period execution process is identified according to the operation parameters, and the process is a process of a subtask of a calculation task of the IDC;
the resource analysis module is used for determining resource information of the periodically executed process according to the operation parameters of the periodically executed process;
and the power consumption estimation module is used for estimating the power consumption information of the periodically executed process according to the resource information of the periodically executed process.
In summary, the operation parameters of the processes can be obtained based on the operation log, so that the periodic processes are identified based on the operation parameters, the processes are the processes of the subtasks of the calculation task of the IDC, then the resource information of the processes executed periodically is determined according to the operation parameters of the processes executed periodically, and the power consumption information of the processes executed periodically is estimated based on the resource information, so that the power consumption of each process can be accurately estimated, and a basis is provided for the IDC to provide the operation flexibility for the power system.
Optionally, the cycle identifying module includes:
the log analysis submodule is used for inquiring log records of a process in an operation log and acquiring operation parameters of the process, wherein the operation parameters comprise: signature information and start processing time;
the repeated analysis submodule is used for screening a repeated execution process according to the signature information and adding the process into the first set;
and the period screening submodule is used for screening the period execution process from the first set according to the starting processing time and adding the period execution process into the second set.
Optionally, the repeated analysis sub-module is configured to count based on the signature information, and determine the number of times of the signature information; judging whether the times of the signature information exceed a repetition threshold value; and when the times of the signature information exceed a repetition threshold, adding the process corresponding to the signature information into a first set.
Optionally, the period screening submodule is configured to determine a time interval between start processing times of two processes in the first set; and calculating the change coefficient of each time interval, and determining the periodically executed process according to the change coefficient.
Optionally, the resource analysis module is configured to determine a corresponding processing time according to the start processing time and the end processing time; determining average resource usage information from the resource usage information and the processing time, the resource usage information comprising: average CPU usage and/or average memory usage.
Optionally, the resource analysis module is further configured to determine a time parameter of the periodically executed process according to the dependency relationship of the periodically executed process.
Optionally, the resource analysis module is configured to determine that the time parameter is a specified value when the execution of the downstream sub-process does not depend on the process; in the case that execution of a specified downstream process depends on the process, determining a time parameter thereof as a time difference between a start processing time of the specified downstream process and an end processing time of the process; in the case that execution of more than one downstream process depends on the process, a target downstream process with the earliest starting processing time among the more than one downstream process is determined, and a time parameter of the target downstream process is determined as a time difference between the starting processing time of the target downstream process and the ending processing time of the process.
Optionally, the power consumption estimating module is configured to input the resource information of the periodically executed process into a power consumption mapping model, so as to obtain estimated power consumption information of the periodically executed process.
Optionally, the method further includes: and the adjusting module is used for adjusting the starting processing time of the periodically executed process according to the power consumption information and the time parameter of the periodically executed process.
Optionally, the adjusting module is configured to obtain power supply information; the power supply information and the power consumption information and the time parameter of the periodically executed process adjust a start processing time of the periodically executed process.
Optionally, the method further includes: and the flexible analysis module is used for determining the flexibility parameters of the IDC according to the power consumption information and the time parameters of the periodically executed process.
The method and the device for quantizing the IDC power utilization flexibility can realize data-driven IDC power utilization flexibility quantization, including identifying the process of the periodic subtasks, extracting the key operation mode, mapping power consumption and calculating task execution, and quantizing the power utilization flexibility under the operation environment of the power system. The power consumption and time flexibility of the computing tasks are estimated by using historical production data, thereby overcoming the limitations of the existing model on heterogeneous computing tasks and server configurations.
Flexibility quantification also takes into account the execution of the periodic computation tasks and the data dependencies between the subtasks. Arrival patterns and execution order constraints for periodic computing tasks are extracted from historical production data sets to assess power consumption flexibility. The real task scheduling and the dependency among the tasks are considered, and the feasibility and the accuracy of the method in the actual running environment are highlighted.
The present application further provides a non-transitory, readable storage medium, where one or more modules (programs) are stored, and when the one or more modules are applied to a device, the device may execute instructions (instructions) of method steps in this application.
Embodiments of the present application provide one or more machine-readable media having instructions stored thereon, which when executed by one or more processors, cause an electronic device to perform the methods as described in one or more of the above embodiments. In the embodiment of the present application, the electronic device includes various types of devices such as a terminal device and a server (cluster).
Embodiments of the present disclosure may be implemented as an apparatus, which may include electronic devices such as a terminal device, a server (cluster), etc. within a data center, using any suitable hardware, firmware, software, or any combination thereof, in a desired configuration. Fig. 11 schematically illustrates an example apparatus 1100 that may be used to implement various embodiments described herein.
For one embodiment, fig. 11 illustrates an example apparatus 1100 having one or more processors 1102, a control module (chipset) 1104 coupled to at least one of the processor(s) 1102, a memory 1106 coupled to the control module 1104, a non-volatile memory (NVM)/storage 1108 coupled to the control module 1104, one or more input/output devices 1110 coupled to the control module 1104, and a network interface 1112 coupled to the control module 1104.
The processor 1102 may include one or more single-core or multi-core processors, and the processor 1102 may include any combination of general-purpose or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, the apparatus 1100 can be implemented as a terminal device, a server (cluster), or the like in the embodiments of the present application.
In some embodiments, the apparatus 1100 may include one or more computer-readable media (e.g., the memory 1106 or the NVM/storage 1108) having instructions 1114 and one or more processors 1102 in combination with the one or more computer-readable media configured to execute the instructions 1114 to implement modules to perform the actions described in this disclosure.
For one embodiment, control module 1104 may include any suitable interface controllers to provide any suitable interface to at least one of the processor(s) 1102 and/or to any suitable device or component in communication with control module 1104.
The control module 1104 may include a memory controller module to provide an interface to the memory 1106. The memory controller module may be a hardware module, a software module, and/or a firmware module.
The memory 1106 may be used, for example, to load and store data and/or instructions 1114 for the device 1100. For one embodiment, memory 1106 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, the memory 1106 may comprise a double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).
For one embodiment, control module 1104 may include one or more input/output controllers to provide an interface to NVM/storage 1108 and input/output device(s) 1110.
For example, NVM/storage 1108 may be used to store data and/or instructions 1114. NVM/storage 1108 may include any suitable non-volatile memory (e.g., flash memory) and/or may include any suitable non-volatile storage device(s) (e.g., one or more Hard Disk Drives (HDDs), one or more Compact Disc (CD) drives, and/or one or more Digital Versatile Disc (DVD) drives).
NVM/storage 1108 may include storage resources that are physically part of the device on which apparatus 1100 is installed, or it may be accessible by the device and need not be part of the device. For example, NVM/storage 1108 may be accessed over a network via input/output device(s) 1110.
Input/output device(s) 1110 may provide an interface for apparatus 1100 to communicate with any other suitable device, input/output devices 1110 may include communication components, audio components, sensor components, and so forth. Network interface 1112 may provide an interface for device 1100 to communicate over one or more networks, and device 1100 may communicate wirelessly with one or more components of a wireless network according to any of one or more wireless network standards and/or protocols, such as access to a communication standard-based wireless network, e.g., WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof.
For one embodiment, at least one of the processor(s) 1102 may be packaged together with logic for one or more controller(s) (e.g., memory controller module) of the control module 1104. For one embodiment, at least one of the processor(s) 1102 may be packaged together with logic for one or more controller(s) of control module 1104 to form a System In Package (SiP). For one embodiment, at least one of the processor(s) 1102 may be integrated on the same die with logic for one or more controller(s) of the control module 1104. For one embodiment, at least one of the processor(s) 1102 may be integrated on the same die with logic for one or more controller(s) of control module 1104 to form a system on chip (SoC).
In various embodiments, the apparatus 1100 may be, but is not limited to: a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet, a netbook, etc.), among other terminal devices. In various embodiments, the apparatus 1100 may have more or fewer components and/or different architectures. For example, in some embodiments, device 1100 includes one or more cameras, keyboards, Liquid Crystal Display (LCD) screens (including touch screen displays), non-volatile memory ports, multiple antennas, graphics chips, Application Specific Integrated Circuits (ASICs), and speakers.
The detection device can adopt a main control chip as a processor or a control module, sensor data, position information and the like are stored in a memory or an NVM/storage device, a sensor group can be used as an input/output device, and a communication interface can comprise a network interface.
For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.
The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing terminal to cause a series of operational steps to be performed on the computer or other programmable terminal to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present application have been described, additional variations and modifications of these embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all such alterations and modifications as fall within the true scope of the embodiments of the application.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or terminal that comprises the element.
The method and the device for processing power consumption of the IDC, the terminal device and the machine-readable medium provided by the application are introduced in detail, and specific examples are applied in the description to explain the principle and the implementation of the application, and the description of the above embodiments is only used for helping to understand the method and the core idea of the application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims (13)

1. A power consumption processing method of an Internet Data Center (IDC), the method comprising:
determining a periodically executed process;
determining resource information of the periodically executed process according to the operation parameters of the periodically executed process;
and estimating the power consumption information of the periodically executed process according to the resource information of the periodically executed process.
2. The method of claim 1, wherein determining the periodically executed processes comprises:
inquiring log records of a process in an operation log, and acquiring operation parameters of the process, wherein the operation parameters comprise: signature information and start processing time;
screening a repeatedly executed process according to the signature information, and adding the repeatedly executed process to a first set;
and screening the periodically executed processes from the first set according to the starting processing time, and adding the periodically executed processes to a second set.
3. The method of claim 2, wherein said screening the repeatedly executed processes from the signature information to add to the first set comprises:
counting based on the signature information, and determining the number of times of the signature information;
judging whether the times of the signature information exceed a repetition threshold value;
and when the times of the signature information exceed a repetition threshold, adding the process corresponding to the signature information into a first set.
4. A method according to claim 2 or 3, wherein the screening of the first set for periodically executed processes in dependence on the start processing time comprises:
determining a time interval of starting processing time of two processes in the first set;
and calculating the change coefficient of each time interval, and determining the periodically executed process according to the change coefficient.
5. The method of claim 1, wherein the determining resource information of the periodically executed process according to the operation parameters of the periodically executed process comprises:
determining corresponding processing time according to the starting processing time and the ending processing time;
determining average resource usage information from the resource usage information and the processing time, the resource usage information comprising: average CPU usage and/or average memory usage.
6. The method of claim 1 or 5, further comprising:
and determining the time parameter of the periodically executed process according to the dependency relationship of the periodically executed process.
7. The method of claim 6, wherein determining the time parameter of the periodically executed process according to the dependency relationship of the periodically executed process comprises:
determining a time parameter of a downstream sub-process to be a specified value when the execution of the process is independent of the process;
in the case that execution of a specified downstream process depends on the process, determining a time parameter thereof as a time difference between a start processing time of the specified downstream process and an end processing time of the process;
in the case that execution of more than one downstream process depends on the process, a target downstream process with the earliest starting processing time among the more than one downstream process is determined, and a time parameter of the target downstream process is determined as a time difference between the starting processing time of the target downstream process and the ending processing time of the process.
8. The method of claim 6, wherein estimating power consumption information of the periodically executed process based on the resource information of the periodically executed process comprises:
and inputting the resource information of the periodically executed process into a power consumption mapping model to obtain the estimated power consumption information of the periodically executed process.
9. The method of claim 8, further comprising:
and adjusting the starting processing time of the periodically executed process according to the power consumption information and the time parameter of the periodically executed process.
10. The method of claim 9, wherein the adjusting the starting processing time of the periodically executed process according to the power consumption information and the time parameter of the periodically executed process comprises:
acquiring power supply information;
the power supply information and the power consumption information and the time parameter of the periodically executed process adjust a start processing time of the periodically executed process.
11. The method of claim 8, further comprising:
and determining the flexibility parameter of the IDC according to the power consumption information and the time parameter of the process executed periodically.
12. An electronic device, comprising: a processor; and
a memory having executable code stored thereon that, when executed, causes the processor to perform the method of any of claims 1-11.
13. One or more machine-readable media having executable code stored thereon that, when executed, causes a processor to perform the method of any of claims 1-11.
CN202210109703.2A 2022-01-29 2022-01-29 Power consumption processing method and device for internet data center IDC and readable medium Active CN114138095B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210109703.2A CN114138095B (en) 2022-01-29 2022-01-29 Power consumption processing method and device for internet data center IDC and readable medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210109703.2A CN114138095B (en) 2022-01-29 2022-01-29 Power consumption processing method and device for internet data center IDC and readable medium

Publications (2)

Publication Number Publication Date
CN114138095A true CN114138095A (en) 2022-03-04
CN114138095B CN114138095B (en) 2022-05-10

Family

ID=80381772

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210109703.2A Active CN114138095B (en) 2022-01-29 2022-01-29 Power consumption processing method and device for internet data center IDC and readable medium

Country Status (1)

Country Link
CN (1) CN114138095B (en)

Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030115239A1 (en) * 2000-09-22 2003-06-19 Atsushi Togawa Arithmetic processing system and arithmetic processing control method, task managemnt system and task management method, and storage medium
CN101067758A (en) * 2007-06-14 2007-11-07 华南理工大学 Energy consumption management method for inserting system
US20100174928A1 (en) * 2009-01-05 2010-07-08 International Business Machines Corporation Optimizer Mechanism to Increase Battery Length for Mobile Devices
US20130132754A1 (en) * 2010-03-23 2013-05-23 Sony Corporation Reducing power consumption by masking a process from a processor performance management system
US20160103711A1 (en) * 2014-10-09 2016-04-14 Vmware, Inc. Methods and systems to optimize data center power consumption
US20170083383A1 (en) * 2015-09-23 2017-03-23 Tomer RIDER Task assignment for processor cores based on a statistical power and frequency model
US20170132523A1 (en) * 2015-11-09 2017-05-11 Nec Laboratories America, Inc. Periodicity Analysis on Heterogeneous Logs
CN108574600A (en) * 2018-03-20 2018-09-25 北京航空航天大学 The QoS guarantee method of the power consumption and resource contention Collaborative Control of cloud computing server
US20180302295A1 (en) * 2017-04-17 2018-10-18 Fujitsu Limited Information processing device, information processing system control method, and recording medium recording management program of information processing device
CN109324891A (en) * 2017-07-31 2019-02-12 沈阳高精数控智能技术股份有限公司 A kind of periodic duty low-power consumption scheduling method of ratio free time distribution
CN110308782A (en) * 2018-03-22 2019-10-08 阿里巴巴集团控股有限公司 Power consumption prediction, control method, equipment and computer readable storage medium
CN111078401A (en) * 2019-12-02 2020-04-28 华侨大学 Periodic task temperature sensing energy consumption optimization method
CN111259561A (en) * 2020-02-07 2020-06-09 华侨大学 Time interval-based processor energy consumption calculation method based on temperature sensing
US20200387334A1 (en) * 2019-06-07 2020-12-10 Fuji Xerox Co., Ltd. Information processing apparatus and non-transitory computer readable medium

Patent Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030115239A1 (en) * 2000-09-22 2003-06-19 Atsushi Togawa Arithmetic processing system and arithmetic processing control method, task managemnt system and task management method, and storage medium
CN101067758A (en) * 2007-06-14 2007-11-07 华南理工大学 Energy consumption management method for inserting system
US20100174928A1 (en) * 2009-01-05 2010-07-08 International Business Machines Corporation Optimizer Mechanism to Increase Battery Length for Mobile Devices
US20130132754A1 (en) * 2010-03-23 2013-05-23 Sony Corporation Reducing power consumption by masking a process from a processor performance management system
US20160103711A1 (en) * 2014-10-09 2016-04-14 Vmware, Inc. Methods and systems to optimize data center power consumption
US20170083383A1 (en) * 2015-09-23 2017-03-23 Tomer RIDER Task assignment for processor cores based on a statistical power and frequency model
US20170132523A1 (en) * 2015-11-09 2017-05-11 Nec Laboratories America, Inc. Periodicity Analysis on Heterogeneous Logs
US20180302295A1 (en) * 2017-04-17 2018-10-18 Fujitsu Limited Information processing device, information processing system control method, and recording medium recording management program of information processing device
CN109324891A (en) * 2017-07-31 2019-02-12 沈阳高精数控智能技术股份有限公司 A kind of periodic duty low-power consumption scheduling method of ratio free time distribution
CN108574600A (en) * 2018-03-20 2018-09-25 北京航空航天大学 The QoS guarantee method of the power consumption and resource contention Collaborative Control of cloud computing server
CN110308782A (en) * 2018-03-22 2019-10-08 阿里巴巴集团控股有限公司 Power consumption prediction, control method, equipment and computer readable storage medium
US20200387334A1 (en) * 2019-06-07 2020-12-10 Fuji Xerox Co., Ltd. Information processing apparatus and non-transitory computer readable medium
CN111078401A (en) * 2019-12-02 2020-04-28 华侨大学 Periodic task temperature sensing energy consumption optimization method
CN111259561A (en) * 2020-02-07 2020-06-09 华侨大学 Time interval-based processor energy consumption calculation method based on temperature sensing

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
张忆文等: "资源受限周期任务低能耗调度算法", 《小型微型计算机系统》 *
王科特等: "基于多核处理器的K线程低能耗的任务调度优化算法", 《计算机科学》 *

Also Published As

Publication number Publication date
CN114138095B (en) 2022-05-10

Similar Documents

Publication Publication Date Title
US9696786B2 (en) System and method for optimizing energy consumption by processors
Simakov et al. A slurm simulator: Implementation and parametric analysis
US8578023B2 (en) Computer resource utilization modeling for multiple workloads
CN106156115B (en) A kind of resource regulating method and device
US20220027744A1 (en) Resource data modeling, forecasting and simulation
US11700210B2 (en) Enhanced selection of cloud architecture profiles
Lu et al. A GRU-based prediction framework for intelligent resource management at cloud data centres in the age of 5G
Sîrbu et al. Power consumption modeling and prediction in a hybrid CPU-GPU-MIC supercomputer
Cao et al. Data-driven flexibility assessment for internet data center towards periodic batch workloads
CN112685153A (en) Micro-service scheduling method and device and electronic equipment
Borghesi et al. Pricing schemes for energy-efficient HPC systems: Design and exploration
CN113296905A (en) Scheduling method, scheduling device, electronic equipment, storage medium and software product
KR20220115357A (en) A method and apparatus for generating future demand forecast data based on attention mechanism
Lin et al. An adaptive workload-aware power consumption measuring method for servers in cloud data centers
Jiang et al. Effective data management strategy and RDD weight cache replacement strategy in Spark
CN117687356A (en) Production scheduling method, terminal device and computer readable storage medium
CN114138095B (en) Power consumption processing method and device for internet data center IDC and readable medium
Witt et al. Learning low-wastage memory allocations for scientific workflows at icecube
Ahmed et al. Enabling demand response for hpc systems through power capping and node scaling
EP3826233B1 (en) Enhanced selection of cloud architecture profiles
Sen et al. Predictive price-performance optimization for serverless query processing
CN115373836A (en) Computing network, computing force measuring method, scheduling device and related products
Claßen et al. Carbon-awareness in ci/cd
Al Qassem et al. Containerized Microservices: A Survey of Resource Management Frameworks
CN105824809B (en) A kind of implementation method and device of ETL scheduling

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information

Address after: Room 554, 5 / F, building 3, 969 Wenyi West Road, Wuchang Street, Yuhang District, Hangzhou City, Zhejiang Province

Applicant after: Alibaba (China) Co.,Ltd.

Address before: Room 508, 5 / F, building 4, No. 699, Wangshang Road, Changhe street, Binjiang District, Hangzhou City, Zhejiang Province

Applicant before: Alibaba (China) Co.,Ltd.

CB02 Change of applicant information
GR01 Patent grant
GR01 Patent grant