CN107315636B - Resource availability early warning method and device - Google Patents

Abstract

The invention provides a resource availability early warning method, which comprises the following steps: establishing a resource usage amount estimation model; predicting the resource usage of the next time period through the resource usage prediction model; and when the estimated resource usage amount of the next period exceeds a preset threshold value, sending out a resource availability early warning. The invention also provides a resource availability degree early warning device.

Description

Method and device for early warning of resource availability

technical field

The invention relates to the technical field of business support, in particular to a method and device for early warning of Hadoop resource availability.

Background technique

Hadoop is one of the mainstream software of the current big data platform. It provides a distributed massive data storage (HDFS, Hadoop Distributed File System), distributed large-scale computing (MapReduce) and a general resource management system (YARN, Yet Another Resource). Negotiator), which has the advantages of high fault tolerance, ease of use and scalability, and is widely used in data mining, online analytical processing (OLAP, Online Analytical Processing), business analysis, etc. Market segmentation and customer relationship management, forecasting future market trends, etc., provide decision support for business leaders and realize the purpose of data value-added realization. At present, Hadoop has been widely used in many fields such as the Internet, communication, and finance.

The existing YARN-based Hadoop architecture is shown in Figure 1, which mainly includes a series of global resource manager (RM, ResourceManager), application master management program (AM, ApplicationMaster), node manager (NM, NodeManager) and container Container. module.

The overall execution process of Hadoop based on the YARN framework is as follows:

Step 1: The user submits applications such as MapReduce through the client JobClient, and applies for resources to the RM;

Step 2: The global application manager (ASM, ApplicationsManager) and resource scheduler (RS, ResourceScheduler) in the RM assign the first container Container to the application after accepting the request, and find the corresponding NM to communicate with it, and send out the The command to start AM in the Container;

Step 3: AM registers itself in the RM, and then uses the polling method to apply for resources for each task through the remote procedure call RPC protocol, mainly including CPU, memory, etc.;

Step 4: When the AM receives the resource, it will communicate with the NM, and the NM will start the task to be executed;

Step 5: Each task reports its current status to the AM through the RPC protocol, and the AM monitors the running status of all tasks. If the task fails to run, it will re-apply for resources and then restart the task;

Step 6: When the execution of the application is completed, the AM logs out to the RM and closes itself, and recycles and releases related resources.

Although Hadoop's existing YARN architecture provides good support for resource management and task scheduling monitoring, it can only monitor task status at present, and cannot issue early warnings to users when resources are insufficient, so resources cannot be adjusted in advance, which may lead to If the progress of the task is close to completion but the resources are seriously insufficient, the task may only be restarted and resources allocated, which will result in a waste of time and resources.

SUMMARY OF THE INVENTION

In view of this, the embodiments of the present invention expect to provide a resource availability early warning method and apparatus, which can and can effectively reduce resource waste and improve resource utilization.

In order to achieve the above object, the technical scheme of the present invention is achieved in this way:

An embodiment of the present invention provides a method for early warning of resource availability, and the method includes:

Establish a resource usage estimation model;

Estimate the resource usage in the next time period by using the resource usage estimation model;

When the estimated resource usage in the next cycle exceeds a preset threshold, a resource availability warning is issued.

In the above scheme, the establishment of a resource usage estimation model includes:

Determine the resource consumption increment corresponding to the progress increment of the task in the next time period according to the progress of all parallel tasks in each time period and the increment of consumed resources, as well as the time period regularity and correlation;

Comparing the consuming resource increments corresponding to the determined task progress increments in the next time period multiple times with the consuming resource increments corresponding to the actual task progress increments, dynamically adjusting the weights to reduce errors;

Select the weight corresponding to the minimum error to establish a resource usage estimation model.

In the above solution, the estimation of the resource usage in the next time period by the resource usage estimation model includes but is not limited to: estimation of the CPU resources in the next time period by the resource usage estimation model. Usage and memory resource usage.

In the above solution, when the estimated resource usage in the next cycle exceeds the first threshold, it includes: the estimated resource usage in the next cycle exceeds the threshold of the remaining available resources.

In the above solution, when the estimated resource usage in the next cycle exceeds a preset threshold, issuing a resource availability warning includes: when the estimated resource usage in the next cycle exceeds a preset threshold , according to the priority, importance and dependencies of the task to issue different levels of resource availability early warning.

The embodiment of the present invention also provides a resource availability early warning device, the device includes: a model establishment module, a resource estimation module, and a resource early warning module, wherein,

The model establishment module is used to establish a resource usage estimation model;

The resource estimation module is used to estimate the resource usage in the next time period through the resource usage estimation model;

The resource early warning module is configured to issue a resource availability early warning when the estimated resource usage in the next cycle exceeds a preset threshold.

In the above scheme, the model establishment module is specifically used for:

Determine the resource consumption increment corresponding to the progress increment of the task in the next time period according to the progress of all parallel tasks in each time period and the increment of consumed resources, as well as the time period regularity and correlation;

Comparing the consuming resource increments corresponding to the determined task progress increments in the next time period multiple times with the consuming resource increments corresponding to the actual task progress increments, dynamically adjusting the weights to reduce errors;

Select the weight corresponding to the minimum error to establish a resource usage estimation model.

In the above solution, the resource estimation module is specifically configured to: estimate the CPU resource usage and the memory resource usage in the next time period through the resource usage estimation model.

In the above solution, when the resource early warning module judges that the estimated resource usage in the next cycle exceeds the first threshold, the step includes: the resource early warning module determines that the estimated resource usage in the next cycle exceeds the threshold of the remaining available resources. .

In the above solution, the resource early warning module is specifically configured to: when the estimated resource usage in the next cycle exceeds a preset threshold, issue early warning of resource availability at different levels according to the priority, importance and dependency of the task.

The resource availability early warning method and device provided by the embodiments of the present invention first establish a resource usage estimation model, and then estimate the resource usage in the next time period through the resource usage estimation model. When the estimated resource usage in the next cycle exceeds the preset threshold, a resource availability warning is issued. In this way, a situation in which a large number of tasks fail due to insufficient available resources can be avoided, the insufficiency of the original operation mechanism can be improved, and resource waste can be effectively reduced and resource utilization can be improved.

Description of drawings

Figure 1 is a diagram of the Hadoop architecture based on YARN;

2 is a schematic flowchart of a method for early warning of resource availability according to an embodiment of the present invention;

3 is a schematic diagram of the principle of a method for early warning of resource availability according to an embodiment of the present invention;

4 is a schematic structural diagram of an apparatus for early warning of resource availability according to an embodiment of the present invention;

5 is a schematic structural diagram of a resource availability early warning system according to an embodiment of the present invention;

FIG. 6 is a schematic work flow diagram of a resource availability early warning system according to an embodiment of the present invention.

Detailed ways

In the embodiment of the present invention, a resource usage estimation model is first established, and then the resource usage estimation model is used to estimate the resource usage in the next time period. When the estimated resource usage in the next period is When the preset threshold is exceeded, a resource availability warning is issued.

Under normal circumstances, the normal operation of Hadoop is inseparable from the support of sufficient resources such as CPU and memory. The availability of various resources has different effects on the operation of Hadoop tasks. For example, if the CPU resources are insufficient, it may not directly cause If the task fails, it will cause the overall running speed and processing efficiency to drop significantly, resulting in a large number of task backlogs and queues; if the memory resources are not enough, it is often fatal, memory overflow will occur, task execution will be interrupted, and many tasks will fail. Therefore, it is necessary to monitor the availability of Hadoop resources in real time and give early warning before the availability of resources is low.

The embodiment of the present invention provides a method for early warning of Hadoop resource availability. First, a dynamic feedback learning model based on time series is established. The preset rules learn the potential rules and associations in this time period and generate the increment of consumed resources corresponding to the increment of task progress in the next period of time, and then compare the actual task progress and the increment value of consumed resources to continuously and repeatedly dynamically adjust The weight reduction error optimization model is to take the optimal weight with the smallest error in the current time series from a series of weights to form a model, and then estimate the resource size required to complete the remaining tasks on the basis of this model. If the resource value exceeds the threshold of the remaining available resources, different levels of early warnings will be issued according to the priority, importance and dependencies of the tasks, indicating that low resource availability may cause a large number of tasks to fail, and the resource value of the container will be automatically adjusted for the later stage. Provide a reference to provide a basis for users to manually configure resource values, so as to avoid the failure of a large number of tasks due to insufficient available resources, improve the shortcomings of the original operating mechanism, and effectively reduce resource waste and improve resource utilization.

The implementation of the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. FIG. 2 is a schematic flowchart of a method for early warning of resource availability according to Embodiment 1 of the present invention. As shown in FIG. 2 , the method for early warning of resource availability in this embodiment includes the following steps:

Step 201: establish a resource usage estimation model;

In the embodiment of the present invention, the establishment of the resource usage estimation model includes: determining the progress increment of tasks in the next time period according to the progress of all parallel tasks in each time period and the increment of consumed resources, as well as the time period regularity and correlation Corresponding consumption resource increment; compare the consumption resource increment corresponding to the determined next time period task progress increment and the consumption resource increment corresponding to the actual task progress increment multiple times, and dynamically adjust the weights to reduce errors ; Select the weight corresponding to the minimum error to establish a resource usage estimation model.

Specifically, according to the progress of all parallel tasks and the increment of consumed resources within the incremental range of the time series, the potential rules and associations in the current time period are learned according to preset rules, and the task progress increment corresponding to the next time period is generated. Consume the increment of resources, then compare the actual task progress and the incremental value of consumed resources, continuously and repeatedly adjust the weights dynamically to reduce the error optimization model, and take the optimal weight with the smallest error in the current time series from a series of weights form a model.

表示，那么在时间序列范围内t_i时刻到t_i+1时刻内任务j_k的进度增量为

在单位时间内任务j_k进度的平均增量如公式(1-1)所示：FIG. 3 is a schematic diagram of the principle of a method for early warning of resource availability according to an embodiment of the present invention. As shown in FIG. 3 , the set of all tasks in Hadoop is represented as J _ALL ={j ₁ ,j ₂ ,...,j _N }, where N represents the number of all tasks. The time series is set as a period of time from time t _i to time t _i+1 , then the time series is Δt=t _i+1 −t _i . The set of all running tasks in the time series is denoted as J _RUNNING ={j ₁ ,j ₂ ,...,j _n },n≤N. The progress status of all running tasks in Hadoop is monitored and expressed as a percentage, that is, the set JP _RUNNING = {jp ₁ ,jp ₂ ,...,jp _n }, where any task j _k progresses at time t _i using

represents, then the progress increment of task j _k from time t _i to time t _i+1 in the time series is:

The average increment of the progress of task j _k in unit time is shown in formula (1-1):

Correspondingly, calculate the average increment of all task progress in unit time, as shown in formula (1-2):

来衡量，那么这个最快进度的任务剩余的所需完成的进度为

与之同时运行的所有任务的平均进度增量

通过公式(1-3)计算：At time t _i+1 in the time series, the fastest progress among all the running tasks can be used with

To measure, then the remaining progress of this fastest progressing task is

Average progress delta for all tasks running concurrently with it

Calculated by formula (1-3):

In the process of resource availability early warning, the resource consumption of learning tasks is dynamically fed back mainly from the aspects of CPU and memory. Assuming that in the time series from time t _i to time t _i+1 , the increment of CPU resources consumed by all running tasks is expressed as a percentage, that is, Δcp=cp ⁱ⁺¹ -cp ⁱ , then all running tasks consume CPU per unit time The calculation method of resources is as formula (1-4):

During the remaining progress period of completing the fastest progress task within the time series of this cycle, the calculation method of the increment of CPU resources still needed to complete all running tasks in the next time cycle is as shown in formula (1-5):

Compare the actual CPU resource consumption value cp ⁱ⁺² in the next time period and adjust the weight σ _c to reduce the error, as shown in formula (1-6):

从这些权值中取最小误差的最优权值作为当前σ_c值，该值在这个时间序列内随资源消耗的不断变化动态地反复学习优化调整取得最优值。Form a series of weight sets in the range of time series

From these weights, the optimal weight with the smallest error is taken as the current value of σ _c , and this value is dynamically and repeatedly learned to optimize and adjust to obtain the optimal value with the continuous change of resource consumption in this time series.

Similarly, in the time series from time t _i to time t _i+1 , the method of calculating the memory resource consumption of all running tasks per unit time is shown in formula (1-7):

During the remaining progress period of completing the fastest progress task in this cycle, the calculation method of the memory resource increment required to complete all running tasks in the next time cycle is as shown in formula (1-8):

Then compare the actual consumption of memory resource value mp ⁱ⁺² in the next time period and adjust the weight σ _m to reduce the error, such as formula (1-9):

并且动态地反复学习优化调整取最小误差的最优权值作为当前σ_m值。Similarly, a series of weight sets are formed in the time series range

And iteratively learns and optimizes dynamically and takes the optimal weight with the smallest error as the current σ _m value.

Step 202: Estimate the resource usage in the next time period through the resource usage estimation model;

In the embodiment of the present invention, estimating the resource usage in the next time period by using the resource usage estimation model includes but not limited to: estimating the resource usage estimation model in the next time period. CPU resource usage and memory resource usage.

Specifically, on the basis of the optimal resource usage estimation model with minimum error, the progress status of the task with the fastest progress in the time series is used as the benchmark, and the remaining tasks are estimated to be completed according to the current resource consumption of all parallel tasks. the size of the resources required;

In this embodiment of the present invention, in order to ensure that all running tasks can be completed normally, the most important thing is to ensure that there are sufficient resources during the _peak period when the most parallel tasks are running. The number of parallel tasks is the largest at time _i+1 . When the task jp _k with the fastest progress is completed, some resources will be released. At this time, the number of parallel tasks will decrease, and the consumption of related resources such as CPU and memory will also decrease accordingly. Therefore, within the time series range The success of these parallel tasks mainly depends on whether there are enough CPU, memory and other available resources during the time when the fastest progress tasks are running.

The resource estimation model established in step 201 can be used to estimate the size of the resources required to complete the remaining tasks. If the current moment is t _T , the calculation method for estimating the required CPU resource cp ^T is shown in formula (1-10) :

The calculation method for estimating the required memory resource mp ^T is shown in (1-11):

Step 203: When the estimated resource usage in the next cycle exceeds a preset threshold, issue a resource availability warning.

In this embodiment of the present invention, when the estimated resource usage in the next cycle exceeds the first threshold, it includes: the estimated resource usage in the next cycle exceeds the threshold of the remaining available resources.

When the estimated resource usage in the next cycle exceeds the preset threshold, issuing a resource availability warning includes: when the estimated resource usage in the next cycle exceeds the preset threshold, according to the task's Priority, importance, and dependencies issue different levels of resource availability alerts.

Specifically, if the estimated resource usage required to complete the remaining tasks exceeds the threshold of the remaining available resources, different levels of early warnings will be issued according to the different priorities, importance and dependencies of the tasks, indicating that the resource availability may be low. will cause a large number of tasks to fail;

假设CPU资源可用度预警阈值为μ_c，那么预警规则如公式(1-12)所示：The actual remaining available CPU resources of Hadoop at time t _T is

Assuming that the CPU resource availability warning threshold is μ _c , the warning rules are shown in formula (1-12):

设内存资源可用度预警阈值为μ_m，则预警规则如公式(1-13)所示：The actual remaining amount of available memory resources at time t _T is

Set the early warning threshold of memory resource availability to μ _m , the early warning rules are shown in formula (1-13):

Among them, the early warning thresholds μ _c and μ _m can be divided into different levels of alarms such as serious alarms, important alarms, and general alarms according to actual needs. Severe alarms mainly refer to alarms that have a fatal impact on systems, platforms, or applications and require immediate intervention, which are the highest-level alarms; major alarms mainly refer to alarms that partially affect systems, platforms, or applications, which are intermediate-level alarms General alarms are mainly warning-type alarms, which may not have a direct impact on the system, platform, or application. This is the lowest-level alarm; finer-grained alarms can be set according to the actual situation. If the priority of the currently running task is high and important, and many subsequent tasks depend on the task, a higher-level alarm will be generated. In practical applications, the priority, importance, and dependency can be quantified and corresponding thresholds can be set. In this way, different levels of alarms can be generated in a complex and changeable environment.

According to the above method, the early warning of the availability of other resources can also be realized, which realizes real-time monitoring of all available Hadoop resources, and alerts maintenance personnel in advance to expand capacity when resources are insufficient, so as to avoid the reduction of operating efficiency due to insufficient resources. , mission failures, and data loss.

An embodiment of the present invention also provides a resource availability early warning device. FIG. 4 is a schematic structural diagram of a resource availability early warning device according to an embodiment of the present invention. As shown in FIG. 4 , the device includes: a model establishment module 41 and a resource estimation module 42. Resource early warning module 43, wherein,

The model establishment module 41 is used to establish a resource usage estimation model;

In the embodiment of the present invention, the model establishment module is specifically used for:

Determine the resource consumption increment corresponding to the progress increment of the task in the next time period according to the progress of all parallel tasks in each time period and the increment of consumed resources, as well as the time period regularity and relationship; compare the determined next time multiple times The resource consumption increment corresponding to the periodic task progress increment and the consumption resource increment corresponding to the actual task progress increment are dynamically adjusted to reduce the error; the weight corresponding to the minimum error is selected to establish a resource usage estimation model.

Specifically, according to the progress of all parallel tasks and the increment of consumed resources within the incremental range of the time series, the potential rules and associations in the current time period are learned according to preset rules, and the task progress increment corresponding to the next time period is generated. Consume the increment of resources, then compare the actual task progress and the incremental value of consumed resources, continuously and repeatedly adjust the weights dynamically to reduce the error optimization model, and take the optimal weight with the smallest error in the current time series from a series of weights form a model.

The resource estimation module 42 is used to estimate the resource usage in the next time period through the resource usage estimation model;

In the embodiment of the present invention, the resource estimation module is specifically configured to: estimate the CPU resource usage and the memory resource usage in the next time period by using the resource usage estimation model.

Specifically, on the basis of the optimal resource usage estimation model with minimum error, the progress status of the task with the fastest progress in the time series is used as the benchmark, and the remaining tasks are estimated to be completed according to the current resource consumption of all parallel tasks. The size of the required resource.

The resource early warning module 43 is configured to issue a resource availability early warning when the estimated resource usage in the next cycle exceeds a preset threshold.

In the embodiment of the present invention, when the resource early warning module determines that the estimated resource usage in the next cycle exceeds the first threshold, the step includes: the resource early warning module determines that the estimated resource usage in the next cycle exceeds the remaining available resources When the estimated resource usage of the next cycle exceeds the preset threshold, different levels of resource availability warnings are issued according to the priority, importance and dependency of the task.

An embodiment of the present invention further provides a resource availability early warning system. FIG. 5 is a schematic structural diagram of a resource availability early warning system according to an embodiment of the present invention. As shown in FIG. 5 , a model of the resource availability early warning device according to the embodiment of the present invention is shown. The establishment module 41 and the resource estimation module 42 are located in the application task early warning device 51 (AW, ApplicationWarner) in Figure 5, and the resource early warning module 43 is located in the global resource early warning device 52 (RW, ResourceWarner) in Figure 5;

In terms of overall architecture, the resource availability early warning system according to the embodiment of the present invention adds functional modules RW and AW with resource availability monitoring and early warning mechanisms on the basis of the original YARN framework module architecture of Hadoop, respectively realizing global resource monitoring and control. Availability monitoring and early warning and resource availability monitoring and early warning of local container Containers, according to preset rules, push notifications of low resource availability to users in a timely manner, so that users can handle them in advance and provide decision support for operation and maintenance.

The resource availability early-warning system according to the embodiment of the present invention establishes a resource availability early-warning mechanism based on a time-series dynamic feedback learning model. Feedback to AW, AW learns the potential rules and relationships in this time period according to the preset rules, and generates the consumption resource increment corresponding to the task progress increment in the next time period, and then compares the actual task progress and consumption resource increment repeatedly. The dynamic adjustment weights reduce the error optimization model, take the optimal weights with the smallest error in the current time series from a series of weights to form a model, and then estimate the size of the resources required to complete the remaining tasks on this basis, if When the required resource value exceeds the threshold value of the remaining available resources, the AW will notify the RW, and the RW will issue different levels of early warning according to the different priorities, importance and dependencies of the tasks.

In the resource availability early warning system according to the embodiment of the present invention, the user submits application programs such as MapReduce, respectively, applies for resources to the RM and applies to the RW for resource availability monitoring and early warning. AW registers resources and alerts respectively, and then NM starts each task, and the task enters the running state, in which AW performs resource monitoring and alerting according to the rules defined by formulas (1-1) to (1-13), and informs RW if resources are insufficient. And send out early warning information, when the task is completed normally, release resources and close the alarm, and finally end and exit the process.

FIG. 6 is a schematic diagram of the workflow of the resource availability early warning system according to the embodiment of the present invention. As shown in FIG. 6 , the workflow of the resource availability early warning system according to the embodiment of the present invention includes the following steps:

Step 601: the user submits the application, applies for resources to the RM, and requests the RW to start the monitoring and early warning of resource availability;

In this step, the user submits the application program in the JobClient mode, applies for resources to the RM, and requests the RW to start the monitoring and early warning of resource availability.

Step 602: RM allocates Container and communicates with NM to request to start AW, RW communicates with NM to request to start AW;

In this step, the RM allocates the first container Container to the application after accepting the request, communicates with the corresponding NM, and issues a command to start the AM in the Container. At the same time, the RW also accepts the request to communicate with the NM, and requests to start the application task monitoring AW in the Container in order to monitor and warn each Container of resource availability.

Step 603: AM registers with RM and receives resources, AW registers and reports resources with RW;

In this step, AM registers itself with RM to apply for CPU, memory and other resources for each task. At the same time, AW registers itself with RW and reports various information such as the type and size of resources already received by each task, so that RW can classify them according to them. and adopt different processing strategies.

Step 604: the AM requests the NM to start the task, and the NM notifies the AW to monitor and warn the resource availability;

In this step, the AM requests the NM to start and execute each task, and the NM notifies the AW to monitor and warn the resource availability. Then, the NM starts each task, and all tasks enter the official running state.

Step 605: AM manages and monitors task status, and AW monitors and warns resource availability;

In this step, AM is responsible for management and status monitoring of all tasks during operation, and AW monitors and warns of resource availability.

Step 606: AW judges whether to carry out an early warning, when a serious warning needs to be carried out, step 607 is performed, and when no warning is required or a non-serious warning needs to be carried out, step 608 is carried out;

In this step, the progress of the task and the availability of resources in the Container are recorded within the specified time series, and the feedback learning is continuously repeated according to formulas (1-1) to (1-13) to obtain the optimal model with the minimum error and predict Estimate the amount of resources required in the next cycle, and issue an alarm when the availability of the remaining resources is found to be lower than the threshold of the estimated required resources. When AM finds an irreversible task in a fatal warning state or has failed, it re-applies for resources and restarts the task.

Step 607: issue a serious warning, and return to step 602;

Step 608: The RW implements global resource availability monitoring and early warning, and notifies the user of the warning information.

In this step, the RW synthesizes the resource availability and alarm information reported from the AW to implement global resource availability monitoring and early warning, and push notifications of the alarm information to the user.

Step 609 : the application ends, the AM logs out from the RM to release resources, the AW closes the alarm from the RW, and the process ends.

In this step, after the normal execution of the application program is completed, the AM logs out of the RM to release resources, and the corresponding AW also closes its own alarms from the RW, and finally ends the process.

The resource availability early warning method, device and system according to the embodiments of the present invention take the resource availability early warning mechanism based on the time series dynamic feedback learning model as the core method, and add a function of resource availability monitoring early warning mechanism to the original Hadoop architecture Modules, such as RW and AW, respectively implement global resource availability monitoring and early warning and local container resource availability monitoring and early warning, and timely notify users of low resource availability alarms according to preset rules for early processing and monitoring. resource allocation. In the specific implementation process, a dynamic feedback learning model based on time series is first established, and the progress of all parallel tasks and the increments of resource consumption within each time series increment are dynamically and real-time fed back to AW. AW follows preset rules. Learn the potential rules and associations in this time period and generate the increment of consumed resources corresponding to the increment of task progress in the next time period, and then compare the actual task progress with the increment value of consumed resources, and continuously and repeatedly dynamically adjust the weight to decrease Error optimization model, take the optimal weight with the smallest error in the current time series from a series of weights to form a model, and then estimate the resource size required to complete the remaining tasks on the basis of this model, if the required resource value exceeds the remaining value The threshold of available resources is notified by AW to RW, and RW sends out different levels of early warning according to the different priorities, importance and dependencies of tasks.

The method, device and system for early warning of resource availability according to the embodiments of the present invention make up for the deficiencies that the existing Hadoop does not have relevant modules to provide users with data and factual basis for allocating resources, and that resources and task states can only be monitored in real time and cannot be predicted in advance Defects of insufficient resources, avoid waste of resources, reduce capital costs, reduce the work pressure of maintenance personnel, and provide decision support for operation and maintenance. This solution has high practicability in practical applications.

The realization function of each processing module in the resource availability early warning apparatus shown in FIG. 4 can be understood by referring to the relevant description of the foregoing resource availability early warning method. Those skilled in the art should understand that the function of each processing module in the resource availability early warning device shown in FIG. 4 can be realized by a program running on the processor, or can be realized by a specific logic circuit, for example, a central processing CPU (CPU), Microprocessor (MPU), Digital Signal Processor (DSP), or Field Programmable Gate Array (FPGA).

In the several embodiments provided by the present invention, it should be understood that the disclosed method and apparatus may be implemented in other manners. The apparatus embodiments described above are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods, for example, multiple modules or components may be combined, or Can be integrated into another system, or some features can be ignored, or not implemented. In addition, the communication connection between the components shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or modules, and may be electrical, mechanical or other forms.

The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules; Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional module in each embodiment of the present invention may be all integrated into one processing module, or each module may be separately used as a module, or two or more modules may be integrated into one module; the above integration The module can be realized in the form of hardware, or it can be realized in the form of hardware plus software function module.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by program instructions related to hardware, the aforementioned program can be stored in a computer-readable storage medium, and when the program is executed, the execution includes: The steps of the above method embodiments; and the aforementioned storage medium includes: a removable storage device, a read-only memory (ROM, Read-Only Memory), a magnetic disk or an optical disk and other media that can store program codes.

Alternatively, if the above-mentioned integrated modules in the embodiments of the present invention are implemented in the form of software function modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention may be embodied in the form of software products in essence or the parts that make contributions to the prior art. The computer software products are stored in a storage medium and include several instructions for A computer device (which may be a personal computer, a server, or a network device, etc.) is caused to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a removable storage device, a ROM, a magnetic disk, or an optical disk.

The resource availability early warning method and device described in the embodiments of the present invention only take the above embodiment as an example, but are not limited to this. Those of ordinary skill in the art should understand that it is still possible to perform the technical solutions described in the foregoing embodiments. Modification, or equivalent replacement of some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention.

Claims (8)

1. A method for early warning of resource availability, wherein the method comprises: Establish a resource usage estimation model; Estimate the resource usage in the next time period by using the resource usage estimation model; When the estimated resource usage in the next cycle exceeds a preset threshold, issue a resource availability warning; Wherein, the establishment of a resource usage estimation model includes: Determine the resource consumption increment corresponding to the progress increment of the task in the next time period according to the progress of all parallel tasks in each time period and the increment of consumed resources, as well as the time period regularity and correlation; Comparing the consuming resource increments corresponding to the determined task progress increments in the next time period multiple times with the consuming resource increments corresponding to the actual task progress increments, dynamically adjusting the weights to reduce errors; Select the weight corresponding to the minimum error to establish a resource usage estimation model. 2. method according to claim 1 is characterized in that, described by described resource usage estimation model, the resource usage estimation of next time period includes but not limited to: by described resource usage estimation model , to estimate the CPU resource usage and memory resource usage for the next time period. 3. The method according to claim 1, wherein when the estimated resource usage of the next cycle exceeds the first threshold, the method comprises: the estimated resource usage of the next cycle exceeds the threshold of the remaining available resources . 4. The method according to claim 1 or 3, wherein when the estimated resource usage in the next cycle exceeds a preset threshold, issuing a resource availability early warning comprises: when the estimated resource usage exceeds a preset threshold. When the resource usage in the next cycle exceeds the preset threshold, different levels of resource availability warnings are issued according to the priority, importance and dependencies of the tasks. 5. A resource availability early warning device, characterized in that the device comprises: a model building module, a resource estimation module, and a resource early warning module, wherein, The model establishment module is used to establish a resource usage estimation model; The resource estimation module is used to estimate the resource usage in the next time period through the resource usage estimation model; The resource early warning module is configured to issue a resource availability early warning when the estimated resource usage in the next cycle exceeds a preset threshold; Wherein, the model building module is specifically configured to determine the resource consumption increment corresponding to the progress increment of the task in the next time period according to the progress of all parallel tasks in each time period and the increment of consumed resources, as well as the regularity of the time period and the relationship ; Comparing the consuming resource increments corresponding to the determined task progress increments in the next time period multiple times with the consuming resource increments corresponding to the actual task progress increments, dynamically adjusting the weights to reduce errors; Select the weight corresponding to the minimum error to establish a resource usage estimation model. 6. device according to claim 5, is characterized in that, described resource estimation module is specifically used for: by described resource usage estimation model, estimate CPU resource usage amount and memory resource usage amount of next time period . 7. The device according to claim 5, wherein when the resource early-warning module judges that the estimated resource usage in the next cycle exceeds the first threshold value, it comprises: the resource early-warning module judges that the estimated next cycle of resource usage exceeds the first threshold. Resource usage exceeds the threshold for remaining available resources. 8. The device according to claim 5 or 7, wherein the resource early warning module is specifically used for: when the estimated resource usage of the next cycle exceeds a preset threshold, according to the priority of the task, the degree of importance As well as dependencies, different levels of resource availability alerts are issued.

Images