CN117194019A - Method, device, equipment and medium for controlling system resources - Google Patents
Method, device, equipment and medium for controlling system resources Download PDFInfo
- Publication number
- CN117194019A CN117194019A CN202311118686.XA CN202311118686A CN117194019A CN 117194019 A CN117194019 A CN 117194019A CN 202311118686 A CN202311118686 A CN 202311118686A CN 117194019 A CN117194019 A CN 117194019A
- Authority
- CN
- China
- Prior art keywords
- system resources
- trend
- task
- resource
- allocation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 119
- 230000008569 process Effects 0.000 claims abstract description 63
- 238000013468 resource allocation Methods 0.000 claims abstract description 56
- 238000012544 monitoring process Methods 0.000 claims abstract description 28
- 238000004458 analytical method Methods 0.000 claims abstract description 16
- 230000015654 memory Effects 0.000 claims description 62
- 230000004044 response Effects 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 11
- 238000012549 training Methods 0.000 claims description 11
- 238000005516 engineering process Methods 0.000 claims description 9
- 238000010801 machine learning Methods 0.000 claims description 8
- 238000007726 management method Methods 0.000 claims description 8
- 238000004422 calculation algorithm Methods 0.000 claims description 7
- 238000007619 statistical method Methods 0.000 claims description 7
- 238000012731 temporal analysis Methods 0.000 claims description 7
- 238000000700 time series analysis Methods 0.000 claims description 7
- 238000013528 artificial neural network Methods 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 6
- 238000009499 grossing Methods 0.000 claims description 6
- 238000012417 linear regression Methods 0.000 claims description 6
- YHXISWVBGDMDLQ-UHFFFAOYSA-N moclobemide Chemical compound C1=CC(Cl)=CC=C1C(=O)NCCN1CCOCC1 YHXISWVBGDMDLQ-UHFFFAOYSA-N 0.000 claims description 6
- 230000000306 recurrent effect Effects 0.000 claims description 6
- 230000001932 seasonal effect Effects 0.000 claims description 6
- 238000003062 neural network model Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 5
- 239000002699 waste material Substances 0.000 abstract description 6
- 230000001276 controlling effect Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007787 long-term memory Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention provides a method, a device, equipment and a readable medium for controlling system resources, wherein the method comprises the following steps: collecting the use information of system resources every time a preset time passes, and storing the use information into an internal data structure; responding to the use information stored in the internal data structure to reach a preset amount, reading the use information and predicting the use trend of future system resources by using a corresponding analysis method; according to the predicted using trend of the system resources, the system resources are allocated; and monitoring the use condition of system resources of each process and task in real time, and adjusting the allocation of the system resources according to a preset system resource allocation strategy. By using the scheme of the invention, the system resources can be adjusted in real time according to the task requirements and the system state, the waste of the system resources can be reduced, and the performance of the task and the resource utilization rate can be improved.
Description
Technical Field
The present invention relates to the field of computers, and more particularly, to a method, apparatus, device, and readable medium for system resource control.
Background
With the development of computer technology, resource management and scheduling of various computing devices has become increasingly important. As a widely used operating system, the Linux system has a crucial effect on system performance and stability due to a resource management and scheduling mechanism. However, the existing Linux system resource control method and system still have defects in the aspect of dynamically adjusting resource allocation and load balancing, and cannot completely meet the modern computing demands. The method is usually fixed in terms of resource scheduling, cannot be adjusted in real time according to task demands and system states, and can cause resource waste or task performance reduction.
Disclosure of Invention
Accordingly, an object of an embodiment of the present invention is to provide a method, apparatus, device, and readable medium for controlling system resources, by using the technical solution of the present invention, system resources can be adjusted in real time according to task requirements and system states, waste of system resources can be reduced, and performance of tasks and resource utilization rate can be improved.
Based on the above object, an aspect of an embodiment of the present invention provides a method for controlling system resources, including the steps of:
collecting the use information of system resources every time a preset time passes, and storing the use information into an internal data structure;
responding to the use information stored in the internal data structure to reach a preset amount, reading the use information and predicting the use trend of future system resources by using a corresponding analysis method;
according to the predicted using trend of the system resources, the system resources are allocated;
and monitoring the use condition of system resources of each process and task in real time, and adjusting the allocation of the system resources according to a preset system resource allocation strategy.
According to one embodiment of the present invention, collecting usage information of system resources every time a preset time elapses, and storing the usage information into an internal data structure includes:
reading a/proc/stat file in the system, and analyzing the utilization rate of each CPU core;
acquiring performance indexes of hit rate and branch prediction accuracy of the CPU cache by using a perf tool;
acquiring detailed information of the use condition of the memory and the use condition of the exchange area by using a system tool;
reading a per proc/disks file in the system, and analyzing the I/O request and transmission time data of each disk device;
acquiring performance indexes of disk I/O, bandwidth and delay by using a preset type of tool;
reading the bandwidth use and data packet sending and receiving conditions of a network interface obtained by a proc/net/dev file in a system;
and monitoring the service condition of the network traffic and analyzing the network bandwidth occupation condition of each process.
According to one embodiment of the invention, reading the usage information and predicting usage trends of future system resources using corresponding analysis methods includes:
in response to predicting a trend in future use of system resources using a time series analysis method, constructing a first prediction model from historical data using an ARIMA model, and predicting a trend in use of resources over a period of time in the future;
predicting seasonal and trending resource usage data by utilizing a Holt-windows exponential smoothing method;
in response to predicting a trend of future system resources using a machine learning method, building a training environment of a second prediction model using a neural network model;
training a second predictive model using a supervised learning algorithm;
processing the sequence data and predicting the resource requirement by using a recurrent neural network or a long-short-term memory network;
in response to predicting a trend of future system resource usage using a statistical method, calculating a moving average of the resource usage data using a moving average method, and finding a smoothed trend;
and (5) using a linear regression method to find the linear trend of the resource usage data so as to predict the demand trend of future resources.
According to one embodiment of the present invention, allocating system resources according to a predicted trend of use of the system resources includes:
evaluating task priority according to task type, task emergency degree and task resource competing degree;
and allocating the system resources according to the task priority, the predicted use trend of the system resources and the resource use limit.
According to one embodiment of the invention, the real-time monitoring of the system resource usage of each process and task includes:
acquiring the CPU and memory use conditions of each process in real time, and monitoring the resource use information in the/proc and/sys file system through a file monitoring technology;
the use condition of the disk I/O of each process is monitored, and the network bandwidth occupation condition of each process is monitored by using an iftop tool or a nethogs tool.
According to one embodiment of the invention, the system resource allocation policy includes adjusting CPU allocation, adjusting memory allocation, adjusting disk I/O allocation, and adjusting network resource allocation.
According to one embodiment of the present invention, adjusting allocation of system resources according to a preset system resource allocation policy includes:
the CPU frequency and time slices of each process and task are adjusted according to a system resource allocation strategy, and the CPU utilization time and share are limited by utilizing a CPU subsystem in a CPU group;
using a memory subsystem in the cgroup to adjust the memory allocation of each process and task according to a system resource allocation strategy, wherein the memory allocation comprises memory limitation and memory priority;
the I/O scheduler is used for adjusting the priority of the disk I/O of each process and task according to the system resource allocation strategy, and the blkio subsystem in the cgroup is used for setting the limit of the I/O bandwidth;
and setting the network bandwidth limit and the priority of each process and task according to a system resource allocation strategy by using a flow control tool, and realizing hierarchical management of network resources by using a net_cls subsystem in the cgroup.
In another aspect of the embodiment of the present invention, there is also provided an apparatus for controlling system resources, including:
the system comprises an acquisition module, a storage module and a storage module, wherein the acquisition module is configured to collect the use information of system resources every time a preset time passes, and store the use information into an internal data structure;
the prediction module is configured to respond to the fact that the usage information stored in the internal data structure reaches a preset amount, read the usage information and predict the usage trend of future system resources by using a corresponding analysis method;
the allocation module is configured to allocate the system resources according to the predicted use trend of the system resources;
the adjusting module is configured to monitor the use condition of system resources of each process and task in real time and adjust the distribution of the system resources according to a preset system resource distribution strategy.
In another aspect of the embodiments of the present invention, there is also provided a computer apparatus including:
at least one processor; and
and a memory storing computer instructions executable on the processor, the instructions when executed by the processor performing the steps of any of the methods described above.
In another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of any of the methods described above.
The invention has the following beneficial technical effects: the method for controlling the system resources provided by the embodiment of the invention collects the use information of the system resources every time a preset time passes and stores the use information into an internal data structure; responding to the use information stored in the internal data structure to reach a preset amount, reading the use information and predicting the use trend of future system resources by using a corresponding analysis method; according to the predicted using trend of the system resources, the system resources are allocated; the system resource use condition of each process and task is monitored in real time, and the technical scheme of system resource allocation is adjusted according to a preset system resource allocation strategy, so that the system resource can be adjusted in real time according to the task requirement and the system state, the waste of the system resource can be reduced, and the performance of the task and the resource use rate can be improved.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are necessary for the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention and that other embodiments may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart diagram of a method of system resource control in accordance with one embodiment of the invention;
FIG. 2 is a schematic diagram of an apparatus for system resource control in accordance with one embodiment of the present invention;
FIG. 3 is a schematic diagram of a computer device according to one embodiment of the invention;
fig. 4 is a schematic diagram of a computer-readable storage medium according to one embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention will be described in further detail with reference to the accompanying drawings.
With the above object in view, in a first aspect, an embodiment of a method for controlling system resources is provided. Fig. 1 shows a schematic flow chart of the method.
As shown in fig. 1, the method may include the steps of:
s1, collecting the use information of the system resources every time a preset time passes, and storing the use information into an internal data structure.
And S2, responding to the use information stored in the internal data structure to reach a preset amount, reading the use information and predicting the use trend of future system resources by using a corresponding analysis method.
And S3, distributing the system resources according to the predicted using trend of the system resources.
S4, monitoring the use condition of system resources of each process and task in real time, and adjusting the allocation of the system resources according to a preset system resource allocation strategy.
By using the technical scheme of the invention, the system resources can be adjusted in real time according to the task requirements and the system state, the waste of the system resources can be reduced, and the performance of the task and the resource utilization rate can be improved.
In a preferred embodiment of the present invention, collecting usage information of system resources every time a preset time elapses, and storing the usage information into an internal data structure includes:
reading a/proc/stat file in the system, and analyzing the utilization rate of each CPU core;
acquiring performance indexes of hit rate and branch prediction accuracy of the CPU cache by using a perf tool;
acquiring detailed information of the use condition of the memory and the use condition of the exchange area by using a system tool;
reading a per proc/disks file in the system, and analyzing the I/O request and transmission time data of each disk device;
acquiring performance indexes of disk I/O, bandwidth and delay by using a preset type of tool;
reading the bandwidth use and data packet sending and receiving conditions of a network interface obtained by a proc/net/dev file in a system;
and monitoring the service condition of the network traffic and analyzing the network bandwidth occupation condition of each process. The collection of the usage information includes CPU usage collection, memory usage collection, disk I/O usage collection, and network I/O usage collection. CPU usage collection: and reading/proc/stat files, analyzing the utilization rate of each CPU core, and acquiring performance indexes such as CPU cache hit rate, branch prediction accuracy rate and the like by using a perf tool. b. And (3) collecting the memory use condition: and acquiring detailed information such as memory use, switch area use and the like by using a Linux tool such as vmstat or free. c. Disk I/O usage collection: reading/proc/disks files, analyzing data such as I/O requests, transmission time and the like of each disk device, and acquiring performance indexes such as disk I/O bandwidth, delay and the like by using an iostat or iostop tool. d. Network I/O usage collection: and reading/proc/net/dev files, obtaining bandwidth use of a network interface, sending and receiving data packets, monitoring network traffic use conditions by using tools such as iftop or nethogs, and analyzing network bandwidth occupation conditions of each process.
In a preferred embodiment of the invention, reading the usage information and predicting the usage trend of future system resources using the corresponding analysis method comprises:
in response to predicting a trend in future use of system resources using a time series analysis method, constructing a first prediction model from historical data using an ARIMA model, and predicting a trend in use of resources over a period of time in the future;
predicting seasonal and trending resource usage data by utilizing a Holt-windows exponential smoothing method;
in response to predicting a trend of future system resources using a machine learning method, building a training environment of a second prediction model using a neural network model;
training a second predictive model using a supervised learning algorithm;
processing the sequence data and predicting the resource requirement by using a recurrent neural network or a long-short-term memory network;
in response to predicting a trend of future system resource usage using a statistical method, calculating a moving average of the resource usage data using a moving average method, and finding a smoothed trend;
and (5) using a linear regression method to find the linear trend of the resource usage data so as to predict the demand trend of future resources. The corresponding analysis methods include a time series analysis method, a machine learning method and a statistical method, a. Time series analysis: and constructing a prediction model according to historical data by using an ARIMA model, predicting the resource use trend in a future period, and predicting the seasonal and trending resource use data by using a Holt-windows exponential smoothing method. b. The machine learning method comprises the following steps: a model training environment is built by using an existing machine learning framework and library (such as TensorFlow, pyTorch, scikit-learn and the like), a predictive model is trained by using a Support Vector Machine (SVM) or a supervised learning algorithm such as a random forest and the like, and sequence data is processed by using a Recurrent Neural Network (RNN) or a long and short term memory network (LSTM) to predict resource requirements. c. The statistical method comprises the following steps: and calculating a sliding average value of the resource usage data by using a moving average method, finding a smoothed trend, finding a linear trend of the resource usage data by using a linear regression method, and predicting future resource demands.
In a preferred embodiment of the present invention, allocating system resources according to the predicted trend of use of the system resources comprises:
evaluating task priority according to task type, task emergency degree and task resource competing degree;
and allocating the system resources according to the task priority, the predicted use trend of the system resources and the resource use limit. Firstly, task priority evaluation is carried out, different resource weights are allocated according to task types (such as compute intensive and I/O intensive), task emergency degrees (such as deadline and user grade) are considered, resource allocation priority is adjusted, and resource allocation is balanced by combining resource preemption degrees (such as CPU preemption rate and memory contention condition). And then generating a resource allocation strategy, maintaining a resource allocation sequence by using a priority queue or other data structures, and generating a corresponding cgroup configuration file according to a prediction result, task priority and resource use limit, so as to realize dynamic adjustment of resource allocation to adapt to the change of task demands and system environments.
In a preferred embodiment of the present invention, monitoring the system resource usage of each process and task in real time includes:
acquiring the CPU and memory use conditions of each process in real time, and monitoring the resource use information in the/proc and/sys file system through a file monitoring technology;
the use condition of the disk I/O of each process is monitored, and the network bandwidth occupation condition of each process is monitored by using an iftop tool or a nethogs tool. The CPU and memory use condition of each process is obtained in real time by using a ps or top tool, the resource use information in the file system of the proc and the sys is monitored by using an inotify or other file monitoring technology, the disk I/O use condition of each process is monitored by using an iotop tool, and the network bandwidth occupation condition of each process is monitored by using an ifop or nethogs tool.
In a preferred embodiment of the present invention, the system resource allocation policy includes adjusting CPU allocation, adjusting memory allocation, adjusting disk I/O allocation, and adjusting network resource allocation.
In a preferred embodiment of the present invention, adjusting allocation of system resources according to a preset system resource allocation policy includes:
the CPU frequency and time slices of each process and task are adjusted according to a system resource allocation strategy, and the CPU utilization time and share are limited by utilizing a CPU subsystem in a CPU group;
using a memory subsystem in the cgroup to adjust the memory allocation of each process and task according to a system resource allocation strategy, wherein the memory allocation comprises memory limitation and memory priority;
the I/O scheduler is used for adjusting the priority of the disk I/O of each process and task according to the system resource allocation strategy, and the blkio subsystem in the cgroup is used for setting the limit of the I/O bandwidth;
and setting the network bandwidth limit and the priority of each process and task according to a system resource allocation strategy by using a flow control tool, and realizing hierarchical management of network resources by using a net_cls subsystem in the cgroup. When CPU allocation is adjusted, a CFS scheduler provided by a Linux kernel is used for adjusting the CPU frequency and time slices of each process and task according to a resource allocation strategy, and CPU use time and share are limited by utilizing a CPU subsystem in a CPU. When the memory allocation is adjusted, a memory subsystem in the cgroup is used for adjusting the memory allocation of each process and task according to the resource allocation strategy, wherein the memory allocation comprises memory restriction, memory priority and the like. When the disk I/O allocation is adjusted, an I/O scheduler (such as a CFQ scheduler) is used, the disk I/O priority of each process and task is adjusted according to a resource allocation strategy, and the I/O bandwidth limit is set by utilizing a blkio subsystem in the cgroup. When the network resource allocation is regulated, a tc (flow control) tool is used, the network bandwidth limit and the priority of each process and task are set according to a resource allocation strategy, and the net_cls subsystem in the cgroup is utilized to realize hierarchical management of the network resource.
By using the technical scheme of the invention, the system resources can be adjusted in real time according to the task requirements and the system state, the waste of the system resources can be reduced, and the performance of the task and the resource utilization rate can be improved.
It should be noted that, it will be understood by those skilled in the art that all or part of the procedures in implementing the methods of the above embodiments may be implemented by a computer program to instruct related hardware, and the above program may be stored in a computer readable storage medium, and the program may include the procedures of the embodiments of the above methods when executed. Wherein the storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), or the like. The computer program embodiments described above may achieve the same or similar effects as any of the method embodiments described above.
Furthermore, the method disclosed according to the embodiment of the present invention may also be implemented as a computer program executed by a CPU, which may be stored in a computer-readable storage medium. When executed by a CPU, performs the functions defined above in the methods disclosed in the embodiments of the present invention.
With the above object in mind, in a second aspect of the embodiments of the present invention, an apparatus for controlling system resources is provided, as shown in fig. 2, an apparatus 200 includes:
the system comprises an acquisition module, a storage module and a storage module, wherein the acquisition module is configured to collect the use information of system resources every time a preset time passes, and store the use information into an internal data structure;
the prediction module is configured to respond to the fact that the usage information stored in the internal data structure reaches a preset amount, read the usage information and predict the usage trend of future system resources by using a corresponding analysis method;
the allocation module is configured to allocate the system resources according to the predicted use trend of the system resources;
the adjusting module is configured to monitor the use condition of system resources of each process and task in real time and adjust the distribution of the system resources according to a preset system resource distribution strategy.
Based on the above object, a third aspect of the embodiments of the present invention proposes a computer device. Fig. 3 is a schematic diagram of an embodiment of a computer device provided by the present invention. As shown in fig. 3, an embodiment of the present invention includes the following means: at least one processor 21; and a memory 22, the memory 22 storing computer instructions 23 executable on the processor, the instructions when executed by the processor performing the method of:
collecting the use information of system resources every time a preset time passes, and storing the use information into an internal data structure;
responding to the use information stored in the internal data structure to reach a preset amount, reading the use information and predicting the use trend of future system resources by using a corresponding analysis method;
according to the predicted using trend of the system resources, the system resources are allocated;
and monitoring the use condition of system resources of each process and task in real time, and adjusting the allocation of the system resources according to a preset system resource allocation strategy.
In a preferred embodiment of the present invention, collecting usage information of system resources every time a preset time elapses, and storing the usage information into an internal data structure includes:
reading a/proc/stat file in the system, and analyzing the utilization rate of each CPU core;
acquiring performance indexes of hit rate and branch prediction accuracy of the CPU cache by using a perf tool;
acquiring detailed information of the use condition of the memory and the use condition of the exchange area by using a system tool;
reading a per proc/disks file in the system, and analyzing the I/O request and transmission time data of each disk device;
acquiring performance indexes of disk I/O, bandwidth and delay by using a preset type of tool;
reading the bandwidth use and data packet sending and receiving conditions of a network interface obtained by a proc/net/dev file in a system;
and monitoring the service condition of the network traffic and analyzing the network bandwidth occupation condition of each process.
In a preferred embodiment of the invention, reading the usage information and predicting the usage trend of future system resources using the corresponding analysis method comprises:
in response to predicting a trend in future use of system resources using a time series analysis method, constructing a first prediction model from historical data using an ARIMA model, and predicting a trend in use of resources over a period of time in the future;
predicting seasonal and trending resource usage data by utilizing a Holt-windows exponential smoothing method;
in response to predicting a trend of future system resources using a machine learning method, building a training environment of a second prediction model using a neural network model;
training a second predictive model using a supervised learning algorithm;
processing the sequence data and predicting the resource requirement by using a recurrent neural network or a long-short-term memory network;
in response to predicting a trend of future system resource usage using a statistical method, calculating a moving average of the resource usage data using a moving average method, and finding a smoothed trend;
and (5) using a linear regression method to find the linear trend of the resource usage data so as to predict the demand trend of future resources.
In a preferred embodiment of the present invention, allocating system resources according to the predicted trend of use of the system resources comprises:
evaluating task priority according to task type, task emergency degree and task resource competing degree;
and allocating the system resources according to the task priority, the predicted use trend of the system resources and the resource use limit.
In a preferred embodiment of the present invention, monitoring the system resource usage of each process and task in real time includes:
acquiring the CPU and memory use conditions of each process in real time, and monitoring the resource use information in the/proc and/sys file system through a file monitoring technology;
the use condition of the disk I/O of each process is monitored, and the network bandwidth occupation condition of each process is monitored by using an iftop tool or a nethogs tool.
In a preferred embodiment of the present invention, the system resource allocation policy includes adjusting CPU allocation, adjusting memory allocation, adjusting disk I/O allocation, and adjusting network resource allocation.
In a preferred embodiment of the present invention, adjusting allocation of system resources according to a preset system resource allocation policy includes:
the CPU frequency and time slices of each process and task are adjusted according to a system resource allocation strategy, and the CPU utilization time and share are limited by utilizing a CPU subsystem in a CPU group;
using a memory subsystem in the cgroup to adjust the memory allocation of each process and task according to a system resource allocation strategy, wherein the memory allocation comprises memory limitation and memory priority;
the I/O scheduler is used for adjusting the priority of the disk I/O of each process and task according to the system resource allocation strategy, and the blkio subsystem in the cgroup is used for setting the limit of the I/O bandwidth;
and setting the network bandwidth limit and the priority of each process and task according to a system resource allocation strategy by using a flow control tool, and realizing hierarchical management of network resources by using a net_cls subsystem in the cgroup.
Based on the above object, a fourth aspect of the embodiments of the present invention proposes a computer-readable storage medium. Fig. 4 is a schematic diagram of an embodiment of a computer-readable storage medium provided by the present invention. As shown in fig. 4, the computer-readable storage medium 31 stores a computer program 32 that, when executed by a processor, performs the following method:
collecting the use information of system resources every time a preset time passes, and storing the use information into an internal data structure;
responding to the use information stored in the internal data structure to reach a preset amount, reading the use information and predicting the use trend of future system resources by using a corresponding analysis method;
according to the predicted using trend of the system resources, the system resources are allocated;
and monitoring the use condition of system resources of each process and task in real time, and adjusting the allocation of the system resources according to a preset system resource allocation strategy.
In a preferred embodiment of the present invention, collecting usage information of system resources every time a preset time elapses, and storing the usage information into an internal data structure includes:
reading a/proc/stat file in the system, and analyzing the utilization rate of each CPU core;
acquiring performance indexes of hit rate and branch prediction accuracy of the CPU cache by using a perf tool;
acquiring detailed information of the use condition of the memory and the use condition of the exchange area by using a system tool;
reading a per proc/disks file in the system, and analyzing the I/O request and transmission time data of each disk device;
acquiring performance indexes of disk I/O, bandwidth and delay by using a preset type of tool;
reading the bandwidth use and data packet sending and receiving conditions of a network interface obtained by a proc/net/dev file in a system;
and monitoring the service condition of the network traffic and analyzing the network bandwidth occupation condition of each process.
In a preferred embodiment of the invention, reading the usage information and predicting the usage trend of future system resources using the corresponding analysis method comprises:
in response to predicting a trend in future use of system resources using a time series analysis method, constructing a first prediction model from historical data using an ARIMA model, and predicting a trend in use of resources over a period of time in the future;
predicting seasonal and trending resource usage data by utilizing a Holt-windows exponential smoothing method;
in response to predicting a trend of future system resources using a machine learning method, building a training environment of a second prediction model using a neural network model;
training a second predictive model using a supervised learning algorithm;
processing the sequence data and predicting the resource requirement by using a recurrent neural network or a long-short-term memory network;
in response to predicting a trend of future system resource usage using a statistical method, calculating a moving average of the resource usage data using a moving average method, and finding a smoothed trend;
and (5) using a linear regression method to find the linear trend of the resource usage data so as to predict the demand trend of future resources.
In a preferred embodiment of the present invention, allocating system resources according to the predicted trend of use of the system resources comprises:
evaluating task priority according to task type, task emergency degree and task resource competing degree;
and allocating the system resources according to the task priority, the predicted use trend of the system resources and the resource use limit.
In a preferred embodiment of the present invention, monitoring the system resource usage of each process and task in real time includes:
acquiring the CPU and memory use conditions of each process in real time, and monitoring the resource use information in the/proc and/sys file system through a file monitoring technology;
the use condition of the disk I/O of each process is monitored, and the network bandwidth occupation condition of each process is monitored by using an iftop tool or a nethogs tool.
In a preferred embodiment of the present invention, the system resource allocation policy includes adjusting CPU allocation, adjusting memory allocation, adjusting disk I/O allocation, and adjusting network resource allocation.
In a preferred embodiment of the present invention, adjusting allocation of system resources according to a preset system resource allocation policy includes:
the CPU frequency and time slices of each process and task are adjusted according to a system resource allocation strategy, and the CPU utilization time and share are limited by utilizing a CPU subsystem in a CPU group;
using a memory subsystem in the cgroup to adjust the memory allocation of each process and task according to a system resource allocation strategy, wherein the memory allocation comprises memory limitation and memory priority;
the I/O scheduler is used for adjusting the priority of the disk I/O of each process and task according to the system resource allocation strategy, and the blkio subsystem in the cgroup is used for setting the limit of the I/O bandwidth;
and setting the network bandwidth limit and the priority of each process and task according to a system resource allocation strategy by using a flow control tool, and realizing hierarchical management of network resources by using a net_cls subsystem in the cgroup.
Furthermore, the method disclosed according to the embodiment of the present invention may also be implemented as a computer program executed by a processor, which may be stored in a computer-readable storage medium. The above-described functions defined in the methods disclosed in the embodiments of the present invention are performed when the computer program is executed by a processor.
Furthermore, the above-described method steps and system units may also be implemented using a controller and a computer-readable storage medium storing a computer program for causing the controller to implement the above-described steps or unit functions.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
In one or more exemplary designs, the functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one location to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a general purpose or special purpose computer or general purpose or special purpose processor. Further, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, digital Versatile Disc (DVD), floppy disk, blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that as used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The foregoing embodiment of the present invention has been disclosed with reference to the number of embodiments for the purpose of description only, and does not represent the advantages or disadvantages of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, and the program may be stored in a computer readable storage medium, where the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will appreciate that: the above discussion of any embodiment is merely exemplary and is not intended to imply that the scope of the disclosure of embodiments of the invention, including the claims, is limited to such examples; combinations of features of the above embodiments or in different embodiments are also possible within the idea of an embodiment of the invention, and many other variations of the different aspects of the embodiments of the invention as described above exist, which are not provided in detail for the sake of brevity. Therefore, any omission, modification, equivalent replacement, improvement, etc. of the embodiments should be included in the protection scope of the embodiments of the present invention.
Claims (10)
1. A method of system resource control, comprising the steps of:
collecting the use information of system resources every time a preset time passes, and storing the use information into an internal data structure;
responding to the use information stored in the internal data structure to reach a preset amount, reading the use information and predicting the use trend of future system resources by using a corresponding analysis method;
according to the predicted using trend of the system resources, the system resources are allocated;
and monitoring the use condition of system resources of each process and task in real time, and adjusting the allocation of the system resources according to a preset system resource allocation strategy.
2. The method of claim 1, wherein collecting usage information for system resources and storing the usage information in the internal data structure every time a preset time elapses comprises:
reading a/proc/stat file in the system, and analyzing the utilization rate of each CPU core;
acquiring performance indexes of hit rate and branch prediction accuracy of the CPU cache by using a perf tool;
acquiring detailed information of the use condition of the memory and the use condition of the exchange area by using a system tool;
reading a per proc/disks file in the system, and analyzing the I/O request and transmission time data of each disk device;
acquiring performance indexes of disk I/O, bandwidth and delay by using a preset type of tool;
reading the bandwidth use and data packet sending and receiving conditions of a network interface obtained by a proc/net/dev file in a system;
and monitoring the service condition of the network traffic and analyzing the network bandwidth occupation condition of each process.
3. The method of claim 1, wherein reading the usage information and predicting a trend in future usage of the system resource using the corresponding analysis method comprises:
in response to predicting a trend in future use of system resources using a time series analysis method, constructing a first prediction model from historical data using an ARIMA model, and predicting a trend in use of resources over a period of time in the future;
predicting seasonal and trending resource usage data by utilizing a Holt-windows exponential smoothing method;
in response to predicting a trend of future system resources using a machine learning method, building a training environment of a second prediction model using a neural network model;
training a second predictive model using a supervised learning algorithm;
processing the sequence data and predicting the resource requirement by using a recurrent neural network or a long-short-term memory network;
in response to predicting a trend of future system resource usage using a statistical method, calculating a moving average of the resource usage data using a moving average method, and finding a smoothed trend;
and (5) using a linear regression method to find the linear trend of the resource usage data so as to predict the demand trend of future resources.
4. The method of claim 1, wherein allocating system resources based on the predicted trend of use of system resources comprises:
evaluating task priority according to task type, task emergency degree and task resource competing degree;
and allocating the system resources according to the task priority, the predicted use trend of the system resources and the resource use limit.
5. The method of claim 1, wherein monitoring system resource usage of individual processes and tasks in real time comprises:
acquiring the CPU and memory use conditions of each process in real time, and monitoring the resource use information in the/proc and/sys file system through a file monitoring technology;
the use condition of the disk I/O of each process is monitored, and the network bandwidth occupation condition of each process is monitored by using an iftop tool or a nethogs tool.
6. The method of claim 1, wherein the system resource allocation policy comprises adjusting CPU allocation, adjusting memory allocation, adjusting disk I/O allocation, and adjusting network resource allocation.
7. The method of claim 6, wherein adjusting allocation of system resources according to a pre-set system resource allocation policy comprises:
the CPU frequency and time slices of each process and task are adjusted according to a system resource allocation strategy, and the CPU utilization time and share are limited by utilizing a CPU subsystem in a CPU group;
using a memory subsystem in the cgroup to adjust the memory allocation of each process and task according to a system resource allocation strategy, wherein the memory allocation comprises memory limitation and memory priority;
the I/O scheduler is used for adjusting the priority of the disk I/O of each process and task according to the system resource allocation strategy, and the blkio subsystem in the cgroup is used for setting the limit of the I/O bandwidth;
and setting the network bandwidth limit and the priority of each process and task according to a system resource allocation strategy by using a flow control tool, and realizing hierarchical management of network resources by using a net_cls subsystem in the cgroup.
8. An apparatus for system resource control, the apparatus comprising:
the system comprises an acquisition module, a storage module and a storage module, wherein the acquisition module is configured to collect the use information of system resources every time a preset time passes, and store the use information into an internal data structure;
the prediction module is configured to read the use information and predict the use trend of future system resources by using a corresponding analysis method in response to the use information stored in the internal data structure reaching a preset amount;
the allocation module is configured to allocate the system resources according to the predicted using trend of the system resources;
the system resource allocation module is configured to monitor the system resource use condition of each process and task in real time and adjust the allocation of the system resources according to a preset system resource allocation strategy.
9. A computer device, comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, which when executed by the processor, perform the steps of the method of any one of claims 1-7.
10. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311118686.XA CN117194019A (en) | 2023-08-31 | 2023-08-31 | Method, device, equipment and medium for controlling system resources |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311118686.XA CN117194019A (en) | 2023-08-31 | 2023-08-31 | Method, device, equipment and medium for controlling system resources |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117194019A true CN117194019A (en) | 2023-12-08 |
Family
ID=88986269
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311118686.XA Pending CN117194019A (en) | 2023-08-31 | 2023-08-31 | Method, device, equipment and medium for controlling system resources |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117194019A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118195837A (en) * | 2024-05-15 | 2024-06-14 | 巨洋神州科技集团有限公司 | Energy monitoring method, device, equipment and medium based on micro-grid dual-carbon platform |
-
2023
- 2023-08-31 CN CN202311118686.XA patent/CN117194019A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118195837A (en) * | 2024-05-15 | 2024-06-14 | 巨洋神州科技集团有限公司 | Energy monitoring method, device, equipment and medium based on micro-grid dual-carbon platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Bunyakitanon et al. | End-to-end performance-based autonomous VNF placement with adopted reinforcement learning | |
| CN106776005B (en) | Resource management system and method for containerized application | |
| US8352951B2 (en) | Method and apparatus for utility-based dynamic resource allocation in a distributed computing system | |
| US7640231B2 (en) | Approach based on self-evolving models for performance guarantees in a shared storage system | |
| Khorsand et al. | A self‐learning fuzzy approach for proactive resource provisioning in cloud environment | |
| Minarolli et al. | Tackling uncertainty in long-term predictions for host overload and underload detection in cloud computing | |
| CN107925612A (en) | Network monitoring system, network monitoring method and program | |
| Bi et al. | SLA-based optimisation of virtualised resource for multi-tier web applications in cloud data centres | |
| US11579933B2 (en) | Method for establishing system resource prediction and resource management model through multi-layer correlations | |
| CN112685170A (en) | Dynamic optimization of backup strategies | |
| US11757790B2 (en) | Method and server for adjusting allocation of computing resources to plurality of virtualized network functions (VNFs) | |
| CN111752706B (en) | Resource allocation method, device and storage medium | |
| CN117194019A (en) | Method, device, equipment and medium for controlling system resources | |
| Mohammadi Bahram Abadi et al. | Self-adaptive architecture for virtual machines consolidation based on probabilistic model evaluation of data centers in Cloud computing | |
| HoseinyFarahabady et al. | Q-flink: A qos-aware controller for apache flink | |
| Faraji‐Mehmandar et al. | A proactive fog service provisioning framework for Internet of Things applications: An autonomic approach | |
| CN116467082A (en) | Big data-based resource allocation method and system | |
| CN113641445B (en) | Cloud resource self-adaptive configuration method and system based on depth deterministic strategy | |
| Lu et al. | InSTechAH: Cost-effectively autoscaling smart computing hadoop cluster in private cloud | |
| Hirashima et al. | Proactive-reactive auto-scaling mechanism for unpredictable load change | |
| Leena Sri et al. | An empirical model of adaptive cloud resource provisioning with speculation | |
| Surya et al. | Prediction of resource contention in cloud using second order Markov model | |
| Faraji-Mehmandar et al. | A self-learning approach for proactive resource and service provisioning in fog environment | |
| Cheng et al. | AIOC2: A deep Q-learning approach to autonomic I/O congestion control in Lustre | |
| CN115913967A (en) | Micro-service elastic scaling method based on resource demand prediction in cloud environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |