CN102662762A - Method for effectively controlling use of memory resource of fat node - Google Patents
Method for effectively controlling use of memory resource of fat node Download PDFInfo
- Publication number
- CN102662762A CN102662762A CN2012100887904A CN201210088790A CN102662762A CN 102662762 A CN102662762 A CN 102662762A CN 2012100887904 A CN2012100887904 A CN 2012100887904A CN 201210088790 A CN201210088790 A CN 201210088790A CN 102662762 A CN102662762 A CN 102662762A
- Authority
- CN
- China
- Prior art keywords
- memory
- fat node
- node
- operating position
- resource
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention provides a method for effectively controlling use of a memory resource of a fat node. The method comprises the following steps: timely monitoring the using condition of the memory resource of the fat node by use of a tool supplied by a Linux system; and comparing monitoring data with a given threshold value by use of a closed-loop controlled model, wherein if the memory utilization ratio exceeds the given threshold value, a progress control mode is used so as to prevent fat node crash caused by excessive utilization of memory, and therefore the waste of a large quantity of computing resources is avoided. By the method, the reasonable and effective utilization of the fat node resource is effectively ensured and various control strategy adjustments can be carried out according the actual application conditions of users, and therefore the complexity of the users in using the system resources is reduced simultaneously when the resources are utilized maximally and then the system is enabled to be convenient, stable and efficient in operation.
Description
Technical field
The present invention relates to computer operating system management and optimization field, be specifically related to a kind of technology of under the (SuSE) Linux OS of multi-user, multitask, the internal memory resources effective being monitored, being managed and optimize.
Background technology
Industry and scientific research field are more and more for the demand of large-scale calculations at present; And such computation requirement often need have a large amount of CPU computation core numbers, memory source, storage space; Carry out on the computing node of express network communication; This just makes also that fat node is increasing and is used by users, becomes the main force of computing field.
As a large-scale computing node, normally used is the (SuSE) Linux OS of multi-user, multitask.Because to the requirement of operation efficiency, the user does not hope that program uses the swap subregion, thus the swap subregion often the actual physics internal memory of relative system is less.This also just makes and is being difficult to carry out the Memory Exchange between large-scale physical memory and the swap on the fat node; Therefore the needed memory size of process that requires to move in the system will guarantee in the physical memory size as far as possible; In case surpass the physical memory actual size; The machine of delaying that just probably causes total system, ongoing all computings are all fallen short of success for lack of final effort on feasible this fat node.
And for a user in the enterprising row operation of fat node; General being difficult to has estimation very accurately to the program of oneself operation in the needed actual memory quantity of calculating of each different phase, and the operation program of also inconvenient simultaneously monitoring constantly oneself is to the operating position of internal memory.And use for a plurality of users, on the fat node of a plurality of task runs, thereby how to avoid effectively that to cause the machine of delaying of total system be the subject matter that the present invention solves because the program of operation surpasses the actual physics internal memory to the demand of internal memory.
Summary of the invention
The objective of the invention is to effectively system memory resource monitored and managed; To prevent owing to internal memory uses the excessive fat node machine of delaying that causes; Thereby avoid causing the waste of great amount of calculation resource; Reduce the complexity that the user uses system resource, make that system is more convenient, more stable, operation more efficiently.
The objective of the invention is to realize by following mode; Instrument through using linux system itself to provide carries out the timing monitoring to the memory source operating position of fat node; Compare through gathering memory source operating position and preset threshold, according to the process control strategy system process is adjusted, and gather adjusted internal memory operating position once more and threshold value compares according to comparative result; Form the ring-type control system of a sealing; Make total system be controlled at stable equilibrium state, when memory usage had surpassed setting threshold, just the mode through process control was controlled at the internal memory that uses within the threshold range; The concrete steps that the present invention realizes are following:
1) according to the physical memory and the swap size of fat node, suitable threshold is set;
2) obtain current internal memory of system and swap operating position;
3) current Installed System Memory and swap operating position and threshold value are compared, if then withdraw from, if then got into for the 4th step greater than threshold range less than threshold range;
4) utilize linux system order ps to extract the memory usage that current system moving and be equal to or greater than domestic consumer's process of 0.1%, comprising ID number of process, user, cpu utilization rate, memory usage, actual run time, cpu working time, process status, fill order etc.;
5) choose target process according to the process control strategy, the process control strategy of supporting at present comprises 3 kinds:
1. choose the maximum process of current committed memory resource;
2. choose current cpu working time of minimum process (promptly using the minimum process of CPU computational resource);
3. according to the time of operation submission, choose the process of last submit job;
6) the target process kill that selects is fallen;
7) outputting log file information tells the operation of corresponding its operation of user to be stopped, and the time that stops, the information that the process that stops is relevant with process.
A control flow completion of all domestic consumer's operation processes on the fat node being used memory source so far.This process realizes with the form of Shell script, and the plan target through add-on system is regularly carried out this script, thereby forms a closed-loop system of memory source being used control.
The invention has the beneficial effects as follows: the instrument that utilizes linux system itself to provide carries out the timing monitoring to the memory source operating position of fat node; And adopt a kind of model of closed-loop control that monitor data and preset threshold are compared; When memory usage has surpassed preset threshold just the mode through process control to prevent since internal memory use and excessively cause the fat node machine of delaying, thereby avoid causing the waste of great amount of calculation resource.In the resource maximum using, reduced the complexity that the user uses system resource, made that system is more convenient, more stable, operation more efficiently.
Description of drawings
Accompanying drawing 1 is the closed-loop control model of the fat node memory resource use of effectively control;
Accompanying drawing 2 is the script realization flow figure of the fat node memory resource use of effectively control.
Embodiment
In order to make the object of the invention, technical scheme more clear, be elaborated to the committed step among the present invention below in conjunction with accompanying drawing.
A kind of model of closed-loop control is adopted in this invention; Shown in accompanying drawing 1; The memory source operating position of the crontab plan target instrument timing acquiring system that carries through linux system, and compare with preset threshold, compared result is controlled through the process control module.The input of this system is the variation to the internal memory resource occupation of preset threshold and process; Output is the memory size of the actual use of system; Feedback is to the data acquisition of Installed System Memory operating position and conversion, and the process control module guarantees that the output of system is stabilized within the threshold range.
The present invention realizes through the Shell script that to effective control that memory source on the fat node uses concrete realization flow figure is shown in accompanying drawing 2.
Obtain the current memory usage information of system through following order,
usedmem=`free?|?awk?'$1~"-/+"?{print?$3}'`
usedswap=`free?|?awk?'$1~"Swap"?{print?$3}'`
Obtain system's current process information through following order:
ps-eo?user,pid,pcpu,pmem,stat,start_time,cputime,etime,args?|?awk?'($1!="root"&&$1!="USER"&&$4>0.0)?{print?$0}'
To the start_time that gets access to, cputime, it is unit that etime need convert to the second time of carrying out, through realizing with minor function,
function?timetos
{
time=$1
dayflag=`echo?"$time"?|?awk?-F-?'{print?NF}'`
if?[?"$dayflag"?-eq?2?];?then
daytos=`echo?"$time"?|?awk?-F-?'{print?$1*24*3600}'`
tmptime=`echo?"$time"?|?awk?-F-?'{print?$2}'`
else
daytos=0
tmptime=`echo?"$time"?|?awk?-F-?'{print?$1}'`
fi
timeflag=`echo?"$tmptime"?|?awk?-F:?'{print?NF}'`
if?[?"$timeflag"?-eq?3?];?then
htos=`echo?"$tmptime"?|?awk?-F:?'{print?$1*3600}'`
mtos=`echo?"$tmptime"?|?awk?-F:?'{print?$2*60}'`
stos=`echo?"$tmptime"?|?awk?-F:?'{print?$3}'`
elif?[?"$timeflag"?-eq?2?];?then
htos=0
mtos=`echo?"$tmptime"?|?awk?-F:?'{print?$1*60}'`
stos=`echo?"$tmptime"?|?awk?-F:?'{print?$2}'`
fi
result=$((10#$daytos+10#$htos+10#$mtos+10#$stos))
echo?"$result"
}
Stop target process through following order,
kill?-9?"$pid1"
Freemem.sh adds into plan target with this script, and make this script regularly carry out the internal memory operating position and detect,
crontab?-e
0-59 * * * * sh/var/log/freemem/freemem.sh 3 # take the third strategy, and per minute detects once
Make regular check on the daily record file, then can notify this user, export as follows in the journal file with the daily record form if any the process that is stopped,
Wed Apr 14 20:24:01 AST 2010 # (cutting off the time of this process)
Kill?the?process?by?the?option?3,?please?tell?the?user?liujc?!
liujc 22457?99.9?93.5?R 09:17?11:05:35 11:06:05?velveth_de
The instrument that method among the present invention utilizes linux system itself to provide carries out the timing monitoring to the memory source operating position of computing node; And adopt a kind of model of closed-loop control that monitor data and preset threshold are compared; When memory usage has surpassed preset threshold just the mode through process control to prevent since internal memory use and excessively cause the fat node machine of delaying, thereby avoid causing the waste of great amount of calculation resource.In the resource maximum using, reduced the complexity that the user uses system resource, made that system is more convenient, more stable, operation more efficiently.This method is of universal significance for the (SuSE) Linux OS that uses multi-user, multitask, but especially the server architecture effect to fat node type is more remarkable.
Except that the described technical characterictic of instructions, be the known technology of those skilled in the art.
Claims (1)
1. effectively control the method that fat node memory resource is used for one kind; It is characterized in that the memory source operating position of fat node being carried out the timing monitoring through the instrument that uses linux system itself to provide; Compare through gathering memory source operating position and preset threshold, according to the process control strategy system process is adjusted, and gather adjusted internal memory operating position once more and threshold value compares according to comparative result; Form the ring-type control system of a sealing; Make total system be controlled at stable equilibrium state, when memory usage had surpassed setting threshold, just the mode through process control was controlled at the internal memory that uses within the threshold range;
Concrete steps are following:
1) according to the physical memory and the swap size of fat node, suitable threshold is set;
2) obtain current internal memory of system and swap operating position;
3) current Installed System Memory and swap operating position and threshold value are compared, if then withdraw from, if then got into for the 4th step greater than threshold range less than threshold range;
4) utilize linux system order ps to extract the memory usage that current system moving and be equal to or greater than domestic consumer's process of 0.1%, comprising ID number of process, user, cpu utilization rate, memory usage, actual run time, cpu working time, process status, fill order;
5) choose target process according to the process control strategy, the process control strategy of supporting at present comprises 3 kinds:
(1) chooses the maximum process of current committed memory resource;
(2) choose current cpu working time of minimum process, promptly use the minimum process of CPU computational resource;
(3), choose the process of last submit job according to the time of operation submission;
6) the target process kill that selects is fallen;
7) outputting log file information tells the operation of corresponding its operation of user to be stopped, and the time that stops, the information that the process that stops is relevant with process;
Pass through above-mentioned steps; A control flow completion of all domestic consumer's operation processes on the fat node being used memory source; This process realizes with the form of Shell script; Plan target through add-on system is regularly carried out this script, thereby forms a closed-loop system of memory source being used control.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012100887904A CN102662762A (en) | 2012-03-30 | 2012-03-30 | Method for effectively controlling use of memory resource of fat node |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012100887904A CN102662762A (en) | 2012-03-30 | 2012-03-30 | Method for effectively controlling use of memory resource of fat node |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102662762A true CN102662762A (en) | 2012-09-12 |
Family
ID=46772261
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2012100887904A Pending CN102662762A (en) | 2012-03-30 | 2012-03-30 | Method for effectively controlling use of memory resource of fat node |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102662762A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103577306A (en) * | 2013-11-04 | 2014-02-12 | 浪潮(北京)电子信息产业有限公司 | Active-detecting high-availability software operating method in cloud platform |
| CN104239193A (en) * | 2014-09-04 | 2014-12-24 | 浪潮电子信息产业股份有限公司 | Linux-based CPU (Central Processing Unit) and memory usage rate collection method |
| CN105389201A (en) * | 2014-09-03 | 2016-03-09 | 中国石油化工股份有限公司 | Process management method and system thereof based on high-performance computing cluster |
| CN105740064A (en) * | 2014-12-12 | 2016-07-06 | 龙芯中科技术有限公司 | Method and device for preventing server from restarting in Linux system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1547121A (en) * | 2003-12-10 | 2004-11-17 | 中国科学院计算技术研究所 | Method for monitoring large-scale cluster system |
| CN101256506A (en) * | 2008-03-20 | 2008-09-03 | 金魁 | System for managing application program of computer network |
| CN101883003A (en) * | 2009-05-08 | 2010-11-10 | 华北电力大学 | Mandatory running method of computer operating system |
| CN101902481A (en) * | 2010-08-10 | 2010-12-01 | 厦门市美亚柏科信息股份有限公司 | Real-time monitoring method and device for webpage Trojan horse |
| CN102141934A (en) * | 2011-02-28 | 2011-08-03 | 浪潮(北京)电子信息产业有限公司 | Method and device for controlling process on fat node |
-
2012
- 2012-03-30 CN CN2012100887904A patent/CN102662762A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1547121A (en) * | 2003-12-10 | 2004-11-17 | 中国科学院计算技术研究所 | Method for monitoring large-scale cluster system |
| CN101256506A (en) * | 2008-03-20 | 2008-09-03 | 金魁 | System for managing application program of computer network |
| CN101883003A (en) * | 2009-05-08 | 2010-11-10 | 华北电力大学 | Mandatory running method of computer operating system |
| CN101902481A (en) * | 2010-08-10 | 2010-12-01 | 厦门市美亚柏科信息股份有限公司 | Real-time monitoring method and device for webpage Trojan horse |
| CN102141934A (en) * | 2011-02-28 | 2011-08-03 | 浪潮(北京)电子信息产业有限公司 | Method and device for controlling process on fat node |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103577306A (en) * | 2013-11-04 | 2014-02-12 | 浪潮(北京)电子信息产业有限公司 | Active-detecting high-availability software operating method in cloud platform |
| CN105389201A (en) * | 2014-09-03 | 2016-03-09 | 中国石油化工股份有限公司 | Process management method and system thereof based on high-performance computing cluster |
| CN105389201B (en) * | 2014-09-03 | 2018-11-13 | 中国石油化工股份有限公司 | A kind of process management method and its system based on High Performance Computing Cluster |
| CN104239193A (en) * | 2014-09-04 | 2014-12-24 | 浪潮电子信息产业股份有限公司 | Linux-based CPU (Central Processing Unit) and memory usage rate collection method |
| CN105740064A (en) * | 2014-12-12 | 2016-07-06 | 龙芯中科技术有限公司 | Method and device for preventing server from restarting in Linux system |
| CN105740064B (en) * | 2014-12-12 | 2019-07-16 | 龙芯中科技术有限公司 | The method and device for preventing server-side from restarting in linux system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP4488072B2 (en) | Server system and power reduction method for server system | |
| CN102141934A (en) | Method and device for controlling process on fat node | |
| US20090235097A1 (en) | Data Center Power Management | |
| CN102521055B (en) | Virtual machine resource allocating method and virtual machine resource allocating system | |
| US8547840B1 (en) | Bandwidth allocation of bursty signals | |
| CN102624546B (en) | Control method, control equipment and control system for capping power consumption | |
| CN104991830A (en) | YARN resource allocation and energy-saving scheduling method and system based on service level agreement | |
| CN102759984A (en) | Power supply and performance management system for virtualization server cluster | |
| DE102015118705A1 (en) | Technologies for performance-based off-line data center task scheduling | |
| CN104412649A (en) | Heartbeat adjusting method, device and terminal | |
| CN102662762A (en) | Method for effectively controlling use of memory resource of fat node | |
| Chiu et al. | Electric grid balancing through lowcost workload migration | |
| CN103399800A (en) | Dynamic load balancing method based on Linux parallel computing platform | |
| US8015423B1 (en) | Temporally normalized processor utilization | |
| CN111120383A (en) | Control method and control device for equipment fan, switch and storage medium | |
| CN102339233A (en) | Cloud computing centralized management platform | |
| Niu et al. | JouleMR: Towards cost-effective and green-aware data processing frameworks | |
| CN102208986A (en) | Cluster power consumption distribution and control method | |
| CN105007209B (en) | A kind of poll means of communication based on RS485 | |
| WO2018114740A1 (en) | A local sdn controller and corresponding method of performing network control and management functions | |
| CN107132903B (en) | Energy-saving management implementation method, device and network equipment | |
| CN103049326B (en) | Method and system for managing job program of job management and scheduling system | |
| CN104468710A (en) | Mixed big data processing system and method | |
| CN116028193B (en) | Big data task dynamic high-energy-efficiency scheduling method and system for mixed part cluster | |
| Niu et al. | Not all joules are equal: Towards energy-efficient and green-aware data processing frameworks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20120912 |