CN107908534A - The analysis method of task run situation in a kind of large-scale cluster - Google Patents
The analysis method of task run situation in a kind of large-scale cluster Download PDFInfo
- Publication number
- CN107908534A CN107908534A CN201710946143.5A CN201710946143A CN107908534A CN 107908534 A CN107908534 A CN 107908534A CN 201710946143 A CN201710946143 A CN 201710946143A CN 107908534 A CN107908534 A CN 107908534A
- Authority
- CN
- China
- Prior art keywords
- cluster
- resource
- task
- tasks
- analysis method
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3452—Performance evaluation by statistical analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
- G06F11/3419—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment by assessing time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
- G06F11/3433—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment for load management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Probability & Statistics with Applications (AREA)
- Complex Calculations (AREA)
Abstract
The invention discloses a kind of analysis method of task run situation in large-scale cluster, including：Cluster gives each generic task distribution resource, each generic task is run in the cluster；By the normal ending status of all tasks and improper done state, database is stored in；The operation result data of all tasks are carried out to the analysis of different dimensions from database；Association process flow is output to the various dimension datas that come are analyzed, with reference to the result dimension data of cluster resource data and all tasks, exports the optimization process of resource usage policy；After cluster obtains the optimization process of resource usage policy, the cluster resource of distribution required for the task run in next cycle is manually or automatically instructed.Beneficial effect：Pass through the analysis method, the cluster owner is allowed easily and intuitively to understand each task run situation in current cluster, and more preferably cluster resource allocation strategy is obtained according to task run situation, be conducive to improve the utilization rate of cluster resource, so as to save the cost of investment of cluster resource.
Description
Technical field
The present invention relates to big data technical field, it particularly relates to task run situation in a kind of large-scale cluster
Analysis method.
Background technology
Big data technology is maked rapid progress；The cluster scale run in enterprise is also increasing, and the tenant in cluster is more next
More, the various tasks that cluster needs to run tenant are with regard to more and more.The species of task in one cluster is relatively more, there is data
Collection task, data cleansing task, data load task, data calculating task, convergence task, data analysis task, number
According to export task, data query task, big data platform monitoring task etc., so more task types, every kind of task type
There are several or more than ten, even tens task, so more tasks to run in the cluster again, and so more task bases
It is periodic task on this, there are 15 minute, hour, days equigranular, the operation result situation for having so multitask daily is existing
Lacking one kind to these task run result situations（The resource expended including operation duration, operation result state, operation accounts for
Than, shared CPU, MEM etc.）Analysis method.
The problem of in correlation technique, not yet propose effective solution at present.
The content of the invention
For the above-mentioned technical problem in correlation technique, the present invention proposes task run situation in a kind of large-scale cluster
Analysis method, it is possible to increase cluster utilization ratio.
To realize above-mentioned technical purpose, the technical proposal of the invention is realized in this way：
The analysis method of task run situation, comprises the following steps in a kind of large-scale cluster：
S1 clusters give each generic task distribution resource, each generic task is run in the cluster；
The normal ending status of all tasks and improper done state are stored in database by S2；
The operation result data of all tasks are carried out the analysis of different dimensions by S3 from database；
S4 is output to association process flow next various dimension datas are analyzed, with reference to cluster resource data and all tasks
Result dimension data, export the optimization process of resource usage policy；
After S5 clusters obtain the optimization process of resource usage policy, the task run institute in next cycle is manually or automatically instructed
Need the cluster resource distributed.
Further, the different dimensions analysis carried out in S3 includes but not limited to run time dimension, state dimension and money
Source dimensional analysis.
Beneficial effects of the present invention：By the analysis method, make the cluster owner convenient and be visually known current cluster
In each task run situation, and more preferably cluster resource allocation strategy is obtained according to task run situation, is conducive to improve
The utilization rate of cluster resource, so as to save the cost of investment of cluster resource.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to institute in embodiment
Attached drawing to be used is needed to be briefly described, it should be apparent that, drawings in the following description are only some implementations of the present invention
Example, for those of ordinary skill in the art, without creative efforts, can also obtain according to these attached drawings
Obtain other attached drawings.
Fig. 1 is the stream of the analysis method of task run situation in a kind of large-scale cluster described according to embodiments of the present invention
Journey schematic diagram.
Embodiment
Below in conjunction with the attached drawing in the embodiment of the present invention, the technical solution in the embodiment of the present invention is carried out clear, complete
Site preparation describes, it is clear that described embodiment is only part of the embodiment of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, those of ordinary skill in the art's all other embodiments obtained, belong to what the present invention protected
Scope.
As shown in Figure 1, according to embodiments of the present invention in a kind of large-scale cluster task run situation analysis side
Method, comprises the following steps：
S1 clusters give each generic task distribution resource, each generic task is run in the cluster；
The normal ending status of all tasks and improper done state are stored in database by S2；
The operation result data of all tasks are carried out the analysis of different dimensions by S3 from database；
S4 is output to association process flow next various dimension datas are analyzed, with reference to cluster resource data and all tasks
Result dimension data, export the optimization process of resource usage policy；
After S5 clusters obtain the optimization process of resource usage policy, the task run institute in next cycle is manually or automatically instructed
Need the cluster resource distributed.
Further, the different dimensions analysis carried out in S3 includes but not limited to run time dimension, state dimension and money
Source dimensional analysis.
Beneficial effects of the present invention：By analyzing the various task run result situations in big data cluster, allow
The cluster owner is convenient and is visually known each task run situation in current cluster, and is obtained according to task run situation
More preferably cluster resource allocation strategy, is conducive to improve the utilization rate of cluster resource, so as to save the cost of investment of cluster resource.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all essences in the present invention
With within principle, any modification, equivalent replacement, improvement and so on, should all be included in the protection scope of the present invention god.
Claims (3)
1. the analysis method of task run situation in a kind of large-scale cluster, it is characterised in that comprise the following steps：
S1 clusters give each generic task distribution resource, each generic task is run in the cluster；
The done state of all tasks is stored in database by S2；
The operation result data of all tasks are carried out the analysis of different dimensions by S3 from database；
S4 is output to association process flow next various dimension datas are analyzed, with reference to cluster resource data and all tasks
Result dimension data, export the optimization process of resource usage policy；
After S5 clusters obtain the optimization process of resource usage policy, the task run institute in next cycle is manually or automatically instructed
Need the cluster resource distributed.
2. the analysis method of task run situation in large-scale cluster according to claim 1, it is characterised in that the knot
Pencil state includes normal ending status and improper done state.
3. the analysis method of task run situation in large-scale cluster according to claim 1, it is characterised in that in S3 into
Capable different dimensions analysis includes but not limited to run time dimension, state dimension and resource dimension analysis.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710946143.5A CN107908534A (en) | 2017-10-12 | 2017-10-12 | The analysis method of task run situation in a kind of large-scale cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710946143.5A CN107908534A (en) | 2017-10-12 | 2017-10-12 | The analysis method of task run situation in a kind of large-scale cluster |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107908534A true CN107908534A (en) | 2018-04-13 |
Family
ID=61840470
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710946143.5A Pending CN107908534A (en) | 2017-10-12 | 2017-10-12 | The analysis method of task run situation in a kind of large-scale cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107908534A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102724220A (en) * | 2011-03-29 | 2012-10-10 | 无锡物联网产业研究院 | Method and apparatus for task cooperation, and system for internet of things |
| US8639818B1 (en) * | 2012-12-25 | 2014-01-28 | Kaspersky Lab Zao | System and method for reliable and timely task completion in a distributed computing environment |
| CN105718479A (en) * | 2014-12-04 | 2016-06-29 | 中国电信股份有限公司 | Execution strategy generation method and device under cross-IDC (Internet Data Center) big data processing architecture |
-
2017
- 2017-10-12 CN CN201710946143.5A patent/CN107908534A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102724220A (en) * | 2011-03-29 | 2012-10-10 | 无锡物联网产业研究院 | Method and apparatus for task cooperation, and system for internet of things |
| US8639818B1 (en) * | 2012-12-25 | 2014-01-28 | Kaspersky Lab Zao | System and method for reliable and timely task completion in a distributed computing environment |
| CN105718479A (en) * | 2014-12-04 | 2016-06-29 | 中国电信股份有限公司 | Execution strategy generation method and device under cross-IDC (Internet Data Center) big data processing architecture |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103458052B (en) | Resource scheduling method and device based on IaaS cloud platform | |
| CN106155791B (en) | A kind of workflow task dispatching method under distributed environment | |
| CN102360310B (en) | Multitask process monitoring method in distributed system environment | |
| CN102063336B (en) | Distributed computing multiple application function asynchronous concurrent scheduling method | |
| CN108762921B (en) | A kind of method for scheduling task and device of the on-line optimization subregion of Spark group system | |
| CN102508709B (en) | Distributed-cache-based acquisition task scheduling method in purchase, supply and selling integrated electric energy acquiring and monitoring system | |
| CN102193832B (en) | Cloud computing resource scheduling method and applied system | |
| CN111200528B (en) | Intelligent linkage method for smart city with edge cloud cooperation | |
| CN103473122B (en) | Workflow system resource scheduling method in cloud computing environment | |
| CN105893158A (en) | Big data hybrid scheduling model on private cloud condition | |
| CN104102513B (en) | A kind of CUDA runtime parameter transparent optimization methods based on Kepler frameworks | |
| JP2014512052A5 (en) | ||
| CN108241528A (en) | A kind of User Defined mass network secure data dynamic collecting method | |
| CN105096181A (en) | E-commerce transaction method and E-commerce transaction system for big data | |
| CN105488134A (en) | Big data processing method and big data processing device | |
| CN108132840A (en) | Resource regulating method and device in a kind of distributed system | |
| CN104156505B (en) | A kind of Hadoop cluster job scheduling method and devices based on user behavior analysis | |
| CN104111876A (en) | Dynamic resource management device and method based on Oracle resource plan | |
| CN109753362A (en) | A kind of confluence Method of Scheduling Parallel of hydrological distribution model | |
| CN107239853B (en) | Intelligent housekeeper system based on cloud computing and working method thereof | |
| CN103325012A (en) | Parallel computing dynamic task distribution method applicable to grid security correction | |
| CN109936465A (en) | A kind of cloud platform resource utilization appraisal procedure and device | |
| CN107908534A (en) | The analysis method of task run situation in a kind of large-scale cluster | |
| CN110264009B (en) | Shared automobile dispatching system and dispatching method thereof | |
| CN109144693A (en) | A kind of power adaptive method for scheduling task and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180413 |