CN106293949A - Resource dispatching strategy based on baseline analysis under a kind of computing environment - Google Patents
Resource dispatching strategy based on baseline analysis under a kind of computing environment Download PDFInfo
- Publication number
- CN106293949A CN106293949A CN201610689781.9A CN201610689781A CN106293949A CN 106293949 A CN106293949 A CN 106293949A CN 201610689781 A CN201610689781 A CN 201610689781A CN 106293949 A CN106293949 A CN 106293949A
- Authority
- CN
- China
- Prior art keywords
- resource
- data
- dispatching strategy
- resource dispatching
- scheduling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5077—Logical partitioning of resources; Management or configuration of virtualized resources
Abstract
The present invention be more particularly directed to resource dispatching strategy based on baseline analysis under a kind of computing environment.Resource dispatching strategy based on baseline analysis under this computing environment, sets up lexical analysis system, and described lexical analysis system includes log analyzing module, data analysis module and scheduling of resource module；Described log analyzing module includes daily record uploading unit and daily record resolution unit, described data analysis module includes Data Integration and analytic unit and checks analysis result unit, and described scheduling of resource module includes resource dispatching strategy signal generating unit and resource dispatching strategy performance element.Resource dispatching strategy based on baseline analysis under this computing environment, propose a lexical analysis system, the collection by daily record data of the lexical analysis system, data are analyzed, thus generate the scheduling strategy being conducive to improving overall efficiency, by the execution of scheduling strategy, improve the effective rate of utilization of node, it is achieved thereby that the lifting of overall efficiency.
Description
Technical field
The present invention relates to field of cloud computer technology, particularly to scheduling of resource based on baseline analysis under a kind of computing environment
Strategy.
Background technology
The essence of " cloud " is the non-physical of system itself, it is believed that cloud computing is for the customized void of user
Intending calculating system, therefore Intel Virtualization Technology becomes the bottom foundation stone that cloud computing realizes.Virtualization is to represent taking out of computer resource
As method, can be by the resource after abstract with the way access accessing abstract front resource consistence, this money by Intel Virtualization Technology
The abstract method in source is not limited by the physical configuration of realization, geographical position and underlying resource.
Along with the maturation of Intel Virtualization Technology and developing rapidly of the Internet, each big business information system more turns to various
Privately owned cloud or publicly-owned cloud, the data of generation also increase rapidly.People recognize the data importance to enterprise, carry out data
Analysis mining can provide purpose and information for the decision-making of enterprise.The data generally created and produce are destructuring or half structure
The data changed, are analyzed if these data are downloaded to relevant database, can send out the substantial amounts of time and money of expense.At this moment
Create such as Map/Reduce distributed computing framework, thus carry out analysis and the excavation of large data collection.
Cloud computing, utilizes system architecture technology that thousands of station servers are integrated, provides the user and provide flexibly
Source distribution and task scheduling ability.Intel Virtualization Technology is as one of key technology in cloud computing, and it can be by a physics meter
Calculation machine becomes the virtual computer system of multiple stage.The bottom architecture such as physical resource are carried out abstract by Intel Virtualization Technology, make hardware set
Difference between Bei and compatibility are transparent to upper layer application, thus realize the unified management of resources all kinds of to bottom.Virtualization skill
Art is a kind of method allocated and calculate resource, and it can be by different levels (hardware, software, data, network, the storage of application system
Deng) isolate, thus break data center, network, server, data, apply and store in physical equipment between draw
Point, it is achieved unified management and dynamically use physical resource and virtual resource, improve motility and the elasticity of system structure.
Accompanying drawing 1 is virtualized system architecture schematic diagram.Virtual machine monitoring software VMM in accompanying drawing, in Intel Virtualization Technology
(Virtual Machine Monitor), or referred to as Hypervisor, it has access to that all hardware equipment on server.
When startup of server and when calling Hypervisor, it can load the operating system on all virtual-machine client, gives every simultaneously
The physical resources such as network, CPU, disk and the internal memory that the distribution of individual virtual machine is appropriate.Hypervisor is responsible for coordinating these hardware money
The access in source, the most also applies security protection between each virtual machine.
In large-scale cloud computing environment, physical host is ten hundreds of, and the resource consumed is the hugest.Virtual
Change technology provides fictitious host computer, fictitious host computer to be the main objects under cloud computing environment for cloud computing, needs to consume main frame
The resources such as cpu, internal memory, disk and network interface card.But virtual machine is different by the physical resource shared by the configuration of virtual machine, the most empty
The consumption of resource is also not quite similar by plan machine by the difference of purposes.Based on above-mentioned situation, the present invention proposes a kind of computing environment
Under resource dispatching strategy based on baseline analysis.
Summary of the invention
The present invention is in order to make up the defect of prior art, it is provided that divide based on baseline under a kind of simple efficient computing environment
The resource dispatching strategy of analysis.
The present invention is achieved through the following technical solutions:
Resource dispatching strategy based on baseline analysis under a kind of computing environment, it is characterised in that: set up lexical analysis system, described
Lexical analysis system includes log analyzing module, data analysis module and scheduling of resource module；Described log analyzing module includes
Daily record uploading unit and daily record resolution unit, described data analysis module includes Data Integration and analytic unit and checks analysis knot
Really unit, described scheduling of resource module includes resource dispatching strategy signal generating unit and resource dispatching strategy performance element.
Described log analyzing module uses Map/Reduce programming model, and Mapper is responsible for reading daily record and resolving, and generates
The log object being prone to read gives Reducer process, and Reducer is responsible for merging all daily record datas, and exports HDFS
Storage.
Described data analysis module first passes through to be integrated data, and form as requested merges；Then utilize
K-means logarithm shows cluster analysis factually, improves computational load amount and efficiency, utilizes Map/Reduce programming framework to K-
Means realizes parallelization.
Described scheduling of resource module, according to Data Integration result, arranges virtual machine related information, by generating phase
The scheduling strategy closed, improves the utilization rate of node, and simultaneously after scheduling strategy generates, resource dispatching strategy performance element can be held
The scheduler task of row associated virtual machine, thus improve usefulness on the whole.
The invention has the beneficial effects as follows: resource dispatching strategy based on baseline analysis under this computing environment, it is proposed that
Lexical analysis system, the collection by daily record data of the lexical analysis system, data are analyzed, thus generate and be conducive to improving
The scheduling strategy of overall efficiency, by the execution of scheduling strategy, improves the effective rate of utilization of node, it is achieved thereby that overall effect
The lifting of energy.
Accompanying drawing explanation
Accompanying drawing 1 is virtualized system architecture schematic diagram.
Accompanying drawing 2 is the simple architecture schematic diagram of Hadoop.
Accompanying drawing 3 is lexical analysis system architecture schematic diagram of the present invention.
Detailed description of the invention
In order to make the technical problem to be solved, technical scheme and beneficial effect clearer, below tie
Closing drawings and Examples, the present invention will be described in detail.It should be noted that, specific embodiment described herein is only used
To explain the present invention, it is not intended to limit the present invention.
Resource dispatching strategy based on baseline analysis under this computing environment, sets up lexical analysis system, described lexical analysis
System includes log analyzing module, data analysis module and scheduling of resource module；Described log analyzing module includes that daily record is uploaded
Unit and daily record resolution unit, described data analysis module includes Data Integration and analytic unit and checks analysis result unit,
Described scheduling of resource module includes resource dispatching strategy signal generating unit and resource dispatching strategy performance element.
Described log analyzing module uses Hadoop framework, for realizing carrying out unformatted daily record data parsing lattice
Formulaization stores, and utilizes Map/Reduce programming framework to carry out parallelization and resolves daily record, extracts virtual machine performance number in daily record
Data are used, according to certain form storage after statistics according to (CPU, internal memory usage amount) and virtual machine process.Specifically,
Described log analyzing module uses Map/Reduce programming model, and Mapper is responsible for reading daily record and resolving, and generates and is prone to read
Log object give Reducer process, Reducer is responsible for merging all daily record datas, and export to HDFS store.
Described data analysis module first passes through to be integrated data, and form as requested merges；Then utilize
K-means algorithm logarithm shows cluster analysis factually, improves computational load amount and efficiency, utilizes Map/Reduce programming framework to K-
Means realizes parallelization.
Described scheduling of resource module, according to Data Integration result, arranges virtual machine related information, by generating phase
The scheduling strategy closed, improves the utilization rate of node, and simultaneously after scheduling strategy generates, resource dispatching strategy performance element can be held
The scheduler task of row associated virtual machine, thus improve usefulness on the whole.
1, about Hadoop framework
The core of Hadoop is made up of Map/Reduce and Hadoop Distributed File System.The bottom is
HDFS, it stores the file on all nodes of Hadoop, is made up of NameNode and DataNode.Map/Reduce is in
The last layer of HDFS, is made up of JobTracker and TaskTracker.
2, about K-means algorithm
K-means algorithm is as being clustering algorithm based on division the most classical.What this algorithm was maximum is a little to be succinctly
Quickly, and just can be applied on distributed computing framework by simple amendment, become what large-scale data was analyzed
Instrument.
K-means algorithm input numerical value is K and N number of data object, then N number of data object is divided into K cluster, makes to obtain
Cluster meet following condition:
(1), in same cluster, data object similarity is higher；
(2), in different clusters, data object similarity is relatively low.
Each cluster has a cluster centre, and the barycenter of the most each cluster is typically equal with the data object in each cluster
Value calculates cluster centre and embodies the feature of this cluster.
The basic thought of K-means algorithm is: set up K central point, calculates the similarity of object and central point, with in
The object that heart point similarity is high divides corresponding classification into, updates central point until obtaining preferable cluster result.
Claims (4)
1. resource dispatching strategy based on baseline analysis under a computing environment, it is characterised in that: set up lexical analysis system, institute
State lexical analysis system and include log analyzing module, data analysis module and scheduling of resource module；Described log analyzing module bag
Including daily record uploading unit and daily record resolution unit, described data analysis module includes Data Integration and analytic unit and checks analysis
Result unit, described scheduling of resource module includes resource dispatching strategy signal generating unit and resource dispatching strategy performance element.
Resource dispatching strategy based on baseline analysis under computing environment the most according to claim 1, it is characterised in that: described
Log analyzing module uses Map/Reduce programming model, and Mapper is responsible for reading daily record and resolving, and generates the day being prone to read
Will object gives Reducer process, and Reducer is responsible for merging all daily record datas, and exports HDFS storage.
Resource dispatching strategy based on baseline analysis under computing environment the most according to claim 1, it is characterised in that: described
Data analysis module first passes through to be integrated data, and form as requested merges；Then utilize K-means to data
Realize cluster analysis, improve computational load amount and efficiency, utilize Map/Reduce programming framework that K-means is realized parallelization.
Resource dispatching strategy based on baseline analysis under computing environment the most according to claim 1, it is characterised in that: described
Scheduling of resource module, according to Data Integration result, arranges virtual machine related information, by generating relevant scheduling strategy,
Improving the utilization rate of node, simultaneously after scheduling strategy generates, resource dispatching strategy performance element can perform associated virtual machine
Scheduler task, thus improve usefulness on the whole.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610689781.9A CN106293949A (en) | 2016-08-19 | 2016-08-19 | Resource dispatching strategy based on baseline analysis under a kind of computing environment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610689781.9A CN106293949A (en) | 2016-08-19 | 2016-08-19 | Resource dispatching strategy based on baseline analysis under a kind of computing environment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106293949A true CN106293949A (en) | 2017-01-04 |
Family
ID=57660747
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610689781.9A Pending CN106293949A (en) | 2016-08-19 | 2016-08-19 | Resource dispatching strategy based on baseline analysis under a kind of computing environment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106293949A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107277080A (en) * | 2017-08-23 | 2017-10-20 | 深信服科技股份有限公司 | A kind of is the internet risk management method and system of service based on safety |
| CN107784440A (en) * | 2017-10-23 | 2018-03-09 | 国网辽宁省电力有限公司 | A kind of power information system resource allocation system and method |
| CN109783211A (en) * | 2018-12-14 | 2019-05-21 | 成都四方伟业软件股份有限公司 | A kind of batch task scheduling system and dispatching method based on business diary |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140189703A1 (en) * | 2012-12-28 | 2014-07-03 | General Electric Company | System and method for distributed computing using automated provisoning of heterogeneous computing resources |
| CN104111996A (en) * | 2014-07-07 | 2014-10-22 | 山大地纬软件股份有限公司 | Health insurance outpatient clinic big data extraction system and method based on hadoop platform |
-
2016
- 2016-08-19 CN CN201610689781.9A patent/CN106293949A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140189703A1 (en) * | 2012-12-28 | 2014-07-03 | General Electric Company | System and method for distributed computing using automated provisoning of heterogeneous computing resources |
| CN104111996A (en) * | 2014-07-07 | 2014-10-22 | 山大地纬软件股份有限公司 | Health insurance outpatient clinic big data extraction system and method based on hadoop platform |
Non-Patent Citations (3)
| Title |
|---|
| 匡华等: "移动通信运营商办公云", 《移动通信》 * |
| 李梓萌等: "基于基线分析的云计算资源自动分配技术", 《现代计算机》 * |
| 王鹏: "云平台下日志分析系统的设计与实现", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107277080A (en) * | 2017-08-23 | 2017-10-20 | 深信服科技股份有限公司 | A kind of is the internet risk management method and system of service based on safety |
| CN107784440A (en) * | 2017-10-23 | 2018-03-09 | 国网辽宁省电力有限公司 | A kind of power information system resource allocation system and method |
| CN109783211A (en) * | 2018-12-14 | 2019-05-21 | 成都四方伟业软件股份有限公司 | A kind of batch task scheduling system and dispatching method based on business diary |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Fernández et al. | Big Data with Cloud Computing: an insight on the computing environment, MapReduce, and programming frameworks | |
| Cano et al. | Towards geo-distributed machine learning | |
| CN103873498B (en) | Cloud platform resource-adaptive method for early warning and system | |
| CN105049218B (en) | PhiCloud clouds charging method and system | |
| CN109492002A (en) | A kind of storage of smart grid big data and analysis system and processing method | |
| CN103838617A (en) | Method for constructing data mining platform in big data environment | |
| Taft et al. | STeP: Scalable tenant placement for managing database-as-a-service deployments | |
| Agneeswaran et al. | Paradigms for realizing machine learning algorithms | |
| Costa et al. | The SusCity big data warehousing approach for smart cities | |
| Ye et al. | Cloud-based big data mining & analyzing services platform integrating R | |
| CN106293949A (en) | Resource dispatching strategy based on baseline analysis under a kind of computing environment | |
| Zhang et al. | Splitting large medical data sets based on normal distribution in cloud environment | |
| Vrbić | Data mining and cloud computing | |
| Singh et al. | Big data: technologies, trends and applications | |
| Piccialli et al. | S-InTime: A social cloud analytical service oriented system | |
| Arvanitis et al. | Automated Performance Management for the Big Data Stack. | |
| Zhang et al. | Integrated design and development of intelligent scenic area rural tourism information service based on hybrid cloud | |
| Sridhar et al. | A study of big data analytics in clouds with a security perspective | |
| Ribeiro et al. | A scalable data integration architecture for smart cities: implementation and evaluation | |
| Khan et al. | Computational performance analysis of cluster-based technologies for big data analytics | |
| Naga Rama Devi | Emerging trends in big data analytics—A study | |
| Dadheech et al. | Performance improvement of heterogeneous Hadoop clusters using MapReduce for big data | |
| Zhao et al. | Simple parallel genetic algorithm using cloud computing | |
| Mrozek et al. | Technological Roadmap | |
| Kumar et al. | A New Framework for Effective Processing of WSN Application Generated Big Data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170104 |
|
| RJ01 | Rejection of invention patent application after publication |