CN104735095B - A kind of cloud computing platform job scheduling method and device - Google Patents

A kind of cloud computing platform job scheduling method and device Download PDF

Info

Publication number
CN104735095B
CN104735095B CN201310697757.6A CN201310697757A CN104735095B CN 104735095 B CN104735095 B CN 104735095B CN 201310697757 A CN201310697757 A CN 201310697757A CN 104735095 B CN104735095 B CN 104735095B
Authority
CN
China
Prior art keywords
mrow
resource
msup
cloud computing
computing platform
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201310697757.6A
Other languages
Chinese (zh)
Other versions
CN104735095A (en
Inventor
何淼
曾键
陈刚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Mobile Group Sichuan Co Ltd
Original Assignee
China Mobile Group Sichuan Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Mobile Group Sichuan Co Ltd filed Critical China Mobile Group Sichuan Co Ltd
Priority to CN201310697757.6A priority Critical patent/CN104735095B/en
Publication of CN104735095A publication Critical patent/CN104735095A/en
Application granted granted Critical
Publication of CN104735095B publication Critical patent/CN104735095B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H04L67/1029—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers using data related to the state of servers by a load balancer
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H04L67/1004—Server selection for load balancing

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Debugging And Monitoring (AREA)
  • Multi Processors (AREA)

Abstract

The invention discloses a kind of cloud computing platform job scheduling method and device.Wherein this method includes：The operation received is parsed, obtains the characteristic parameter of the operation；According to the resource requirement of operation described in the calculation of characteristic parameters of the operation；It is that the operation distributes resource according to the resource requirement of the operation and cloud computing platform resources left information；Resource allocation result is sent to cloud computing platform and performs operation.

Description

A kind of cloud computing platform job scheduling method and device
Technical field
The present invention relates to technical field of computer information processing, more particularly to a kind of cloud computing platform job scheduling method and Device.
Background technology
With the continuous improvement of the continuous growth and lean operation management requirement of userbase, in mobile communication carrier The analysis system in portion, such as BASS, VGOP, FOS system all suffer from the impact in big data epoch, and original system architecture can not Meets the needs of business, the problem of below generally existing：1st, process performance deficiency, can not meet big data and unstructured data Process performance requirement；2nd, maintenance cost height is built, original " minicomputer+high-end storage+relevant database " pattern is soft or hard Part is purchased and maintenance cost is all very high；3rd, system reliability is not high；4th, autgmentability is poor, can not fast linear extension meet The requirement that data processing increases；
And the appearance using Hadoop as the cloud computing technology of representative, preferably resolve these problems, therefore mobile communication The analysis system of operator all progressively is implemented to make the transition to cloud computing platform.But constantly expand with the scale of cloud computing platform, must The most effective carrying miscellaneous service under certain resource limit need to be considered, therefore, how to improve the resource utilization of cloud computing platform Becoming influences the key factor of cloud computing platform using effect, so as to a problem for needing urgently to solve as us.
BASS（Business Analyse Support System）For mobile operator operation analysis systems, refer to Business decision-making support, market management analysis and line marketing service support support information system for the intelligence of service goal；VGOP (Value-added Service General Operation Platform) is mobile operator value-added service comprehensive operations Platform, it is responsible for providing the data service support system for unifying operation ability across business platform；FOS(data Flow Operation Management System) it is that mobile operator flows manage comprehensive operation managing system, there is provided analysis mining, accurate marketing With the IT support systems of managerial ability.
Under existing framework, the submission operation of each operation system independence states resource requirement in advance to cloud computing platform. Cloud computing platform receives operation and is put into unified job queue, and job scheduling module is according to a simple first in first out （Fifo queue）Dispatching algorithm carries out job scheduling, and the resource requirement proposed in advance according to each operation carries out resource allocation, first To first.The resource requirement of all operations is required to confirm in advance, relies on and close between the contention for resources and operation between multitask System is also required to plan in advance before operation is submitted.
Existing framework has higher dispatching efficiency when being used in single operation system, but in the complexity of multisystem Under service environment, exist resource using it is unbalanced, can not react traffic performance, lack dynamic dispatching mechanism the problems such as, specifically such as Under：
（One）Resource uses unbalanced
Cloud computing platform shares 400 CPU cores（CPU core）Physical resource.In a certain period, only one work The BASS interface data that industry ID is 2.1 handles job run, and the resource requirement of the operation is 100 CPU core, much smaller than cloud The existing available resource of calculating platform, but the resource requirement that cloud computing platform can only be submitted according to the operation distributes 100 CPU Core, more resources can not be distributed to the operation, cause cloud computing platform resources idle, and the run time of the operation also compared with It is long.If automatic increase distributes the resource of the operation, then the job run time can be greatly shortened.
（Two）It can not be dispatched according to job priority
Cloud computing platform shares 400 CPU core physical resource.In a certain period, the FOS systems that existing ID is 1.3 To collect layer data processing operation to bring into operation, the operation takes 300 CPU core, it is contemplated that the Job execution time is 2 hours, Priority level is low.If the one-shot job for now having the multi-platform data statistics that an ID is 4.1 is submitted, the job priority rank For height, resource requirement is 200 CPUcore, it is contemplated that run time is 10 minutes.According to the existing manager of cloud computing platform Formula, the operation that ID is 4.1, which needs to wait in line release resource after the operation that ID is 1.3 is fully completed, can just bring into operation, and need Treat 2 hours or so, it is impossible to meet the time requirement of high priority operation.If the operation that pause ID is 1.3 is excellent, preferentially first carry out ID is 4.1 operation, then only needs for more than 10 minutes can complete high priority operation.
（Three）It can not be scheduled according to traffic performance and dependence
There is dependence in 2 operations of cloud computing platform, the BASS systems combined data that operation ID is 2.3 handles operation The result of operation is handled dependent on the slight combined data of BASS systems that operation ID is 2.2.Because of some reasons（Operation submission time Delay, operation abnormal interrupt, contention for resources etc.）When causing the operation that ID is 2.3 in queue to come before the operation that ID is 2.2, Existing way to manage will run the operation that ID is 2.3 first, cause the work data result abnormal.In multiple business systems Under the complex environment of system, between system and internal system operation exist dependence and strict serial process order, existing frame Structure can not automatically process these relations, can not more ensure the processing on time of critical path operation.
The content of the invention
In order to solve can not to be dispatched in the prior art according to job priority, can not be carried out according to traffic performance and dependence Job scheduling, cloud computing platform resource use unbalanced technical problem, and the present invention proposes a kind of cloud computing platform job scheduling Method and device.
One aspect of the present invention, there is provided a kind of cloud computing platform job scheduling method, including：
The operation received is parsed, obtains the characteristic parameter of the operation；
According to the resource requirement of operation described in the calculation of characteristic parameters of the operation；
It is that the operation distributes resource according to the resource requirement of the operation and cloud computing platform resources left information；
Resource allocation result is sent to cloud computing platform and performs operation.
Another aspect of the present invention, there is provided a kind of cloud computing platform job scheduling device, including：
Operation parsing module, for being parsed to the operation received, obtain the characteristic parameter of the operation；
Resource Calculation module, the resource requirement for operation described in the calculation of characteristic parameters according to the operation；
Resource distribution module, it is described for the resource requirement according to the operation and cloud computing platform resources left information Operation distributes resource；
Operation sending module, operation is performed for resource allocation result to be sent into cloud computing platform.
The cloud computing platform job scheduling method and device of the present invention, by the estimation to resource needed for operation, with reference to cloud Calculating platform occupation condition, the operation to cloud computing platform carry out unified real-time dynamic scheduling, reach resource automation The target of distribution is managed, balanced cloud computing platform loads, and improves the utilization rate of existing resource, ensures the order of business processing, Realize resource allocation automatically and reasonably.
Brief description of the drawings
Fig. 1 is the flow chart of cloud computing platform job scheduling method embodiment of the present invention；
Fig. 2 is the schematic diagram of the property list of operation of the present invention and system；
Fig. 3 is the structure chart of cloud computing platform job scheduling device embodiment of the present invention；
Fig. 4 is the flow chart of another embodiment of cloud computing platform job scheduling method of the present invention；
Fig. 5 is the flow chart of cloud computing platform job scheduling method another embodiment of the present invention；
Fig. 6 is the job queue situation schematic diagram that resource distribution module of the present invention obtains from operation sending module；
Fig. 7 is the personality presentation intention of another operation of the present invention and system.
Embodiment
The present invention solves the complicated industry of multisystem by stock assessment algorithm and dynamic dispatching mechanism based on traffic performance Under business environment the problem of the resource management of cloud computing platform.Below in conjunction with accompanying drawing, the present invention is described in detail.
As shown in figure 1, the present invention provides a kind of cloud computing platform job scheduling method embodiment, comprise the following steps：
Step 101, the operation received is parsed, obtains the characteristic parameter of operation；
Step 102, according to the resource requirement of the calculation of characteristic parameters operation of operation；
Step 103, it is that operation distributes resource according to the resource requirement of operation and cloud computing platform resources left information；
Step 104, resource allocation result is sent to cloud computing platform and performs operation.
Above method embodiment is right with reference to cloud computing platform occupation condition by the estimation to resource needed for operation The operation of cloud computing platform carries out unified real-time dynamic scheduling, reaches the target of resource automatic management distribution, balanced cloud meter Platform loads are calculated, the utilization rate of existing resource is improved, ensures the order of business processing, realize resource allocation automatically and reasonably.
In the present embodiment, the resource requirement of operation includes capability requirement and/or storage demand.Above-mentioned steps 102 have Body includes：
（1）The capability requirement of operation is calculated according to below equation：
Wherein, x1For the capability requirement of operation, can be represented with CPU core number；
N is the quantity of type of service corresponding to operation, and type of service corresponding to operation includes：The number of ports of each operation system According to handle, slightly collect, aggregation process, using displaying, temporary statistics etc.；When calculating capability requirement to different business Type calculates respectively, is then collected；
A is the data analysis flow number of operation flow corresponding to operation；
B1 is the Mean mapping of single data stream（Map）Number；
C1 is the average time-consuming of Map tasks, and unit is the second；
D1 is the Map task number of concurrent of single cpu kernel；
B2 is the average stipulations of single data stream（Reduce）Number；
C2 is the average time-consuming of Reduce tasks, and unit is the second；
D2 is the Reduce task number of concurrent of single cpu kernel；
E is job run time requirement, and unit is the second；
F is cloud computing platform ability redundancy coefficient, and the redundancy coefficient uses empirical value, typically takes 10%.
（2）The storage demand of operation is calculated according to below equation：
Wherein, x2For the storage demand of operation, unit Gbit；
N ' is the quantity of data type corresponding to operation, and data type corresponding to operation includes：Interface data, slightly collect Data, combined data, using display data, temporary statistics data etc.；When calculating storage demand to different data type point Do not calculate, then collected；
A ' is monocycle size of data, unit Gbit；
B ' is packing factor, using empirical value, typically takes 2；
C ' is data storage cycles, and unit is day；
D ' is compression ratio, using empirical value, according to the specific compress mode value of selection；
E ' is storage utilization rate, and storage utilization rate uses empirical value, typically takes 10%；
F ' is the storage capacity of single device.
Cloud computing platform resource is primarily referred to as CPU, Memory, Disk and I/O of X86 servers.Due to using base In the distributed cloud computing platform of Hadoop technologies, Hadoop is the software that distributed treatment can be carried out to mass data Framework.Under normal circumstances, Memory and I/O is not in single bottleneck, therefore the calculating of resource requirement is mainly for calculating Ability and memory space.In distributed cloud computing platform based on Hadoop, its calculating task is provided according to Hadoop MapReduce programming models are run, and the resource that an operation is consumed is determined by the scheduling of MapReduce on cloud computing platform Fixed, therefore by real-time job scheduling, optimal is assigned with MapReduce number of tasks, is also achieved that whole cloud computing The optimization of resource allocation management on platform.
Preferably, step 103 includes：Provided according to the capability requirement of operation and/or storage demand and cloud computing platform Source remaining information is that operation distributes resource；Generation resource allocation result includes new job queue, and business performs queue including following Information：Job execution sentence/script, the priority of operation, the execution dependence condition of operation, time requirement and the calculating energy of operation Power demand and/or storage demand.
Preferably, step 103 also includes：Obtain characteristic information and active job queue letter that system is initiated in operation Breath；The characteristic information that industry initiates system includes：The job list, resource occupation, priority, scheduling frequency and dependence condition；
It is further that operation distributes resource according to job queue information in the characteristic information of operation initiation system and operation.
In the present embodiment, the property list of an operation and system can be safeguarded, as shown in Fig. 2 corresponding comprising each operation system The job list and the key factor such as resource occupation, priority, scheduling frequency and dependence, carried out as based on traffic performance The foundation of resource management.In addition, the job queue information being currently running is obtained from cloud computing platform.The industry of operation will be initiated The characteristic information of business system, the job queue information being currently running, and the computing capability for the operation that step obtains before Demand and/or storage demand and cloud computing platform resources left informix are to together, to determine how as distribution resource. So, due to consideration that between operation system and inside operation system each business priority and dependence, more adduction Reason ground is allocated to the resource of cloud computing platform, further improves resource utilization and the order of operation processing.
Preferably, above-mentioned steps 104 include：
Current work queue is updated according to new job queue；
Current work queue after renewal is submitted into cloud computing platform to be performed.
The job queue waited in line is neutralized due to being currently likely present operation, when to new operation progress resource allocation Afterwards, it is necessary to update current job queue, the job queue after renewal is brought up into cloud computing platform.
On the other hand, as shown in figure 3, the present invention also provides a kind of cloud computing platform job scheduling device embodiment, including：
Operation parsing module 31, for being parsed to the operation received, obtain the characteristic parameter of operation；
Resource Calculation module 32, the resource requirement for the calculation of characteristic parameters operation according to operation；
Resource distribution module 33, it is operation point for the resource requirement according to operation and cloud computing platform resources left information With resource；
Operation sending module 34, operation is performed for resource allocation result to be sent into cloud computing platform.
Preferably, the resource requirement of operation includes capability requirement and/or storage demand, Resource Calculation module 32, uses In the capability requirement that operation is calculated according to below equation：
Wherein, x1For the capability requirement of operation；N is the quantity of type of service corresponding to operation；A is corresponding to operation The data analysis flow number of operation flow；B1 is the average Map numbers of single data stream；C1 is the average time-consuming of Map tasks；D1 is single CPU core Map task number of concurrent；B2 is the average Reduce numbers of single data stream；C2 is the average time-consuming of Reduce tasks； D2 is single CPU core Reduce task number of concurrent；E is job run time requirement；F is cloud computing platform ability redundancy system Number；
The storage demand of operation is calculated according to below equation：
Wherein, x2For the storage demand of operation；N ' is the quantity of data type corresponding to operation；A ' is that monocycle data are big It is small；B ' is packing factor；C ' is data storage cycles；D ' is compression ratio；E ' is storage utilization rate；F ' is the storage of single device Ability.
Preferably, resource distribution module 33 includes：
Distribution sub module 331, for the capability requirement according to operation and/or storage demand and cloud computing platform resource Remaining information is that operation distributes resource；
Submodule 332 is generated, includes new job queue for generating resource allocation result, business performs queue including following Information：Job execution sentence/script, the priority of operation, the execution dependence condition of operation, time requirement and the calculating energy of operation Power demand and/or storage demand.
Preferably, resource distribution module also includes：
Acquisition submodule 333, the characteristic information and active job queuing message of system are initiated for obtaining operation；Industry The characteristic information of initiation system includes：The job list, resource occupation, priority, scheduling frequency and dependence condition；
Distribution sub module 331, for further being believed according to job queue in the characteristic information of operation initiation system and operation Cease and distribute resource for operation.
Preferably, operation sending module 34, for updating current work queue according to new job queue；By working as after renewal Preceding job queue is submitted to cloud computing platform and performed.
Above method embodiment is described in detail by taking two typical flows as an example below.
First, the operation initiated BASS systems is scheduled
As shown in figure 4, cloud computing platform job scheduling method embodiment flow is as follows：
Step 401, BASS systems submit the BASS interface data that an ID is 2.1 to handle operation to operation parsing module；
Step 402, after operation parsing module completes operation parsing, the job transfer is given to Resource Calculation module；
Step 403, it is more than 100 CPU core that Resource Calculation module, which calculates resource needed for the operation,；
Step 404, resource distribution module obtains job queue from operation sending module, and existing job queue is sky, is not had There is the operation for being carrying out and being lined up；
Step 405, resource distribution module obtains occupation condition, cloud computing platform existing 400 from cloud computing platform Individual CPU core are idle condition；
Step 406, the resource requirement for the operation that ID is 2.1 is adjusted to 400 CPUcore by resource distribution module, and will The information transfer gives operation sending module；
Step 407, operation sending module renewal job queue, cloud computing platform operation is submitted into the operation that ID is 2.1.
It is operation with reference to cloud computing platform occupation condition by the estimation to resource needed for operation in the present embodiment Resource allocation is carried out, more reasonably the resource of cloud computing platform is allocated, further improves resource utilization.
2nd, the operation initiated other operation systems is scheduled
As shown in figure 5, cloud computing platform job scheduling method embodiment flow is as follows：
Step 501, other operation systems submit an ID facing for 4.1 multi-platform data statistics to operation parsing module Shi Zuoye, the job priority are height；
Step 502, after operation parsing module fulfils assignment parsing, the job transfer is given to Resource Calculation module；
Step 503, rule of thumb resource needed for formula assessment is more than 300 CPUcore to Resource Calculation module；
Step 504, resource distribution module obtains job queue from operation sending module, and queue situation is as shown in Figure 6；
Step 505, resource distribution module obtains occupation condition, cloud computing platform existing 400 from cloud computing platform Individual CPU core are use state, no remaining available resource；
Step 506, resource distribution module analyzes existing job queue and the operation newly submitted, in operation as shown in Figure 7 Association attributes with inquiring about each operation in the property list of system, resource distribution module analysis cloud computing platform resources left situation, Job queue situation, job priority and dependence, priority treatment high priority operation, and according to high priority operation according to Relation pair jobs scheduling is relied to be adjusted：Suspend the operation that ID is 2.5, performed according to operation ID 2.1,1.1,4.1 order, And the job queue after adjustment is transferred to operation execution module；
Step 507, operation sending module renewal job queue, suspend ID as 2.5 operation, be by ID after release resource Cloud computing platform operation is submitted in 2.1 operation.
In the present embodiment, by the estimation to resource needed for operation, with reference to cloud computing platform occupation condition, business system The priority of each business and dependence are that operation carries out resource allocation between system and inside operation system, more reasonably right The resource of cloud computing platform is allocated, and balanced cloud computing platform load, is improved the utilization rate of existing resource, is ensured business processing Order.
The above embodiment of the present invention, by the estimation to resource needed for operation, with reference to cloud computing platform occupation condition, Operation to cloud computing platform carries out unified real-time dynamic scheduling, and major advantage is as follows：
1st, the characteristic of every business carried on cloud computing platform and the resource management of cloud computing platform are associated, body Priority and dependence between existing system with each business of internal system, it is possible to achieve resource allocation automatically and reasonably；
2nd, operation can dynamically be dispatched according to the real time resources service condition of cloud computing platform, balanced cloud computing is put down Platform loads, and realizes that resource management becomes more meticulous, lifts the utilization rate of resource；
3rd, contention for resources is avoided the occurrence of.When resource anxiety, by assessing the priority of operation, time requirement, relying on and close System and resource consumption carry out dynamic dispatching, make business processing rational and orderly；
It should be noted that：Only to illustrate rather than limitation, the present invention is also not limited to above-mentioned above example Citing, all do not depart from the technical scheme of the spirit and scope of the present invention and its improvement, and it all should cover the right in the present invention In claimed range.

Claims (8)

  1. A kind of 1. cloud computing platform job scheduling method, it is characterised in that including：
    The operation received is parsed, obtains the characteristic parameter of the operation；
    According to the resource requirement of operation described in the calculation of characteristic parameters of the operation；
    It is that the operation distributes resource according to the resource requirement of the operation and cloud computing platform resources left information；
    Resource allocation result is sent to cloud computing platform and performs operation；
    The resource requirement of the operation includes capability requirement and storage demand；According to the calculation of characteristic parameters of operation institute Stating the resource requirement of operation includes：
    The capability requirement of the operation is calculated according to below equation：
    <mrow> <msub> <mi>x</mi> <mn>1</mn> </msub> <mo>=</mo> <mfrac> <mrow> <msubsup> <mo>&amp;Sigma;</mo> <mn>1</mn> <mi>n</mi> </msubsup> <mi>A</mi> <mo>&amp;times;</mo> <mrow> <mo>(</mo> <mfrac> <mrow> <mi>B</mi> <mn>1</mn> <mo>&amp;times;</mo> <mi>C</mi> <mn>1</mn> </mrow> <mrow> <mi>D</mi> <mn>1</mn> </mrow> </mfrac> <mo>+</mo> <mfrac> <mrow> <mi>B</mi> <mn>2</mn> <mo>&amp;times;</mo> <mi>C</mi> <mn>2</mn> </mrow> <mrow> <mi>D</mi> <mn>2</mn> </mrow> </mfrac> <mo>)</mo> </mrow> </mrow> <mi>E</mi> </mfrac> <mo>&amp;times;</mo> <mrow> <mo>(</mo> <mn>1</mn> <mo>+</mo> <mi>F</mi> <mo>)</mo> </mrow> <mo>;</mo> </mrow>
    Wherein, x1For the capability requirement of the operation；N is the quantity of type of service corresponding to operation；A is the operation pair The data analysis flow number for the operation flow answered；B1 is the Mean mapping Map numbers of single data stream；C1 is the average consumption of Map tasks When；D1 is the Map task number of concurrent of single cpu kernel；B2 is average stipulations (Reduce) number of single data stream；C2 is Reduce The average of task takes；D2 is the Reduce task number of concurrent of single cpu kernel；E is job run time requirement；F is cloud meter Calculate platform capabilities redundancy coefficient；
    The storage demand of the operation is calculated according to below equation：
    <mrow> <msub> <mi>x</mi> <mn>2</mn> </msub> <mo>=</mo> <mfrac> <mrow> <msubsup> <mi>&amp;Sigma;</mi> <mn>1</mn> <msup> <mi>n</mi> <mo>&amp;prime;</mo> </msup> </msubsup> <mfrac> <mrow> <msup> <mi>A</mi> <mo>&amp;prime;</mo> </msup> <mo>&amp;times;</mo> <msup> <mi>B</mi> <mo>&amp;prime;</mo> </msup> <mo>&amp;times;</mo> <msup> <mi>C</mi> <mo>&amp;prime;</mo> </msup> </mrow> <msup> <mi>D</mi> <mo>&amp;prime;</mo> </msup> </mfrac> </mrow> <mrow> <msup> <mi>E</mi> <mo>&amp;prime;</mo> </msup> <mo>&amp;times;</mo> <msup> <mi>F</mi> <mo>&amp;prime;</mo> </msup> </mrow> </mfrac> <mo>;</mo> </mrow>
    Wherein, x2For the storage demand of the operation；N ' is the quantity of data type corresponding to operation；A ' is that monocycle data are big It is small；B ' is packing factor；C ' is data storage cycles；D ' is compression ratio；E ' is storage utilization rate；F ' is the storage of single device Ability.
  2. 2. according to the method for claim 1, it is characterised in that provided according to the resource requirement of the operation and cloud computing platform Source remaining information is that operation distribution resource includes：
    It is the operation point according to the capability requirement of the operation and storage demand and cloud computing platform resources left information With resource；
    Generation resource allocation result includes new job queue.
  3. 3. according to the method for claim 2, it is characterised in that methods described also includes：
    Obtain operation and initiate the job queue information that the characteristic information of system and cloud computing platform are currently running；The operation The characteristic information of initiation system includes：The job list, resource occupation, priority, scheduling frequency and dependence condition；
    The job queue being further currently running according to the characteristic information of operation initiation system and cloud computing platform is believed Cease and distribute resource for the operation.
  4. 4. according to the method for claim 2, it is characterised in that resource allocation result is sent to cloud computing platform and performs work Industry includes：
    Current work queue is updated according to the new job queue；
    Current work queue after renewal is submitted into cloud computing platform to be performed.
  5. A kind of 5. cloud computing platform job scheduling device, it is characterised in that including：
    Operation parsing module, for being parsed to the operation received, obtain the characteristic parameter of the operation；
    Resource Calculation module, the resource requirement for operation described in the calculation of characteristic parameters according to the operation；
    Resource distribution module, it is the operation for the resource requirement according to the operation and cloud computing platform resources left information Distribute resource；
    Operation sending module, operation is performed for resource allocation result to be sent into cloud computing platform；
    The resource requirement of the operation includes capability requirement and storage demand；The Resource Calculation module, for according to Lower formula calculates the capability requirement of the operation：
    <mrow> <msub> <mi>x</mi> <mn>1</mn> </msub> <mo>=</mo> <mfrac> <mrow> <msubsup> <mo>&amp;Sigma;</mo> <mn>1</mn> <mi>n</mi> </msubsup> <mi>A</mi> <mo>&amp;times;</mo> <mrow> <mo>(</mo> <mfrac> <mrow> <mi>B</mi> <mn>1</mn> <mo>&amp;times;</mo> <mi>C</mi> <mn>1</mn> </mrow> <mrow> <mi>D</mi> <mn>1</mn> </mrow> </mfrac> <mo>+</mo> <mfrac> <mrow> <mi>B</mi> <mn>2</mn> <mo>&amp;times;</mo> <mi>C</mi> <mn>2</mn> </mrow> <mrow> <mi>D</mi> <mn>2</mn> </mrow> </mfrac> <mo>)</mo> </mrow> </mrow> <mi>E</mi> </mfrac> <mo>&amp;times;</mo> <mrow> <mo>(</mo> <mn>1</mn> <mo>+</mo> <mi>F</mi> <mo>)</mo> </mrow> <mo>;</mo> </mrow>
    Wherein, x1For the capability requirement of the operation；N is the quantity of type of service corresponding to operation；A is the operation pair The data analysis flow number for the operation flow answered；B1 is the Mean mapping Map numbers of single data stream；C1 is the average consumption of Map tasks When；D1 is the Map task number of concurrent of single cpu kernel；B2 is average stipulations (Reduce) number of single data stream；C2 is Reduce The average of task takes；D2 is the Reduce task number of concurrent of single cpu kernel；E is job run time requirement；F is cloud meter Calculate platform capabilities redundancy coefficient；
    The storage demand of the operation is calculated according to below equation：
    <mrow> <msub> <mi>x</mi> <mn>2</mn> </msub> <mo>=</mo> <mfrac> <mrow> <msubsup> <mi>&amp;Sigma;</mi> <mn>1</mn> <msup> <mi>n</mi> <mo>&amp;prime;</mo> </msup> </msubsup> <mfrac> <mrow> <msup> <mi>A</mi> <mo>&amp;prime;</mo> </msup> <mo>&amp;times;</mo> <msup> <mi>B</mi> <mo>&amp;prime;</mo> </msup> <mo>&amp;times;</mo> <msup> <mi>C</mi> <mo>&amp;prime;</mo> </msup> </mrow> <msup> <mi>D</mi> <mo>&amp;prime;</mo> </msup> </mfrac> </mrow> <mrow> <msup> <mi>E</mi> <mo>&amp;prime;</mo> </msup> <mo>&amp;times;</mo> <msup> <mi>F</mi> <mo>&amp;prime;</mo> </msup> </mrow> </mfrac> <mo>;</mo> </mrow>
    Wherein, x2For the storage demand of the operation；N ' is the quantity of data type corresponding to operation；A ' is that monocycle data are big It is small；B ' is packing factor；C ' is data storage cycles；D ' is compression ratio；E ' is storage utilization rate；F ' is the storage of single device Ability.
  6. 6. device according to claim 5, it is characterised in that the resource distribution module includes：
    Distribution sub module, believe for the capability requirement according to the operation and storage demand and cloud computing platform resources left Cease and distribute resource for the operation；
    Submodule is generated, includes new job queue for generating resource allocation result.
  7. 7. device according to claim 6, it is characterised in that the resource distribution module also includes：
    Acquisition submodule, the operation team for initiating the characteristic information of system and cloud computing platform for obtaining operation and being currently running Column information；The characteristic information that system is initiated in the operation includes：The job list, resource occupation, priority, scheduling frequency and dependence Condition；
    The distribution sub module, for further current just according to the characteristic information of operation initiation system and cloud computing platform In the job queue information of operation resource is distributed for the operation.
  8. 8. device according to claim 6, it is characterised in that the operation sending module, for according to the new job Queue updates current work queue；Current work queue after renewal is submitted into cloud computing platform to be performed.
CN201310697757.6A 2013-12-18 2013-12-18 A kind of cloud computing platform job scheduling method and device Active CN104735095B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201310697757.6A CN104735095B (en) 2013-12-18 2013-12-18 A kind of cloud computing platform job scheduling method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201310697757.6A CN104735095B (en) 2013-12-18 2013-12-18 A kind of cloud computing platform job scheduling method and device

Publications (2)

Publication Number Publication Date
CN104735095A CN104735095A (en) 2015-06-24
CN104735095B true CN104735095B (en) 2018-02-23

Family

ID=53458529

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201310697757.6A Active CN104735095B (en) 2013-12-18 2013-12-18 A kind of cloud computing platform job scheduling method and device

Country Status (1)

Country Link
CN (1) CN104735095B (en)

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105159782B (en) * 2015-08-28 2018-11-02 北京百度网讯科技有限公司 Based on the method and apparatus that cloud host is Order splitting resource
TWI582607B (en) * 2015-11-02 2017-05-11 廣達電腦股份有限公司 Dynamic resources planning mechanism based on cloud computing and smart device
CN105516242A (en) * 2015-11-23 2016-04-20 华为技术有限公司 Storage resource distribution method and storage resource distribution system
CN105607955A (en) * 2015-12-23 2016-05-25 浪潮集团有限公司 Calculation task distribution method and apparatus
CN107038072B (en) * 2016-02-03 2019-10-25 博雅网络游戏开发（深圳）有限公司 Method for scheduling task and device based on Hadoop system
CN105824705B (en) * 2016-04-01 2019-10-11 广州品唯软件有限公司 A kind of method for allocating tasks and electronic equipment
CN106484520A (en) * 2016-10-17 2017-03-08 北京集奥聚合科技有限公司 A kind of intelligent dispatching method based on data blood relationship and system
CN106776025A (en) * 2016-12-16 2017-05-31 郑州云海信息技术有限公司 A kind of computer cluster job scheduling method and its device
CN107454137B (en) * 2017-06-16 2020-09-15 广州天宁信息技术有限公司 Method, device and equipment for on-line business on-demand service
CN107885589B (en) * 2017-11-22 2021-02-12 贝壳找房(北京)科技有限公司 Job scheduling method and device
CN110209645A (en) * 2017-12-30 2019-09-06 中国移动通信集团四川有限公司 Task processing method, device, electronic equipment and storage medium
CN108960641B (en) * 2018-07-10 2021-07-02 康成投资（中国）有限公司 E-commerce platform operation scheduling method and system
CN109710414B (en) * 2018-12-29 2021-03-26 北京三快在线科技有限公司 Job scheduling method, device, equipment and storage medium
CN110012507B (en) * 2019-04-02 2021-01-26 华南理工大学 Internet of vehicles resource allocation method and system with priority of user experience
CN116302447B (en) * 2023-04-27 2023-08-04 云动时代科技股份有限公司 Cloud platform-based method for managing software and software management system

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102780759A (en) * 2012-06-13 2012-11-14 合肥工业大学 Cloud computing resource scheduling method based on scheduling object space
CN102902344A (en) * 2011-12-23 2013-01-30 同济大学 Method for optimizing energy consumption of cloud computing system based on random tasks

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5028469B2 (en) * 2009-12-14 2012-09-19 株式会社日立製作所 Information processing apparatus, resource schedule method, and resource schedule program

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102902344A (en) * 2011-12-23 2013-01-30 同济大学 Method for optimizing energy consumption of cloud computing system based on random tasks
CN102780759A (en) * 2012-06-13 2012-11-14 合肥工业大学 Cloud computing resource scheduling method based on scheduling object space

Also Published As

Publication number Publication date
CN104735095A (en) 2015-06-24

Similar Documents

Publication Publication Date Title
CN104735095B (en) A kind of cloud computing platform job scheduling method and device
Zeng et al. Joint optimization of task scheduling and image placement in fog computing supported software-defined embedded system
CN107450982B (en) A kind of method for scheduling task based on system mode
CN104657220B (en) Scheduling model and method based on deadline and expense restriction in mixed cloud
Wang et al. A deep learning based energy-efficient computational offloading method in Internet of vehicles
CN103246546B (en) Based on open electric system Numeral Emulation System and the emulation mode thereof of cloud
CN102780759B (en) Based on the cloud computing resource scheduling method in regulation goal space
WO2024060571A1 (en) Heterogeneous computing power-oriented multi-policy intelligent scheduling method and apparatus
CN106844051A (en) The loading commissions migration algorithm of optimised power consumption in a kind of edge calculations environment
CN107003887A (en) Overloaded cpu setting and cloud computing workload schedules mechanism
CN105242956A (en) Virtual function service chain deployment system and deployment method therefor
CN105718364A (en) Dynamic assessment method for ability of computation resource in cloud computing platform
CN107122243A (en) Heterogeneous Cluster Environment and CFD computational methods for CFD simulation calculations
CN111611062B (en) Cloud-edge collaborative hierarchical computing method and cloud-edge collaborative hierarchical computing system
CN105893158A (en) Big data hybrid scheduling model on private cloud condition
Ye et al. A framework for QoS and power management in a service cloud environment with mobile devices
CN103338228A (en) Cloud calculating load balancing scheduling algorithm based on double-weighted least-connection algorithm
CN105426241A (en) Cloud computing data center based unified resource scheduling energy-saving method
KR102052964B1 (en) Method and system for scheduling computing
CN107291544A (en) Method and device, the distributed task scheduling execution system of task scheduling
CN105373432A (en) Cloud computing resource scheduling method based on virtual resource state prediction
CN107040475A (en) Resource regulating method and device
CN109981723A (en) File cache processing system and method, communication system based on deeply study
Karimiafshar et al. Effective utilization of renewable energy sources in fog computing environment via frequency and modulation level scaling
Al-Sinayyid et al. Job scheduler for streaming applications in heterogeneous distributed processing systems

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant