CN105630601A - Resource allocation method and system based on real-time computing - Google Patents
Resource allocation method and system based on real-time computing Download PDFInfo
- Publication number
- CN105630601A CN105630601A CN201410612356.0A CN201410612356A CN105630601A CN 105630601 A CN105630601 A CN 105630601A CN 201410612356 A CN201410612356 A CN 201410612356A CN 105630601 A CN105630601 A CN 105630601A
- Authority
- CN
- China
- Prior art keywords
- task
- machine
- demand
- application
- resources
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention is applicable to technical field of Internet, and provides a resource allocation method based on real-time computing. The method comprises the following steps: acquiring number of multi-dimensional available resources reported by each machine; acquiring a required resource number of at least one dimension applied by an application; and according to the required resource number of the application of the number of the available resources of the machine, allocating resources for the application according to dimensions. Correspondingly, the invention further provides a resource allocation system based on real-time computing. Thereby, the invention can perform resource allocation on machine resources according to multiple dimensions in a real-time computer system, can achieve the maximum use of the machine resources, and solve the problem of unbalanced task allocation in the real-time computer system thoroughly.
Description
Technical field
The application relates to Internet technical field, particularly relates to a kind of based on the resource allocation methods calculated in real time and system.
Background technology
In the real time computation systems such as Storm or JStorm, each application (topology) is all that a data pipeline processes system, every one-level processing unit all needs to apply for some resources, this resource is usually worker process, and a worker process operates in certain port on a machine, and have the resources such as some processors (CPU), internal memory (Memory), hard disk (Disk), network (Network). Therefore need to set resource allocation algorithm, distribute some worker to this application.
When resource is allocated by existing real-time calculating, usual only one of which dimension. Being provided with a slot (task groove) list on every machine, each slot is the set of all available resources, comprises hard disk, processor, internal memory, network etc., and the resource data of each application is almost identical. But in real scene, applying the demand to different resource and different, one-dimensional degree resource allocation algorithm can cause the substantial amounts of wasting of resources, it is impossible to realizes resource and maximally utilizes. For example:
1, different task, demand to resource is different, assume in one application, spout (thread) assembly only needs 1G internal memory, and bolt (thread) assembly needs 8G internal memory, the final situation in application application is exactly, and each task (task) application 8G internal memory, such spout will waste a large amount of internal memory. Spout is the computing node obtaining data from the external world, and bolt is common computing node.
2, assume that application is processor type, one machine can only allow 10 such application to run simultaneously, but such application is little to memory demand, if the ability according to internal memory, reality can allow to run 20 such application, but owing to slot is a resource collection, according to short-board effect, causing arranging at most 10 slot on this machine, memory source does not fully use.
3, the application at processor and internal memory of the bottleneck that simultaneously runs on a machine is assumed, at this moment slot number is calculated just extremely difficult, a conservative value can only be taken as final utilization value, this conservative value is probably the slot number of maximum operation processor type and the minima of the slot number of maximum running memory type, because the final actual situation run at machine cannot be confirmed, it is the many of processor type or type of memory is many on earth.
In summary, prior art there will naturally be inconvenience and defect in actual use, it is therefore necessary to improved.
Summary of the invention
For above-mentioned defect, the purpose of the application is in that to provide a kind of based on the resource allocation methods calculated in real time and system, machine resources can be carried out resource distribution according to various dimensions by it in real-time computer system, the maximization that can realize machine resources uses, and the task in real-time computer system that thoroughly solves distributes unbalanced problem.
To achieve these goals, the application provides a kind of based on the resource allocation methods calculated in real time, has including step:
Obtain the available resources number of the various dimensions that every machine reports;
Obtain the demand number of resources of at least one dimension of application application;
Described demand number of resources according to described application and the described available resources number of described machine, be described allocated resources according to dimension.
According to resource allocation methods described herein, the step of the available resources number of the various dimensions that every machine of described acquisition reports includes:
Obtain the described available resources number of the various dimensions that every described machine reports, including one of hard disk available task groove number, processor available task groove number, internal memory available task groove number, network available task groove number or its combination in any.
According to resource allocation methods described herein, the step of the demand number of resources of at least one dimension of described acquisition application application includes:
Obtain one of the hard disk demand task groove number of described application application, processor demand task groove number, memory requirements task groove number, network demand task groove number or its combination in any;
The described available resources number of the described described demand number of resources according to described application and described machine, includes according to the step that dimension is described allocated resources:
According to one of the described hard disk demand task groove number of described application, described processor demand task groove number, described memory requirements task groove number, described network demand task groove number or its combination in any, and one of the described hard disk available task groove number of described machine, described processor available task groove number, described internal memory available task groove number, described network available task groove number or its combination in any, it is described allocated resources according to dimension.
According to resource allocation methods described herein, described hard disk available task groove number=hard disk number; Or
Described processor available task groove number=(processor check figure 1) * processor radix; Or
Described internal memory available task groove number=(physical memory size/mono-internal memory task groove size) * 80%; Or
Described network available task groove number=maximum network bandwidth/each task groove desire bandwidth.
According to resource allocation methods described herein, described application includes multiple task and process components; When the acquiescence demand number of resources of described task or described process components cannot meet application demand, described application is according to the more described demand number of resources of described application demand application.
According to resource allocation methods described herein, the step of the demand number of resources of at least one dimension of described acquisition application application also includes:
Described available resources number generation current machine list according to the various dimensions that machine every described reports;
The described available resources number of the described described demand number of resources according to described application and described machine, includes according to the step that dimension is described allocated resources:
To all tasks in described application, it is ranked up according to the described demand number of resources of each described task application;
Each described task demand number of resources in each dimension application is calculated according to institute's alignment sequence;
From all machines, find out the machine list of the described demand number of resources meeting each described task, and described task is distributed to the machine that described machine list is corresponding.
According to resource allocation methods described herein, the described machine list finding out the described demand number of resources meeting each described task from all machines, and the step that described task is distributed to machine corresponding to described machine list also includes:
If the described machine list of the described demand number of resources meeting described task cannot be found out from all machines, then judge that the resource of described task is distributed unsuccessfully.
According to resource allocation methods described herein, the weights of the described demand number of resources of described task application are described hard disk demand task groove number, described processor demand task groove number, described memory requirements task groove number and described network demand task groove number from high to low successively;
The described machine list finding out the described demand number of resources meeting each described task from all machines, and the step that described task is distributed to machine corresponding to described machine list includes:
First distribution sub-step, if the described hard disk demand task groove number of described task application, then from the described current machine list meeting described hard disk demand task groove number, described task is assigned on the first machine that described hard disk available task groove number is maximum, if maximum the first machine of described hard disk available task groove number has multiple, carries out next round allocation step;
Second distribution sub-step, if the described processor demand task groove number of described task application, if the described second machine only one of which that described processor available task groove number is maximum, then described task is assigned on described second machine, if maximum the second machine of described processor available task groove number has multiple, carries out next round allocation step;
3rd distribution sub-step, if the described memory requirements task groove number of described task application, if the described 3rd machine only one of which that described internal memory available task groove number is maximum, then described task is assigned on described 3rd machine, if maximum the 3rd machine of described internal memory available task groove number has multiple, carries out next round allocation step;
4th distribution sub-step, if the described network demand task groove number of described task application, if the described 4th machine only one of which that described network available task groove number is maximum, then described task is assigned on described 4th machine, if the 4th machine that described network available task groove number is maximum has multiple, then carry out next round allocation step;
5th distribution sub-step, selects the 5th machine from described 4th machine according to pre-defined rule, described task is assigned on described 5th machine.
According to resource allocation methods described herein, described the step of the 5th machine is selected to include from described 4th machine according to pre-defined rule:
The machine name of described 4th machine is ranked up by character sequence, and therefrom selects described 5th machine of the first tagmeme.
The present invention also provides for a kind of based on the resource allocation system calculated in real time, includes:
First acquisition module, for obtaining the available resources number of the various dimensions that every machine reports;
Second acquisition module, for obtaining the demand number of resources of at least one dimension of application application;
Resource distribution module, for the described available resources number of the described demand number of resources according to described application and described machine, is described allocated resources according to dimension.
According to resource allocation system described herein, described first acquisition module is for obtaining the described available resources number of the various dimensions that every described machine reports, including one of hard disk available task groove number, processor available task groove number, internal memory available task groove number, network available task groove number or its combination in any.
According to resource allocation system described herein, described second acquisition module is for obtaining one of the hard disk demand task groove number of described application application, processor demand task groove number, memory requirements task groove number, network demand task groove number or its combination in any;
Described resource distribution module is for according to one of the described hard disk demand task groove number of described application, described processor demand task groove number, described memory requirements task groove number, described network demand task groove number or its combination in any, and one of the described hard disk available task groove number of described machine, described processor available task groove number, described internal memory available task groove number, described network available task groove number or its combination in any, it is described allocated resources according to dimension.
According to resource allocation system described herein, described hard disk available task groove number=hard disk number; Or
Described processor available task groove number=(processor check figure 1) * processor radix; Or
Described internal memory available task groove number=(physical memory size/mono-internal memory task groove size) * 80%; Or
Described network available task groove number=maximum network bandwidth/each task groove desire bandwidth.
According to resource allocation system described herein, described application includes multiple task and process components; When the acquiescence demand number of resources of described task or described process components cannot meet application demand, described application is according to the more described demand number of resources of described application demand application.
According to resource allocation system described herein, described first acquisition module is additionally operable to the described available resources number generation current machine list of the various dimensions reported according to machine every described;
Described resource distribution module farther includes:
Sorting sub-module, for all tasks in described application, being ranked up according to the described demand number of resources of each described task application;
Number of resources calculating sub module, for calculating each described task demand number of resources in each dimension application according to institute's alignment sequence;
Resource distribution sub module, for finding out the machine list of the described demand number of resources meeting each described task from all machines, and distributes to the machine that described machine list is corresponding by described task.
According to resource allocation system described herein, if described resource distribution sub module is additionally operable to cannot find out the described machine list of the described demand number of resources meeting described task from all machines, then judge that the resource of described task is distributed unsuccessfully.
According to resource allocation system described herein, the weights of the described demand number of resources of described task application are described hard disk demand task groove number, described processor demand task groove number, described memory requirements task groove number and described network demand task groove number from high to low successively;
Described resource distribution sub module farther includes:
First allocation units, if the described hard disk demand task groove number for described task application, then from the described current machine list meeting described hard disk demand task groove number, described task is assigned on the first machine that described hard disk available task groove number is maximum, if maximum the first machine of described hard disk available task groove number has multiple, carries out next allocation units;
Second allocation units, if the described processor demand task groove number for described task application, if the described second machine only one of which that described processor available task groove number is maximum, then described task is assigned on described second machine, if maximum the second machine of described processor available task groove number has multiple, carries out next allocation units;
3rd allocation units, if the described memory requirements task groove number for described task application, if the described 3rd machine only one of which that described internal memory available task groove number is maximum, then described task is assigned on described 3rd machine, if maximum the 3rd machine of described internal memory available task groove number has multiple, carries out next allocation units;
4th allocation units, if the described network demand task groove number for described task application, if the described 4th machine only one of which that described network available task groove number is maximum, then described task is assigned on described 4th machine, if the 4th machine that described network available task groove number is maximum has multiple, then carry out next allocation units;
5th allocation units, for selecting the 5th machine from described 4th machine according to pre-defined rule, are assigned to described task on described 5th machine.
According to resource allocation system described herein, described 5th allocation units are additionally operable to be ranked up the machine name of described 4th machine by character sequence, and therefrom select described 5th machine of the first tagmeme.
The available resources number of the various dimensions reported by every machine in the application, then the demand number of resources of at least one dimension by application application; Then the available resources number according to the demand number of resources of described application and described machine, is allocated resources according to dimension. Described dimension preferably includes hard disk, processor, internal memory, this four dimensions of network. Whereby, machine resources can be carried out resource distribution according to various dimensions by the application in real-time computer system, it may be achieved the maximization of machine resources uses, and the task in real-time computer system that thoroughly solves distributes unbalanced problem.
Accompanying drawing explanation
Fig. 1 is the application structural representation based on the resource allocation system calculated in real time;
Fig. 2 is the structural representation that the application is preferably based on the resource allocation system of calculating in real time;
Fig. 3 is the application flow chart based on the resource allocation methods calculated in real time;
Fig. 4 is the flow chart that the application is preferably based on the resource allocation methods of calculating in real time.
Detailed description of the invention
In order to make the purpose of the application, technical scheme and advantage clearly understand, below in conjunction with drawings and Examples, the application is further elaborated. Should be appreciated that specific embodiment described herein is only in order to explain the application, be not used to limit the application.
In one typical configuration of the application, terminal, the equipment of service network and trusted party all include one or more processor (CPU), input/output interface, network interface and internal memory.
Internal memory potentially includes the forms such as the volatile memory in computer-readable medium, random access memory (RAM) and/or Nonvolatile memory, such as read only memory (ROM) or flash memory (flashRAM). Internal memory is the example of computer-readable medium.
Computer-readable medium includes permanent and impermanency, removable and non-removable media can by any method or technology to realize information storage. information can be computer-readable instruction, data structure, the module of program or other data. the example of the storage medium of computer includes, but it is not limited to phase transition internal memory (PRAM), static RAM (SRAM), dynamic random access memory (DRAM), other kinds of random access memory (RAM), read only memory (ROM), Electrically Erasable Read Only Memory (EEPROM), fast flash memory bank or other memory techniques, read-only optical disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tape, magnetic disk storage or other magnetic storage apparatus or any other non-transmission medium, can be used for the information that storage can be accessed by a computing device. according to defining herein, computer-readable medium does not include non-temporary computer readable media (transitorymedia), such as data signal and the carrier wave of modulation.
Fig. 1 is the application structural representation based on the resource allocation system calculated in real time, described resource allocation system 100 is preferably disposed on the host node (Nimbus) in the real time computation systems such as Storm or Jstorm, and described resource allocation system 100 at least includes the first acquisition module the 10, second acquisition module 20 and resource distribution module 30, wherein:
Described first acquisition module 10, for obtaining the available resources number of the various dimensions that every machine reports. Described machine be generally in real time computation system from node (Supervisor). Preferably, the first acquisition module 10, for obtaining the available resources number of the four dimensions that every machine reports, can use slot (task groove) number, processor can can use slot number with slot number and network with slot number, internal memory including hard disk. It is of course also possible to obtain the available resources number of two or three dimensions in this four dimensions.
Described second acquisition module 20, for obtaining the demand number of resources of at least one dimension of application application. Preferably, the second acquisition module 20 is for obtaining one of the hard disk demand slot number of application application, processor demand slot number, memory requirements slot number, network demand slot number or its combination in any.
Described resource distribution module 30, for the available resources number of the demand number of resources according to described application and described machine, is described allocated resources according to dimension. Preferably, resource distribution module 30 is for according to one of the hard disk demand slot number of application, processor demand slot number, memory requirements slot number, network demand slot number or its combination in any, and the hard disk of machine can, by one of slot number or its combination in any, can can be allocated resources according to dimension with slot number, network with slot number, internal memory with slot number, processor.
Machine resources is carried out resource distribution according to these multiple dimensions by the application, the resource of machine can be allowed to maximize and use. The task in real-time computer system that thoroughly solves distributes unbalanced problem. The application does not limit computer language, java/c++/c etc.
Fig. 2 is the structural representation that the application is preferably based on the resource allocation system of calculating in real time, described resource allocation system 100 is preferably disposed on the host node in the real time computation systems such as Storm or Jstorm, and described resource allocation system 100 at least includes the first acquisition module the 10, second acquisition module 20 and resource distribution module 30, wherein:
Described first acquisition module 10, for obtaining the available resources number of the various dimensions that every machine reports, and the available resources number generation current machine list of the various dimensions reported according to every machine. Described machine be generally in real time computation system from node. Typical resource type has 4 classes: hard disk, processor, internal memory, network. Preferably, the first acquisition module 10, for obtaining the available resources number of the four dimensions that every machine reports, can can can use slot number with slot number and network with slot number, internal memory with slot number, processor including hard disk. It is of course also possible to obtain the available resources number of two or three dimensions in this four dimensions. The available resources number of oneself can regularly be reported by every machine in real time computation system.
The algorithm of the available resources number of every described machine is preferably:
Hard disk can by slot number=hard disk number. Namely having several pieces of disks in machine, then whenever have several disk and can use slot, machine all can comprise a block system dish; Or
Processor can with slot number=(processor check figure 1) * processor radix. The processor assuming a machine is 24 cores, deducts the processor core that an operating system uses, is then 23, then is multiplied by processor radix, be defaulted as 2, then the processor of this machine can be 46 with slot number; Or
Internal memory can by slot number=(physical memory size/mono-internal memory slot size) * 80%. The physical memory assuming a machine is 96G, and an internal memory slot is sized to 1G, then the internal memory of this machine can be 75 with slot number; Or
Network can with slot number=maximum network bandwidth/each slot desire bandwidth. Described maximum network bandwidth refers to tests this machine by test program maximum network bandwidth is how many to host node machine, it is assumed that for 50M, according to mono-slot of 5M, then the network of this machine can be 10 with slot number.
Described second acquisition module 20, for obtaining the demand number of resources of at least one dimension of application application. Preferably, the second acquisition module 20 is for obtaining one of the hard disk demand slot number of application application, processor demand slot number, memory requirements slot number, network demand slot number or its combination in any. It is preferably to, described application includes multiple task (task) and worker assembly (process components), worker assembly refers to and runs the concrete process processing component logic, and in worker assembly, the thread of each spout/bolt is called a task. The acquiescence demand number of resources of each task is set to 1 processor slot, 1 internal memory slot, 0 hard disk slot. The acquiescence demand number of resources of worker assembly is set to 1 network slot. When the acquiescence demand number of resources of task and/or worker assembly cannot meet application demand, API (the ApplicationProgrammingInterface specified is called in application according to application demand, application programming interface), applying for more demand number of resources, namely application can apply for one of more hard disk demand slot number, processor demand slot number, memory requirements slot number, network demand slot number or its combination in any according to demand.
Described resource distribution module 30, for the available resources number of the demand number of resources according to application and machine, is described allocated resources according to dimension. Preferably, resource distribution module 30 is for according to one of the hard disk demand slot number of application, processor demand slot number, memory requirements slot number, network demand slot number or its combination in any, and the hard disk of machine can, by one of slot number or its combination in any, can can be described allocated resources according to dimension with slot number, network with slot number, internal memory with slot number, processor.
It is preferred that resource distribution module 30 farther includes:
Sorting sub-module 31, for all tasks in application, being ranked up according to the demand number of resources of each task application. Here, the weights of the demand number of resources setting task application are hard disk demand slot number, processor demand slot number, memory requirements slot number and network demand slot number from high to low successively. All tasks to JStorm, are ranked up, and sort according to the number of resources of task application, the task that weights are more high, priority allocation resource. Hard disk weights are the highest, and secondly, internal memory and network weight are last, and then the priority according to resource is balanced algorithm for processor.
Number of resources calculating sub module 32, for calculating each described task demand number of resources in each dimension application according to institute's alignment sequence.
Resource distribution sub module 33, is used for, and for finding out the machine list of the described demand number of resources meeting each described task from all machines, and described task is distributed to the machine that described machine list is corresponding. If resource distribution sub module 33 is additionally operable to cannot find out the described machine list of the described demand number of resources meeting described task from all machines, then judge that the resource of described task is distributed unsuccessfully.
It is preferred that resource distribution sub module 33 farther includes:
First allocation units, if the hard disk demand slot number for task application, then can sort with the sequence of slot number according to hard disk from the current machine list meeting hard disk demand slot number, described task is assigned on the first machine that described hard disk available task groove number is maximum, if hard disk can by maximum the first machine only one of which of slot number, then task is assigned on the first machine, if hard disk can have multiple with the first machine that slot number is maximum, carries out next round allocation step.
Second allocation units, if the processor demand slot number for task application, then the first machine can be sorted with the sequence of slot number according to processor, if processor can by maximum the second machine only one of which of slot number, then task is assigned on the second machine, if processor can have multiple with the second machine that slot number is maximum, carries out next round allocation step. If it is understood that described task does not apply for hard disk demand slot number, then directly performing the second allocation units, then the first machine now is equivalent to the machine that meets in the current machine list of processor demand slot number.
3rd allocation units, if the memory requirements slot number for task application, then the second machine can be sorted with the sequence of slot number according to internal memory, if internal memory can by maximum the 3rd machine only one of which of slot number, then task is assigned on the 3rd machine, if internal memory can have multiple with the 3rd machine that slot number is maximum, carries out next round allocation step. It is understandable that, if described task application hard disk demand slot number, but do not apply for processor demand slot number, then do not perform the second allocation units after having performed the first allocation units, and directly performing the 3rd allocation units, then the second machine now is equivalent to described first machine. If described task does not apply for hard disk demand slot number and processor demand slot number, then do not perform the first allocation units and the second allocation units, and directly performing the 3rd allocation units, then the second machine now is equivalent to the machine that meets in the current machine list of processor demand slot number.
4th allocation units, if the network demand slot number for task application, then the 3rd machine can be sorted with the sequence of slot number according to network, if network can by maximum the 4th machine only one of which of slot number, then task is assigned on the 4th machine, if network can have multiple with the 4th machine that slot number is maximum, then carry out next round allocation step. If it is understood that described task does not apply for one of hard disk demand slot number, processor demand slot number, memory requirements slot number or its combination in any, then after skipping corresponding allocation units, performing the 3rd allocation units.
5th allocation units, for selecting the 5th machine from the 4th machine according to pre-defined rule, are assigned to task on the 5th machine. Preferably, the 5th allocation units are additionally operable to be ranked up the machine name of the 4th machine by character sequence, and therefrom select the 5th machine of the first tagmeme.
Such as, because the resource prioritization of hard disk demand slot number is the highest, first will be allocated by hard disk dimension, if task does not apply for hard disk demand slot number, then without considering that hard disk slot sorts, if having applied for hard disk demand slot number, then to the machine list meeting task resource, according to hard disk demand slot number from many to being ranked up less, if the machine only one of which that hard disk demand slot number is maximum, then this task is just assigned on this machine, if the maximum machine of slot number exists multiple, then carry out according to lower dimension, further confirm that. Successively according to processor/internal memory/network these three dimension, further confirm that the machine at task run place. The machine that last time is chosen, carries out slot sequence according to this dimension, and machine priority maximum for slot is the highest, if the machine that slot number is maximum is one, then task is assigned with on this machine, if more than one of the machine that slot data are maximum, then determines whether by next dimension. If judging complete according to hard disk/processor/this four dimensions of internal memory/network, however it remains multiple stage machine, then machine name is ranked up by character sequence, takes First.
Fig. 3 is the application flow chart based on the resource allocation methods calculated in real time, and it can realize by the resource allocation system 100 that pass through as shown in Figure 1 or 2, has including step:
Step S301, obtains the available resources number of the various dimensions that every machine reports.
Described machine be generally in real time computation system from node (Supervisor). Preferably, obtain the available resources number of the four dimensions that every machine reports, slot (task groove) number, processor can be used can can to use slot number with slot number and network with slot number, internal memory including hard disk. It is of course also possible to obtain the available resources number of two or three dimensions in this four dimensions.
Step S302, obtains the demand number of resources of at least one dimension of application application.
Preferably, one of the hard disk demand slot number of application application, processor demand slot number, memory requirements slot number, network demand slot number or its combination in any are obtained.
Step S303, the available resources number according to the demand number of resources of described application and described machine, is described allocated resources according to dimension.
Preferably, according to one of the hard disk demand slot number of application, processor demand slot number, memory requirements slot number, network demand slot number or its combination in any, and the hard disk of machine can, by one of slot number or its combination in any, can can be allocated resources according to dimension with slot number, network with slot number, internal memory with slot number, processor.
Fig. 4 is the flow chart that the application is preferably based on the resource allocation methods of calculating in real time, and it can realize by the resource allocation system 100 that pass through as shown in Figure 2, has including step:
Step S401, obtains the available resources number of the various dimensions that every machine reports.
Preferably, described machine be generally in real time computation system from node. Typical resource type has 4 classes: hard disk, processor, internal memory, network. Preferably, obtain the available resources number of the four dimensions that every machine reports, can can can use slot number with slot number and network with slot number, internal memory with slot number, processor including hard disk. It is of course also possible to obtain the available resources number of two or three dimensions in this four dimensions. The available resources number of oneself can regularly be reported by every machine in real time computation system.
The algorithm of the available resources number of every described machine is preferably:
Hard disk can by slot number=hard disk number. Namely having several pieces of disks in machine, then whenever have several disk and can use slot, machine all can comprise a block system dish; And/or
Processor can with slot number=(processor check figure 1) * processor radix. The processor assuming a machine is 24 cores, deducts the processor core that an operating system uses, is then 23, then is multiplied by processor radix, be defaulted as 2, then the processor of this machine can be 46 with slot number; And/or
Internal memory can by slot number=(physical memory size/mono-internal memory slot size) * 80%. The physical memory assuming a machine is 96G, and an internal memory slot is sized to 1G, then the internal memory of this machine can be 75 with slot number; And/or
Network can with slot number=maximum network bandwidth/each slot desire bandwidth. Described maximum network bandwidth refers to tests this machine by test program maximum network bandwidth is how many to host node machine, it is assumed that for 50M, according to mono-slot of 5M, then the network of this machine can be 10 with slot number.
Step S402, the available resources number generation current machine list according to the various dimensions that every machine reports.
Step S403, obtains the demand number of resources of at least one dimension of application application.
Preferably, one of the hard disk demand slot number of application application, processor demand slot number, memory requirements slot number, network demand slot number or its combination in any are obtained. It is preferred that described application includes multiple task and worker assembly, worker assembly refers to and runs the concrete process processing component logic, and in worker assembly, the thread of each spout/bolt is called a task. The acquiescence demand number of resources of each task is set to 1 processor slot, 1 internal memory slot, 0 hard disk slot. The acquiescence demand number of resources of worker assembly is set to 1 network slot. When the acquiescence demand number of resources of task and/or worker assembly cannot meet application demand, application is called the API specified and is applied for more demand number of resources according to application demand, and namely application can apply for one of more hard disk demand slot number, processor demand slot number, memory requirements slot number, network demand slot number or its combination in any according to demand.
Step S404, to all tasks in application, is ranked up according to the demand number of resources of each task application.
Preferably, the weights of the demand number of resources of task application are hard disk demand slot number, processor demand slot number, memory requirements slot number and network demand slot number from high to low successively.
Step S405, calculates each described task demand number of resources in each dimension application according to institute's alignment sequence.
Step S406, it is judged that whether can find out the machine list of the demand number of resources meeting each task from all machines, if then performing step S407, otherwise performs step S408.
Step S407, if the machine list of the demand number of resources meeting each task can be found out from all machines, distributes to the machine that described machine list is corresponding by described task.
Preferably, this step farther includes:
First allocation step, if task application hard disk demand slot number, then can sort with the sequence of slot number according to hard disk from the current machine list meeting hard disk demand slot number, described task is assigned on the first machine that described hard disk available task groove number is maximum, if hard disk can by maximum the first machine only one of which of slot number, then task is assigned on the first machine, if hard disk can have multiple with the first machine that slot number is maximum, carries out next round allocation step.
Second allocation step, if task application processor demand slot number, then the first machine can be sorted with the sequence of slot number according to processor, if processor can by maximum the second machine only one of which of slot number, then task is assigned on the second machine, if processor can have multiple with the second machine that slot number is maximum, carries out next round allocation step. If it is understood that described task does not apply for hard disk demand slot number, then directly performing the second allocation step, then the first machine now is equivalent to the machine that meets in the current machine list of processor demand slot number.
3rd allocation step, if task application memory requirements slot number, then the second machine can be sorted with the sequence of slot number according to internal memory, if internal memory can by maximum the 3rd machine only one of which of slot number, then task is assigned on the 3rd machine, if internal memory can have multiple with the 3rd machine that slot number is maximum, carries out next round allocation step. It is understandable that, if described task application hard disk demand slot number, but do not apply for processor demand slot number, then do not perform the second allocation step after having performed the first allocation step, and directly performing the 3rd allocation step, then the second machine now is equivalent to described first machine. If described task does not apply for hard disk demand slot number and processor demand slot number, then do not perform the first allocation step and the second allocation step, and directly performing the 3rd allocation step, then the second machine now is equivalent to the machine that meets in the current machine list of processor demand slot number.
4th allocation step, if task application network demand slot number, then the 3rd machine can be sorted with the sequence of slot number according to network, if network can by maximum the 4th machine only one of which of slot number, then task is assigned on the 4th machine, if network can have multiple with the 4th machine that slot number is maximum, then carry out next round allocation step. If it is understood that described task does not apply for one of hard disk demand slot number, processor demand slot number, memory requirements slot number or its combination in any, then after skipping corresponding allocation step, performing the 3rd allocation step.
5th allocation step, selects the 5th machine from the 4th machine according to pre-defined rule, task is assigned on the 5th machine. Preferably, the machine name of the 4th machine is ranked up by character sequence, and therefrom selects the 5th machine of the first tagmeme.
Such as, because the resource prioritization of hard disk demand slot number is the highest, first will be allocated by hard disk dimension, if task does not apply for hard disk demand slot number, then without considering that hard disk slot sorts, if having applied for hard disk demand slot number, then to the machine list meeting task resource, according to hard disk demand slot number from many to being ranked up less, if the machine only one of which that hard disk demand slot number is maximum, then this task is just assigned on this machine, if the maximum machine of slot number exists multiple, then carry out according to lower dimension, further confirm that. Successively according to processor/internal memory/network these three dimension, further confirm that the machine at task run place. The machine that last time is chosen, carries out slot sequence according to this dimension, and machine priority maximum for slot is the highest, if the machine that slot number is maximum is one, then task is assigned with on this machine, if more than one of the machine that slot data are maximum, then determines whether by next dimension. If judging complete according to hard disk/processor/this four dimensions of internal memory/network, however it remains multiple stage machine, then machine name is ranked up by character sequence, takes First.
Step S408, if the machine list of the demand number of resources meeting each task cannot be able to be found out from all machines, then judges that the resource of task is distributed unsuccessfully.
Such as, in one application, wherein the demand number of resources of certain task is 4 processor slot, 8 internal memory slot, 0 hard disk slot. The available resource information in machine list according to each machine analyzes, the internal memory of current all machines can with slot number all less than 8, it will thus be seen that the demand number of resources of this task can be met currently without a machine, therefore resource allocation system 100 judges that the resource of this task is distributed unsuccessfully, now can directly feed back the information such as " resource is distributed unsuccessfully " or prompting " inadequate resource ".
In sum, the available resources number of the various dimensions reported by every machine in the application, then the demand number of resources of at least one dimension by application application; Then the available resources number according to the demand number of resources of described application and described machine, is allocated resources according to dimension. Described dimension preferably includes hard disk, processor, internal memory, this four dimensions of network. Whereby, machine resources can be carried out resource distribution according to various dimensions by the application in real-time computer system, it may be achieved the maximization of machine resources uses, and the task in real-time computer system that thoroughly solves distributes unbalanced problem.
Certainly; the application also can have other various embodiments; when without departing substantially from the application spirit and essence thereof; those of ordinary skill in the art are when can make various corresponding change and deformation according to the application, but these change accordingly and deformation all should belong to the scope of the claims appended by the application.
Claims (18)
1. the resource allocation methods based on calculating in real time, it is characterised in that including step has:
Obtain the available resources number of the various dimensions that every machine reports;
Obtain the demand number of resources of at least one dimension of application application;
Described demand number of resources according to described application and the described available resources number of described machine, be described allocated resources according to dimension.
2. resource allocation methods according to claim 1, it is characterised in that the step of the available resources number of the various dimensions that every machine of described acquisition reports includes:
Obtain the described available resources number of the various dimensions that every described machine reports, including one of hard disk available task groove number, processor available task groove number, internal memory available task groove number, network available task groove number or its combination in any.
3. resource allocation methods according to claim 2, it is characterised in that the step of the demand number of resources of at least one dimension of described acquisition application application includes:
Obtain one of the hard disk demand task groove number of described application application, processor demand task groove number, memory requirements task groove number, network demand task groove number or its combination in any;
The described available resources number of the described described demand number of resources according to described application and described machine, includes according to the step that dimension is described allocated resources:
According to one of the described hard disk demand task groove number of described application, described processor demand task groove number, described memory requirements task groove number, described network demand task groove number or its combination in any, and one of the described hard disk available task groove number of described machine, described processor available task groove number, described internal memory available task groove number, described network available task groove number or its combination in any, it is described allocated resources according to dimension.
4. resource allocation methods according to claim 3, it is characterised in that described hard disk available task groove number=hard disk number; Or
Described processor available task groove number=(processor check figure 1) * processor radix; Or
Described internal memory available task groove number=(physical memory size/mono-internal memory task groove size) * 80%; Or
Described network available task groove number=maximum network bandwidth/each task groove desire bandwidth.
5. resource allocation methods according to claim 3, it is characterised in that described application includes multiple task and process components; When the acquiescence demand number of resources of described task or described process components cannot meet application demand, described application is according to the more described demand number of resources of described application demand application.
6. resource allocation methods according to claim 3, it is characterised in that the step of the demand number of resources of at least one dimension of described acquisition application application also includes:
Described available resources number generation current machine list according to the various dimensions that machine every described reports;
The described available resources number of the described described demand number of resources according to described application and described machine, includes according to the step that dimension is described allocated resources:
To all tasks in described application, it is ranked up according to the described demand number of resources of each described task application;
Each described task demand number of resources in each dimension application is calculated according to institute's alignment sequence;
From all machines, find out the machine list of the described demand number of resources meeting each described task, and described task is distributed to the machine that described machine list is corresponding.
7. resource allocation methods according to claim 6, it is characterized in that, the described machine list finding out the described demand number of resources meeting each described task from all machines, and the step that described task is distributed to machine corresponding to described machine list also includes:
If the described machine list of the described demand number of resources meeting described task cannot be found out from all machines, then judge that the resource of described task is distributed unsuccessfully.
8. resource allocation methods according to claim 6, it is characterized in that, the weights of the described demand number of resources of described task application are described hard disk demand task groove number, described processor demand task groove number, described memory requirements task groove number and described network demand task groove number from high to low successively;
The described machine list finding out the described demand number of resources meeting each described task from all machines, and the step that described task is distributed to machine corresponding to described machine list includes:
First distribution sub-step, if the described hard disk demand task groove number of described task application, then from the described current machine list meeting described hard disk demand task groove number, described task is assigned on the first machine that described hard disk available task groove number is maximum, if maximum the first machine of described hard disk available task groove number has multiple, carries out next round allocation step;
Second distribution sub-step, if the described processor demand task groove number of described task application, if the described second machine only one of which that described processor available task groove number is maximum, then described task is assigned on described second machine, if maximum the second machine of described processor available task groove number has multiple, carries out next round allocation step;
3rd distribution sub-step, if the described memory requirements task groove number of described task application, if the described 3rd machine only one of which that described internal memory available task groove number is maximum, then described task is assigned on described 3rd machine, if maximum the 3rd machine of described internal memory available task groove number has multiple, carries out next round allocation step;
4th distribution sub-step, if the described network demand task groove number of described task application, if the described 4th machine only one of which that described network available task groove number is maximum, then described task is assigned on described 4th machine, if the 4th machine that described network available task groove number is maximum has multiple, then carry out next round allocation step;
5th distribution sub-step, selects the 5th machine from described 4th machine according to pre-defined rule, described task is assigned on described 5th machine.
9. resource allocation methods according to claim 8, it is characterised in that described select the step of the 5th machine to include from described 4th machine according to pre-defined rule:
The machine name of described 4th machine is ranked up by character sequence, and therefrom selects described 5th machine of the first tagmeme.
10. the resource allocation system based on calculating in real time, it is characterised in that include:
First acquisition module, for obtaining the available resources number of the various dimensions that every machine reports;
Second acquisition module, for obtaining the demand number of resources of at least one dimension of application application;
Resource distribution module, for the described available resources number of the described demand number of resources according to described application and described machine, is described allocated resources according to dimension.
11. resource allocation system according to claim 10, it is characterized in that, described first acquisition module is for obtaining the described available resources number of the various dimensions that every described machine reports, including one of hard disk available task groove number, processor available task groove number, internal memory available task groove number, network available task groove number or its combination in any.
12. resource allocation system according to claim 11, it is characterized in that, described second acquisition module is for obtaining one of the hard disk demand task groove number of described application application, processor demand task groove number, memory requirements task groove number, network demand task groove number or its combination in any;
Described resource distribution module is for according to one of the described hard disk demand task groove number of described application, described processor demand task groove number, described memory requirements task groove number, described network demand task groove number or its combination in any, and one of the described hard disk available task groove number of described machine, described processor available task groove number, described internal memory available task groove number, described network available task groove number or its combination in any, it is described allocated resources according to dimension.
13. resource allocation system according to claim 12, it is characterised in that described hard disk available task groove number=hard disk number; Or
Described processor available task groove number=(processor check figure 1) * processor radix; Or
Described internal memory available task groove number=(physical memory size/mono-internal memory task groove size) * 80%; Or
Described network available task groove number=maximum network bandwidth/each task groove desire bandwidth.
14. resource allocation system according to claim 12, it is characterised in that described application includes multiple task and process components; When the acquiescence demand number of resources of described task or described process components cannot meet application demand, described application is according to the more described demand number of resources of described application demand application.
15. resource allocation system according to claim 12, it is characterised in that described first acquisition module is additionally operable to the described available resources number generation current machine list of the various dimensions reported according to machine every described;
Described resource distribution module farther includes:
Sorting sub-module, for all tasks in described application, being ranked up according to the described demand number of resources of each described task application;
Number of resources calculating sub module, for calculating each described task demand number of resources in each dimension application according to institute's alignment sequence;
Resource distribution sub module, for finding out the machine list of the described demand number of resources meeting each described task from all machines, and distributes to the machine that described machine list is corresponding by described task.
16. resource allocation system according to claim 15, it is characterized in that, if described resource distribution sub module is additionally operable to cannot find out the described machine list of the described demand number of resources meeting described task from all machines, then judge that the resource of described task is distributed unsuccessfully.
17. resource allocation system according to claim 15, it is characterized in that, the weights of the described demand number of resources of described task application are described hard disk demand task groove number, described processor demand task groove number, described memory requirements task groove number and described network demand task groove number from high to low successively;
Described resource distribution sub module farther includes:
First allocation units, if the described hard disk demand task groove number for described task application, then from the described current machine list meeting described hard disk demand task groove number, described task is assigned on the first machine that described hard disk available task groove number is maximum, if maximum the first machine of described hard disk available task groove number has multiple, carries out next allocation units;
Second allocation units, if the described processor demand task groove number for described task application, if the described second machine only one of which that described processor available task groove number is maximum, then described task is assigned on described second machine, if maximum the second machine of described processor available task groove number has multiple, carries out next allocation units;
3rd allocation units, if the described memory requirements task groove number for described task application, if the described 3rd machine only one of which that described internal memory available task groove number is maximum, then described task is assigned on described 3rd machine, if maximum the 3rd machine of described internal memory available task groove number has multiple, carries out next allocation units;
4th allocation units, if the described network demand task groove number for described task application, if the described 4th machine only one of which that described network available task groove number is maximum, then described task is assigned on described 4th machine, if the 4th machine that described network available task groove number is maximum has multiple, then carry out next allocation units;
5th allocation units, for selecting the 5th machine from described 4th machine according to pre-defined rule, are assigned to described task on described 5th machine.
18. resource allocation system according to claim 17, it is characterised in that described 5th allocation units are additionally operable to be ranked up the machine name of described 4th machine by character sequence, and therefrom select described 5th machine of the first tagmeme.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410612356.0A CN105630601A (en) | 2014-11-03 | 2014-11-03 | Resource allocation method and system based on real-time computing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410612356.0A CN105630601A (en) | 2014-11-03 | 2014-11-03 | Resource allocation method and system based on real-time computing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105630601A true CN105630601A (en) | 2016-06-01 |
Family
ID=56045578
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410612356.0A Pending CN105630601A (en) | 2014-11-03 | 2014-11-03 | Resource allocation method and system based on real-time computing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105630601A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107357649A (en) * | 2017-05-27 | 2017-11-17 | 北京奇艺世纪科技有限公司 | The determination method, apparatus and electronic equipment of system resource deployment strategy |
| CN110784512A (en) * | 2019-09-17 | 2020-02-11 | 同济大学 | Airborne dynamic cloud system and real-time response resource allocation method thereof |
| CN112905342A (en) * | 2021-02-07 | 2021-06-04 | 广州虎牙科技有限公司 | Resource scheduling method, device, equipment and computer readable storage medium |
| CN113485827A (en) * | 2021-06-30 | 2021-10-08 | 上海森浦信息服务有限公司 | Data processing method, device and storage medium for realizing automatic configuration issuing function |
| CN114254966A (en) * | 2022-03-02 | 2022-03-29 | 爱彼希科技(北京)有限公司 | Method for matching resources by using dimension group |
| CN115665157A (en) * | 2022-11-14 | 2023-01-31 | 杭州谐云科技有限公司 | Balanced scheduling method and system based on application resource types |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050160428A1 (en) * | 2004-01-20 | 2005-07-21 | International Business Machines Corporation | Application-aware system that dynamically partitions and allocates resources on demand |
| CN1921444A (en) * | 2005-08-24 | 2007-02-28 | 上海原动力通信科技有限公司 | Method for classified package dispatching and resource distributing based on service quality |
| US20080066072A1 (en) * | 2006-07-31 | 2008-03-13 | Accenture Global Services Gmbh | Work Allocation Model |
| CN102096461A (en) * | 2011-01-13 | 2011-06-15 | 浙江大学 | Energy-saving method of cloud data center based on virtual machine migration and load perception integration |
| CN102902587A (en) * | 2011-07-28 | 2013-01-30 | 中国移动通信集团四川有限公司 | Distribution type task scheduling method, distribution type task scheduling system and distribution type task scheduling device |
| CN103024078A (en) * | 2012-12-31 | 2013-04-03 | 无锡城市云计算中心有限公司 | Resource allocation method and device in cloud computing environment |
| CN103853618A (en) * | 2014-03-06 | 2014-06-11 | 南京理工大学 | Resource allocation method with minimized cloud system cost based on expiration date drive |
| CN103902384A (en) * | 2014-03-28 | 2014-07-02 | 华为技术有限公司 | Method and device for allocating physical machines for virtual machines |
-
2014
- 2014-11-03 CN CN201410612356.0A patent/CN105630601A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050160428A1 (en) * | 2004-01-20 | 2005-07-21 | International Business Machines Corporation | Application-aware system that dynamically partitions and allocates resources on demand |
| CN1921444A (en) * | 2005-08-24 | 2007-02-28 | 上海原动力通信科技有限公司 | Method for classified package dispatching and resource distributing based on service quality |
| US20080066072A1 (en) * | 2006-07-31 | 2008-03-13 | Accenture Global Services Gmbh | Work Allocation Model |
| CN102096461A (en) * | 2011-01-13 | 2011-06-15 | 浙江大学 | Energy-saving method of cloud data center based on virtual machine migration and load perception integration |
| CN102902587A (en) * | 2011-07-28 | 2013-01-30 | 中国移动通信集团四川有限公司 | Distribution type task scheduling method, distribution type task scheduling system and distribution type task scheduling device |
| CN103024078A (en) * | 2012-12-31 | 2013-04-03 | 无锡城市云计算中心有限公司 | Resource allocation method and device in cloud computing environment |
| CN103853618A (en) * | 2014-03-06 | 2014-06-11 | 南京理工大学 | Resource allocation method with minimized cloud system cost based on expiration date drive |
| CN103902384A (en) * | 2014-03-28 | 2014-07-02 | 华为技术有限公司 | Method and device for allocating physical machines for virtual machines |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107357649A (en) * | 2017-05-27 | 2017-11-17 | 北京奇艺世纪科技有限公司 | The determination method, apparatus and electronic equipment of system resource deployment strategy |
| CN107357649B (en) * | 2017-05-27 | 2020-07-03 | 北京奇艺世纪科技有限公司 | Method and device for determining system resource deployment strategy and electronic equipment |
| CN110784512A (en) * | 2019-09-17 | 2020-02-11 | 同济大学 | Airborne dynamic cloud system and real-time response resource allocation method thereof |
| CN112905342A (en) * | 2021-02-07 | 2021-06-04 | 广州虎牙科技有限公司 | Resource scheduling method, device, equipment and computer readable storage medium |
| CN112905342B (en) * | 2021-02-07 | 2024-03-01 | 广州虎牙科技有限公司 | Resource scheduling method, device, equipment and computer readable storage medium |
| CN113485827A (en) * | 2021-06-30 | 2021-10-08 | 上海森浦信息服务有限公司 | Data processing method, device and storage medium for realizing automatic configuration issuing function |
| CN113485827B (en) * | 2021-06-30 | 2023-04-14 | 上海森浦信息服务有限公司 | Data processing method, device and storage medium for realizing automatic configuration issuing function |
| CN114254966A (en) * | 2022-03-02 | 2022-03-29 | 爱彼希科技(北京)有限公司 | Method for matching resources by using dimension group |
| CN115665157A (en) * | 2022-11-14 | 2023-01-31 | 杭州谐云科技有限公司 | Balanced scheduling method and system based on application resource types |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108595157B (en) | Block chain data processing method, device, equipment and storage medium | |
| CN105630601A (en) | Resource allocation method and system based on real-time computing | |
| CN109005125B (en) | Dynamic current limiting method, device and system | |
| US10572285B2 (en) | Method and apparatus for elastically scaling virtual machine cluster | |
| US20170329643A1 (en) | Distributed node intra-group task scheduling method and system | |
| US20210303344A1 (en) | Task processing method, electronic device, and computer program product | |
| US10320701B1 (en) | Aggregating resource requests | |
| WO2014035309A1 (en) | Re-configuration in cloud computing environments | |
| CN112905342A (en) | Resource scheduling method, device, equipment and computer readable storage medium | |
| CN106980571A (en) | The construction method and equipment of a kind of test use cases | |
| CN111782383A (en) | Task allocation method, server, electronic terminal and computer readable storage medium | |
| CN115033340A (en) | Host selection method and related device | |
| CN107784195A (en) | Data processing method and device | |
| CN113434320A (en) | Information system fault positioning method, device, equipment and storage medium | |
| CN104281587B (en) | A kind of method and device for establishing connection | |
| CN114844791B (en) | Cloud service automatic management and distribution method and system based on big data and storage medium | |
| CN113795032B (en) | Method and device for judging invisible faults of indoor division, storage medium and equipment | |
| CN113867736B (en) | Deployment scheme generation method and device | |
| US10776332B2 (en) | Managing cooperative computer software applications | |
| CN115328612A (en) | Resource allocation method, device, equipment and storage medium | |
| CN111800446B (en) | Scheduling processing method, device, equipment and storage medium | |
| CN109445863A (en) | A kind of data processing method based on FPGA, device, equipment and medium | |
| CN114416357A (en) | Method and device for creating container group, electronic equipment and medium | |
| CN114070855B (en) | Resource allocation method, resource allocation device, resource allocation system, and storage medium | |
| KR101836743B1 (en) | Mobile device selection method for failed task reallocation in mobile distributed computing system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160601 |