CN112148448A - Resource allocation method, device, equipment and computer readable medium - Google Patents
Resource allocation method, device, equipment and computer readable medium Download PDFInfo
- Publication number
- CN112148448A CN112148448A CN202011003579.9A CN202011003579A CN112148448A CN 112148448 A CN112148448 A CN 112148448A CN 202011003579 A CN202011003579 A CN 202011003579A CN 112148448 A CN112148448 A CN 112148448A
- Authority
- CN
- China
- Prior art keywords
- resources
- resource
- executed
- task
- computing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000013468 resource allocation Methods 0.000 title claims abstract description 26
- 230000008859 change Effects 0.000 claims abstract description 28
- 230000004044 response Effects 0.000 claims abstract description 16
- 238000004364 calculation method Methods 0.000 claims description 34
- 238000004590 computer program Methods 0.000 claims description 10
- 238000010586 diagram Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 239000003795 chemical substances by application Substances 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Stored Programmes (AREA)
Abstract
The embodiment of the disclosure discloses a resource allocation method, a resource allocation device, an electronic device and a computer readable medium. One embodiment of the method comprises: responding to the resource demand change of at least one native service in the cluster, and acquiring the usable resources of the running computing task set in the cluster; in response to the used resources of the running computing task set being larger than the available resources, calculating the weight of each running computing task in the running computing task set; based on the weights, terminating a first number of the executed computing tasks so that the updated used resources meet the resource demand change. The implementation mode realizes the self-adaptive deployment of the operation, so that the running of the native service is not influenced while the resources are fully utilized.
Description
Technical Field
Embodiments of the present disclosure relate to the field of computer technologies, and in particular, to a resource allocation method, apparatus, device, and computer-readable medium.
Background
YARN (Another Resource coordinator) is a basic platform for cluster Resource management and scheduling, and supports scheduling of various computing jobs in a large-scale cluster. Native services of an online machine (services that are already running on the machine) typically occupy only a portion of the machine's resources, rather than all. Machine resources not occupied by native services are available for hybrid deployment YARN jobs.
But if the deployed YARN job uses too many machine resources, it will affect the resource usage of the native service. If the deployed YARN job uses too little machine resources, the resources are not fully provided for the YARN job and the machine resources are not fully utilized.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
Some embodiments of the present disclosure propose resource allocation methods, apparatuses, devices and computer readable media to solve the technical problems mentioned in the background section above.
In a first aspect, some embodiments of the present disclosure provide a resource allocation method, including: responding to the resource demand change of at least one native service in the cluster, and acquiring the usable resources of the running computing task set in the cluster; in response to the used resources of the running computing task set being larger than the available resources, calculating the weight of each running computing task in the running computing task set; based on the weights, terminating a first number of the executed computing tasks so that the updated used resources meet the resource demand change.
In a second aspect, some embodiments of the present disclosure provide an apparatus for resource allocation, the apparatus comprising: the acquisition unit is configured to respond to the resource demand change of at least one native service in the cluster, and acquire usable resources of a running calculation task set in the cluster; a computing unit configured to compute a weight of each executed computing task in the set of executed computing tasks in response to a used resource of the set of executed computing tasks being greater than the available resource; a terminating unit configured to terminate a first number of the executed computing tasks based on the weight so that the updated used resources satisfy the resource demand change.
In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, which when executed by one or more processors, cause the one or more processors to implement the method as described in any of the implementations of the first aspect.
In a fourth aspect, some embodiments of the disclosure provide a computer readable medium having a computer program stored thereon, where the program when executed by a processor implements a method as described in any of the implementations of the first aspect.
One of the above-described various embodiments of the present disclosure has the following advantageous effects: the self-adaptive deployment of the operation is realized, so that the running of the native service is not influenced while the resources are fully utilized. In particular, the inventors have found that the problems described in the background arise from the related art because there is no flexible way to adjust the deployment of computing tasks. Based on this, the scheme provides an adaptive computing task deployment method. Specifically, when the resource demand of at least one native service in the cluster changes, the available resources of the running computing task set in the cluster are obtained. And if the used resources of the running computing task set exceed the available resources, calculating the weight of each running computing task in the running computing task set. And then, based on the weight, terminating the first number of the running computing tasks so that the updated used resources meet the resource demand change.
Drawings
The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale.
Fig. 1 and 2 are schematic diagrams of an application scenario of a resource allocation method according to some embodiments of the present disclosure;
fig. 3 is a flow diagram of some embodiments of a resource allocation method according to the present disclosure;
FIG. 4 is a flow diagram of further embodiments of a resource allocation method according to the present disclosure;
fig. 5 is a schematic structural diagram of some embodiments of a resource allocation apparatus according to the present disclosure;
FIG. 6 is a schematic structural diagram of an electronic device suitable for use in implementing some embodiments of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings. The embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict.
It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 and 2 are schematic diagrams illustrating an application scenario in which the resource allocation method of some embodiments of the present disclosure may be applied.
As shown in FIG. 1, a computing device 101 may first obtain available resources 102 of a set of computing tasks (including computing task 1 through computing task 6) already running in a cluster described above in response to a change in resource demand of at least one native service in the cluster. In the present application scenario, the at least one native service includes "service 1" and "service 2". Where the resource requirements of service 2 change. Then, in response to the used resource 103 of the set of executed computing tasks being larger than the available resource 102, the weight of each executed computing task in the set of executed computing tasks is calculated. In the present application scenario, the used resource 103 is 55M larger than the usable resource 102 is 50M. On this basis, the computing device 101 calculates a weight of each computing task in the computing task set. In the present application scenario, as an example, the content indicated by reference numeral 104 is the weight of "calculation task 5", and its value is "2". Finally, as shown in FIG. 2, the computing device 101 may terminate a first number of the executed computing tasks based on the weights to cause the updated used resources 103 to meet the resource demand change. In the context of this application, the first number is "1". The terminated running calculation task is "calculation task 5" with the smallest weight. The updated used resource 201 is 46M, and the change in the resource demand is satisfied.
The computing device 101 may be hardware or software. When the computing device is hardware, it may be implemented as a distributed cluster composed of a plurality of servers or electronic devices, or may be implemented as a single server or a single electronic device. When the computing device is embodied as software, it may be implemented as multiple pieces of software or software modules, for example, to provide distributed services, or as a single piece of software or software module. And is not particularly limited herein.
It should be understood that the number of computing devices 101 in FIG. 1 or FIG. 2 is merely illustrative. There may be any number of computing devices 101, as desired for implementation.
With continued reference to fig. 3, a flow 300 of some embodiments of a resource allocation method according to the present disclosure is shown. The resource allocation method comprises the following steps:
In some embodiments, the above-described executed computing task may be any task other than a native service in the execution body of the resource allocation method.
In some embodiments, the usable resource may be a difference between a total resource of an execution subject of the resource allocation method and a resource occupied by the native service.
In one of some embodimentsThese areIn an alternative implementation manner, the executing entity may first obtain the used resources of the running computing task set. Then, the reserved buffer resources and allocable free resources are obtained. And finally, determining the difference between the sum of the allocable free resources and the used resources and the reserved buffer resources as the usable resources. With the embodiments of the present implementation, by introducing the reserved buffer resources, it is not necessary to terminate the executed computing tasks in the set of executed computing tasks when the change in the resource demand of the native service does not exceed the above-mentioned reserved buffer resources. And the waste of computing resources caused by frequent scheduling is avoided.
As an example, the allocatable free resources described above may be unallocated resources.
In some optional implementation manners of some embodiments, the allocable idle resource may also be a sum of an unallocated resource and an unused resource in resources occupied by a computing task in a stable running state. By adopting the embodiments of the present implementation, the resources in the execution main body are fully utilized by using the unused resources in the resources occupied by the computing task with stable running state as allocable idle resources.
As an example, the above-described running state-stable computing task may be a preset some kind of native service.
Optionally, the computing task with stable operating state may further include: and within the preset time length for starting operation, the native service with the occupied resource fluctuation smaller than a preset threshold value and/or the native service with the operation time length exceeding a time length threshold value. Wherein the duration threshold is determined based on an average value of the fluctuation range of the native service occupancy resource. With the embodiments of the present implementation, the native service with stable operation state can be determined more accurately.
In some embodiments, the weight of the computing task may be determined by at least one of: the size of resources occupied by the computing task, the running time of the computing task and the total estimated running time of the computing task.
In some optional implementation manners of some embodiments, the execution subject may further calculate, for each executed computing task, a weight of the executed computing task based on a preset weight of one or more of task priority, starting time duration, resource usage rate, and task type. The attributes of the calculation tasks in the implementation mode are easy to obtain, and meanwhile, the importance degree of the corresponding calculation tasks can be accurately represented. As an example, the task type may be whether the computing task is an AM (application Master).
In some embodiments, the weight of the computing task may be a weighted sum of the values of the computing task attributes.
In some embodiments, the first number may be a preset number.
In some embodiments, the first number may also be determined based on used and available resources. For example, a ratio of a difference between the available resource and the used resource to an average occupied resource by the deployed computing task may be determined as the first number.
In some embodiments, the execution agent may terminate a first number of the executed computing tasks by marking resources occupied by the first number of the executed computing tasks as unused.
The method provided by some embodiments of the present disclosure achieves adaptive deployment of jobs, so that the running of native services is not affected while resources are fully utilized.
With further reference to fig. 4, a flow 400 of further embodiments of a resource allocation method is illustrated. The process 400 of the resource allocation method includes the following steps:
In some embodiments, the allocable free resources may be unallocated resources.
In some optional implementation manners of some embodiments, the allocable idle resource may also be a sum of an unallocated resource and an unused resource in resources occupied by a computing task in a stable running state. With these embodiments of the present implementation, the resources in the execution body described above are fully utilized.
In some embodiments, the above-described running state-stable computing task may be a preset type of native service.
In some optional implementations of some embodiments, the running state stable computing task may further include: and within the preset time length of starting operation, occupying the native service of which the fluctuation of the resource is less than the preset threshold value and/or the native service after the first time length of starting operation. Wherein the first duration is determined based on an average value of fluctuation amplitudes of at least one native service occupancy resource. With the embodiments of the present implementation, the native service with stable operation state can be determined more accurately.
In some embodiments, the reserved buffer resource may be a preset value.
In some optional implementation manners of some embodiments, the reserved buffer resources may also be directly proportional to the average fluctuation amplitude of the idle resources, and the ratio is a preset value. By adopting the embodiments of the implementation mode, the size of the reserved buffer resources is more reasonable.
In some embodiments, the execution agent may determine, as the weight of the executed computing task, a sum of products of a value of each of the at least one target attribute of the executed computing task and a preset weight of the target attribute.
In some embodiments, the execution agent may further determine, as the weight of the executed computing task, a product of a value of each target attribute of the at least one target attribute of the executed computing task and a preset weight of the target attribute.
In some embodiments, according to actual needs, as an example, the execution subject may sequentially terminate the deployed computing tasks with the highest weight in the running computing task set.
In some embodiments, according to actual needs, as an example, the execution main body may further terminate the executed computing task with the smallest weight in the deployed computing task set in turn.
As can be seen from fig. 4, compared with the description of some embodiments corresponding to fig. 3, the process 400 of the resource allocation method in some embodiments corresponding to fig. 4 embodies the steps of determining available resources and calculating the above-mentioned operated calculated task weight based on the preset weight of one or more of task priority, starting time, resource usage rate, and task type. Thus, by introducing reserved buffer resources, the solutions described in the embodiments may not terminate a run computing task in a set of run computing tasks when the resource demand of the native service does not vary beyond the reserved buffer resources described above. And the waste of computing resources caused by frequent scheduling is avoided. And determining the sum of the unallocated resources and the idle resources in the resources occupied by the computing task in a stable running state as the allocable idle resources to fully utilize the resources in the execution main body. And by introducing the weight of the computing task, resources can be occupied by the relatively more important computing task.
With further reference to fig. 5, as an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a resource allocation apparatus, which correspond to those shown in fig. 3, and which may be applied in various electronic devices.
As shown in fig. 5, the resource allocation apparatus 500 of some embodiments includes: an acquisition unit 501, a calculation unit 502, and a termination unit 503. The acquiring unit 501 is configured to, in response to a change in resource demand of at least one native service in a cluster, acquire a usable resource of a set of running computing tasks in the cluster; a calculating unit 502 configured to calculate a weight of each executed calculation task in the executed calculation task set in response to a used resource of the executed calculation task set being greater than the available resource; a terminating unit 503, configured to terminate a first number of the executed computing tasks based on the weight, so that the updated used resources meet the resource demand change.
In an optional implementation of some embodiments, the obtaining unit 501 is further configured to include: acquiring the used resources of the running computing task set; acquiring allocable idle resources and reserved buffer resources of a plurality of physical machines in a cluster; and determining the difference between the sum of the allocable free resources and the used resources and the reserved buffer resources as the usable resources.
In an alternative implementation of some embodiments, the allocable free resources include unused resources including unallocated resources and computing tasks with stable running states.
In an alternative implementation of some embodiments, the reserved buffer resources are determined based on the fluctuation amplitude of the allocable free resources described above.
In an alternative implementation of some embodiments, running a state-stable computational task includes: and in the preset time length for starting to operate, actually using the calculation tasks with the resource fluctuation less than the preset threshold value and/or the calculation tasks with the operation time length exceeding the time length threshold value.
In an alternative implementation of some embodiments, the duration threshold is determined based on an average of the actual usage of the resource fluctuation amplitude by the computing task.
In an optional implementation of some embodiments, the computing unit 502 is further configured to: and for each running calculation task, calculating the weight of the running calculation task based on one or more preset weights in task priority, starting time, resource utilization rate and task type.
It will be understood that the elements described in the apparatus 500 correspond to various steps in the method described with reference to fig. 3. Thus, the operations, features and resulting advantages described above with respect to the method are also applicable to the apparatus 500 and the units included therein, and are not described herein again.
Referring now to FIG. 6, a block diagram of an electronic device (e.g., the computing device of FIG. 1) 600 suitable for use in implementing some embodiments of the present disclosure is shown. The electronic device in some embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle-mounted terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 6, electronic device 600 may include a processing means (e.g., central processing unit, graphics processor, etc.) 601 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data necessary for the operation of the electronic apparatus 600 are also stored. The processing device 601, the ROM 602, and the RAM603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
Generally, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While fig. 6 illustrates an electronic device 600 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in fig. 6 may represent one device or may represent multiple devices as desired.
In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In some such embodiments, the computer program may be downloaded and installed from a network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. The computer program, when executed by the processing device 601, performs the above-described functions defined in the methods of some embodiments of the present disclosure.
It should be noted that the computer readable medium described in some embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: responding to the resource demand change of at least one native service in the cluster, and acquiring the usable resources of the running computing task set in the cluster; in response to the used resources of the running computing task set being larger than the available resources, calculating the weight of each running computing task in the running computing task set; based on the weights, terminating a first number of the executed computing tasks so that the updated used resources meet the resource demand change.
Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in some embodiments of the present disclosure may be implemented by software, and may also be implemented by hardware. The described units may also be provided in a processor, and may be described as: a processor includes a first determination unit, a termination unit, and an update unit. Where the names of these units do not in some cases constitute a limitation on the units themselves, for example, an update unit may also be described as a "unit that updates a set of deployed computing tasks".
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
According to one or more embodiments of the present disclosure, there is provided a resource allocation method including: responding to the resource demand change of at least one native service in the cluster, and acquiring the usable resources of the running computing task set in the cluster; in response to the used resources of the running computing task set being larger than the available resources, calculating the weight of each running computing task in the running computing task set; based on the weights, terminating a first number of the executed computing tasks so that the updated used resources meet the resource demand change.
According to one or more embodiments of the present disclosure, acquiring available resources of a set of computing tasks already running in the cluster includes: acquiring the used resources of the running computing task set; acquiring allocable idle resources and reserved buffer resources of a plurality of physical machines in a cluster; and determining the difference between the sum of the allocable free resources and the used resources and the reserved buffer resources as the usable resources.
In accordance with one or more embodiments of the present disclosure, allocable idle resources include unused resources including unallocated resources and computing tasks that are operating stably.
According to one or more embodiments of the present disclosure, the reserved buffer resources are determined based on the fluctuation amplitude of the allocable idle resources described above.
According to one or more embodiments of the present disclosure, running a state-stable computing task includes: and in the preset time length for starting to operate, actually using the calculation tasks with the resource fluctuation less than the preset threshold value and/or the calculation tasks with the operation time length exceeding the time length threshold value.
According to one or more embodiments of the present disclosure, the duration threshold is determined based on an average value of the fluctuation amplitudes of the resources actually used by the above-mentioned computing task.
According to one or more embodiments of the present disclosure, calculating a weight of each executed computing task in the set of executed computing tasks further includes: and for each running calculation task, calculating the weight of the running calculation task based on one or more preset weights in task priority, starting time, resource utilization rate and task type.
According to one or more embodiments of the present disclosure, there is provided a resource allocation apparatus including: the acquisition unit is configured to respond to the resource demand change of at least one native service in the cluster, and acquire usable resources of a running calculation task set in the cluster; a computing unit configured to compute a weight of each executed computing task in the set of executed computing tasks in response to a used resource of the set of executed computing tasks being greater than the available resource; a terminating unit configured to terminate a first number of the executed computing tasks based on the weight so that the updated used resources satisfy the resource demand change.
According to one or more embodiments of the present disclosure, the obtaining unit is further configured to include: acquiring the used resources of the running computing task set; acquiring allocable idle resources and reserved buffer resources of a plurality of physical machines in a cluster; and determining the difference between the sum of the allocable free resources and the used resources and the reserved buffer resources as the usable resources.
In accordance with one or more embodiments of the present disclosure, allocable idle resources include unused resources including unallocated resources and computing tasks that are operating stably.
According to one or more embodiments of the present disclosure, the reserved buffer resources are determined based on the fluctuation amplitude of the allocable idle resources described above.
According to one or more embodiments of the present disclosure, running a state-stable computing task includes: and in the preset time length for starting to operate, actually using the calculation tasks with the resource fluctuation less than the preset threshold value and/or the calculation tasks with the operation time length exceeding the time length threshold value.
According to one or more embodiments of the present disclosure, the duration threshold is determined based on an average value of the fluctuation amplitudes of the resources actually used by the above-mentioned computing task.
According to one or more embodiments of the present disclosure, the computing unit is further configured to: and for each running calculation task, calculating the weight of the running calculation task based on one or more preset weights in task priority, starting time, resource utilization rate and task type.
According to one or more embodiments of the present disclosure, there is provided an electronic device including: one or more processors; a storage device having one or more programs stored thereon which, when executed by one or more processors, cause the one or more processors to implement a method as in any above.
According to one or more embodiments of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the method as any one of the above.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is made without departing from the inventive concept as defined above. For example, the above features and (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.
Claims (10)
1. A method of resource allocation, comprising:
responding to the resource demand change of at least one native service in the cluster, and acquiring usable resources of a running computing task set in the cluster;
in response to the used resources of the set of executed computing tasks being greater than the available resources, calculating a weight for each executed computing task in the set of executed computing tasks;
based on the weights, terminating a first number of the executed computing tasks to cause the updated used resources to meet the resource demand change.
2. The method of claim 1, wherein the obtaining of the available resources for the set of running computing tasks in the cluster comprises:
obtaining the used resources of the set of executed computing tasks;
acquiring allocable idle resources and reserved buffer resources of a plurality of physical machines in a cluster;
determining a difference between the sum of the allocable free resources and the used resources and the reserved buffer resources as the usable resources.
3. The method of claim 2, wherein the allocable idle resources include unallocated resources and unused resources of the resources occupied by the running state-stable computing task.
4. The method of claim 2, wherein the reserved buffer resources are determined based on a fluctuation amplitude of the allocable free resources.
5. The method of claim 3, wherein the running state-stable computational tasks comprise:
and in the preset time length for starting to operate, actually using the calculation tasks with the resource fluctuation less than the preset threshold value and/or the calculation tasks with the operation time length exceeding the time length threshold value.
6. The method of claim 5, wherein the duration threshold is determined based on an average of the computing task's actual usage resource fluctuation amplitude.
7. The method of claim 1, wherein the calculating a weight for each executed computing task in the set of executed computing tasks, further comprises:
and for each operated calculation task, calculating the weight of the operated calculation task based on one or more preset weights in task priority, starting time, resource utilization rate and task type.
8. A resource allocation apparatus, comprising:
the acquisition unit is configured to respond to the resource demand change of at least one native service in the cluster, and acquire usable resources of a running calculation task set in the cluster;
a computing unit configured to compute a weight for each executed computing task in the set of executed computing tasks in response to a used resource of the set of executed computing tasks being greater than the available resource;
a termination unit configured to terminate a first number of the executed computing tasks based on the weight so that the updated used resources meet the resource demand change.
9. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.
10. A computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011003579.9A CN112148448B (en) | 2020-09-22 | 2020-09-22 | Resource allocation method, apparatus, device and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011003579.9A CN112148448B (en) | 2020-09-22 | 2020-09-22 | Resource allocation method, apparatus, device and computer readable medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112148448A true CN112148448A (en) | 2020-12-29 |
| CN112148448B CN112148448B (en) | 2024-01-19 |
Family
ID=73896842
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011003579.9A Active CN112148448B (en) | 2020-09-22 | 2020-09-22 | Resource allocation method, apparatus, device and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112148448B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160373251A1 (en) * | 2015-06-18 | 2016-12-22 | Citrix Systems, Inc. | Media Support in Secure Mobile Applications |
| US9760388B1 (en) * | 2015-09-22 | 2017-09-12 | Amazon Technologies, Inc. | Memory monitoring tool |
| US20170286140A1 (en) * | 2016-03-31 | 2017-10-05 | Ca, Inc. | Java-based offload service in a mainframe environment |
| CN107291547A (en) * | 2016-03-31 | 2017-10-24 | 阿里巴巴集团控股有限公司 | A kind of task scheduling processing method, apparatus and system |
| CN109766174A (en) * | 2018-12-24 | 2019-05-17 | 杭州数梦工场科技有限公司 | Resource regulating method, resource scheduling device and computer readable storage medium |
| CN109857544A (en) * | 2018-12-27 | 2019-06-07 | 努比亚技术有限公司 | Resource reclaim control method, terminal and computer readable storage medium |
| CN110727473A (en) * | 2019-10-10 | 2020-01-24 | Oppo广东移动通信有限公司 | Resource control method, device, system, storage medium and electronic equipment |
| CN110784335A (en) * | 2019-09-19 | 2020-02-11 | 烽火通信科技股份有限公司 | Network element resource reservation system under cloud scene |
| CN111290843A (en) * | 2019-05-10 | 2020-06-16 | 展讯半导体(南京)有限公司 | Process management method and device |
| CN111538586A (en) * | 2020-01-23 | 2020-08-14 | 中国银联股份有限公司 | Cluster GPU resource management scheduling system, method and computer readable storage medium |
-
2020
- 2020-09-22 CN CN202011003579.9A patent/CN112148448B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160373251A1 (en) * | 2015-06-18 | 2016-12-22 | Citrix Systems, Inc. | Media Support in Secure Mobile Applications |
| US9760388B1 (en) * | 2015-09-22 | 2017-09-12 | Amazon Technologies, Inc. | Memory monitoring tool |
| US20170286140A1 (en) * | 2016-03-31 | 2017-10-05 | Ca, Inc. | Java-based offload service in a mainframe environment |
| CN107291547A (en) * | 2016-03-31 | 2017-10-24 | 阿里巴巴集团控股有限公司 | A kind of task scheduling processing method, apparatus and system |
| CN109766174A (en) * | 2018-12-24 | 2019-05-17 | 杭州数梦工场科技有限公司 | Resource regulating method, resource scheduling device and computer readable storage medium |
| CN109857544A (en) * | 2018-12-27 | 2019-06-07 | 努比亚技术有限公司 | Resource reclaim control method, terminal and computer readable storage medium |
| CN111290843A (en) * | 2019-05-10 | 2020-06-16 | 展讯半导体(南京)有限公司 | Process management method and device |
| CN110784335A (en) * | 2019-09-19 | 2020-02-11 | 烽火通信科技股份有限公司 | Network element resource reservation system under cloud scene |
| CN110727473A (en) * | 2019-10-10 | 2020-01-24 | Oppo广东移动通信有限公司 | Resource control method, device, system, storage medium and electronic equipment |
| CN111538586A (en) * | 2020-01-23 | 2020-08-14 | 中国银联股份有限公司 | Cluster GPU resource management scheduling system, method and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112148448B (en) | 2024-01-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110391938B (en) | Method and apparatus for deploying services | |
| CN110781373B (en) | List updating method and device, readable medium and electronic equipment | |
| CN113934958B (en) | Page loading method and device, electronic equipment and computer readable medium | |
| CN112379982B (en) | Task processing method, device, electronic equipment and computer readable storage medium | |
| CN114020470A (en) | Resource allocation method, device, readable medium and electronic equipment | |
| CN110768861B (en) | Method, device, medium and electronic equipment for obtaining overtime threshold | |
| CN111240834A (en) | Task execution method and device, electronic equipment and storage medium | |
| CN115237589A (en) | SR-IOV-based virtualization method, device and equipment | |
| CN111580974A (en) | GPU instance distribution method and device, electronic equipment and computer readable medium | |
| CN114625536A (en) | Video memory allocation method, device, medium and electronic equipment | |
| CN112182374B (en) | Inventory control method, apparatus, electronic device, and computer-readable medium | |
| CN111596992A (en) | Navigation bar display method and device and electronic equipment | |
| CN111694670B (en) | Resource allocation method, apparatus, device and computer readable medium | |
| CN113792869B (en) | Video processing method and device based on neural network chip and electronic equipment | |
| CN111694672B (en) | Resource allocation method, task submission method, device, electronic equipment and medium | |
| CN111756833B (en) | Node processing method, node processing device, electronic equipment and computer readable medium | |
| CN112148448B (en) | Resource allocation method, apparatus, device and computer readable medium | |
| CN111339643B (en) | Resource consumption evaluation method and device, electronic equipment and storage medium | |
| CN112163176A (en) | Data storage method and device, electronic equipment and computer readable medium | |
| CN112527454A (en) | Container group scheduling method and device, electronic equipment and computer readable medium | |
| CN111580890A (en) | Method, apparatus, electronic device, and computer-readable medium for processing features | |
| CN116755889B (en) | Data acceleration method, device and equipment applied to server cluster data interaction | |
| CN110633115A (en) | Task distribution method and device, electronic equipment and storage medium | |
| CN116319322B (en) | Power equipment node communication connection method, device, equipment and computer medium | |
| CN117649096B (en) | Paper box production task execution method and device, electronic equipment and computer medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant after: Douyin Vision Co.,Ltd. Address before: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant before: Tiktok vision (Beijing) Co.,Ltd. Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant after: Tiktok vision (Beijing) Co.,Ltd. Address before: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant before: BEIJING BYTEDANCE NETWORK TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |