CN112579304A - Resource scheduling method, device, equipment and medium based on distributed platform - Google Patents
Resource scheduling method, device, equipment and medium based on distributed platform Download PDFInfo
- Publication number
- CN112579304A CN112579304A CN202011644671.3A CN202011644671A CN112579304A CN 112579304 A CN112579304 A CN 112579304A CN 202011644671 A CN202011644671 A CN 202011644671A CN 112579304 A CN112579304 A CN 112579304A
- Authority
- CN
- China
- Prior art keywords
- target
- task
- resource
- target task
- allocable
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 66
- 238000012545 processing Methods 0.000 claims description 48
- 230000008569 process Effects 0.000 claims description 14
- 238000012790 confirmation Methods 0.000 claims description 10
- 238000004422 calculation algorithm Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000010276 construction Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000004806 packaging method and process Methods 0.000 description 2
- 238000013468 resource allocation Methods 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 235000019800 disodium phosphate Nutrition 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
- G06F9/5088—Techniques for rebalancing the load in a distributed system involving task migration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
Abstract
The application provides a resource scheduling method, device, equipment and medium based on a distributed platform, and relates to the technical field of resource scheduling. The method comprises the steps of obtaining operation parameters of historical tasks in a preset time period, wherein the operation parameters comprise actual resource consumption parameters and task types; determining an initial resource application parameter of a target task according to an actual resource consumption parameter of a historical task, wherein the task type of the target task is the same as that of the historical task; according to the initial resource application parameters of the target task, the target resources are allocated for the target task and the target task is executed.
Description
Technical Field
The present application relates to the field of resource scheduling technologies, and in particular, to a resource scheduling method, apparatus, device, and medium based on a distributed platform.
Background
In the field of big data calculation, a stream processing task and a batch processing task are taken as two important task processing modes, wherein the stream processing task is mainly used for processing a data item transmitted in real time; the batch processing task is mainly to operate a large-capacity static data set and return a result after the calculation process is completed.
Conventionally, resource allocation for streaming and batch processing tasks is generally performed according to user application.
However, the existing resource allocation method is single, and the resource applied by the user is larger than the resource required by task operation, so that the problem of resource waste exists.
Disclosure of Invention
An object of the present application is to provide a resource scheduling method, device, apparatus and medium based on a distributed platform, which can improve resource utilization rate, in view of the above deficiencies in the prior art.
In order to achieve the above purpose, the technical solutions adopted in the embodiments of the present application are as follows:
in a first aspect, the present invention provides a resource scheduling method based on a distributed platform, including:
acquiring operation parameters of historical tasks within a preset time period, wherein the operation parameters comprise actual resource consumption parameters and task types;
determining an initial resource application parameter of a target task according to the actual resource consumption parameter of the historical task, wherein the task type of the target task is the same as the task type of the historical task;
and distributing target resources for the target task and executing the target task according to the initial resource application parameters of the target task.
In an optional embodiment, the determining an initial resource application parameter of a target task according to an actual resource consumption parameter of the historical task includes:
acquiring initial resource application parameters of the historical tasks;
and if the initial resource application parameters and the actual resource consumption parameters of the historical tasks meet the preset conditions, adjusting the initial resource application parameters of the historical tasks according to a preset algorithm to obtain the initial resource application parameters of the target tasks.
In an optional embodiment, the allocating target resources to the target task and executing the target task includes:
if the actual resource consumption parameter in the target task execution process is larger than the preset resource threshold corresponding to the target task, closing the target task, saving the running state of the target task, and determining whether allocable target resources exist;
and if determining that the allocable target resources exist, increasing and allocating the target resources for the target task, and restarting and executing the target task according to the running state of the target task.
In an optional embodiment, the determining whether there are allocable target resources includes:
monitoring and acquiring actual operation parameters of other current tasks, wherein the actual operation parameters comprise: actual resource consumption parameters and task types;
determining whether allocable target resources exist in the other tasks or not according to the actual operating parameters of the other tasks and preset resource thresholds corresponding to the other tasks;
if the allocable target resource is determined, increasing and allocating the target resource for the target task, and restarting and executing the target task according to the running state of the target task, wherein the steps of:
and if the allocable target resources are determined, dynamically allocating the allocable target resources to the target task, and restarting and executing the target task according to the running state of the target task.
In an optional embodiment, the determining whether there are allocable target resources includes:
determining whether allocable target resources exist according to the reserved allocable resources;
if the allocable target resource is determined, increasing and allocating the target resource for the target task, and restarting and executing the target task according to the running state of the target task, wherein the steps of:
and if the allocable target resources are determined, dynamically allocating the allocable target resources to the target task, and restarting and executing the target task according to the running state of the target task.
In an optional embodiment, after determining whether there are allocable target resources, the method further includes:
and if no allocable target resource exists, sending alarm information to the user terminal, wherein the alarm information comprises a target task identifier.
In an optional embodiment, if the target task includes a plurality of batch processing tasks, the method further includes:
and responding to the time-sharing processing confirmation instruction, and controlling the batch processing tasks to execute in a time-sharing mode according to a preset rule.
In a second aspect, the present invention provides a resource scheduling apparatus based on a distributed platform, including:
the acquisition module is used for acquiring the operation parameters of the historical tasks in a preset time period, wherein the operation parameters comprise actual resource consumption parameters and task types;
the determining module is used for determining an initial resource application parameter of a target task according to the actual resource consumption parameter of the historical task, wherein the task type of the target task is the same as the task type of the historical task;
and the allocation module is used for allocating target resources for the target task and executing the target task according to the initial resource application parameters of the target task.
In an optional embodiment, the determining module is specifically configured to obtain an initial resource application parameter of the historical task;
and if the initial resource application parameters and the actual resource consumption parameters of the historical tasks meet the preset conditions, adjusting the initial resource application parameters of the historical tasks according to a preset algorithm to obtain the initial resource application parameters of the target tasks.
In an optional embodiment, the allocation module is specifically configured to close the target task, save an operating state of the target task, and determine whether an allocable target resource remains, if an actual resource consumption parameter in the target task execution process is greater than a preset resource threshold corresponding to the target task;
and if determining that the allocable target resources exist, increasing and allocating the target resources for the target task, and restarting and executing the target task according to the running state of the target task.
In an optional embodiment, the allocation module is specifically configured to monitor and acquire actual operating parameters of other current tasks, where the actual operating parameters include: actual resource consumption parameters and task types;
determining whether allocable target resources exist in the other tasks or not according to the actual operating parameters of the other tasks and preset resource thresholds corresponding to the other tasks;
and if the allocable target resources are determined, dynamically allocating the allocable target resources to the target task, and restarting and executing the target task according to the running state of the target task.
In an optional embodiment, the allocating module is specifically configured to determine whether an allocable target resource still exists according to a reserved allocable resource; and if the allocable target resources are determined, dynamically allocating the allocable target resources to the target task, and restarting and executing the target task according to the running state of the target task.
In an optional embodiment, the allocating module is further configured to send warning information to the user terminal if it is determined that no allocable target resource exists, where the warning information includes a target task identifier.
In an optional implementation manner, if the target task includes a plurality of batch processing tasks, the resource scheduling apparatus further includes: and the control module is used for responding to the time-sharing processing confirmation instruction and controlling the batch processing tasks to be executed in a time-sharing mode according to a preset rule.
In a third aspect, the present invention provides an electronic device comprising: the resource scheduling method comprises a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, when an electronic device runs, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to execute the steps of the resource scheduling method based on the distributed platform according to any one of the preceding embodiments.
In a fourth aspect, the present invention provides a storage medium, on which a computer program is stored, where the computer program is executed by a processor to execute the steps of the resource scheduling method based on a distributed platform according to any one of the foregoing embodiments.
The beneficial effect of this application is:
in the resource scheduling method, device, equipment and medium based on the distributed platform, the operation parameters of the historical tasks within the preset time period are obtained, and the operation parameters comprise actual resource consumption parameters and task types; determining an initial resource application parameter of a target task according to an actual resource consumption parameter of a historical task, wherein the task type of the target task is the same as that of the historical task; according to the initial resource application parameters of the target task, the target resources are allocated for the target task and the target task is executed.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.
Fig. 1 is a schematic flowchart of a resource scheduling method based on a distributed platform according to an embodiment of the present application;
fig. 2 is a schematic flowchart of another resource scheduling method based on a distributed platform according to an embodiment of the present application;
fig. 3 is a schematic flowchart of another resource scheduling method based on a distributed platform according to an embodiment of the present application;
fig. 4 is a schematic flowchart of another resource scheduling method based on a distributed platform according to an embodiment of the present application;
fig. 5 is a schematic flowchart of another resource scheduling method based on a distributed platform according to an embodiment of the present application;
fig. 6 is a functional module schematic diagram of a resource scheduling apparatus based on a distributed platform according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
Fig. 1 is a schematic flow diagram of a resource scheduling method based on a distributed platform according to an embodiment of the present application, where the method may be applied to any node in a distributed cluster, and the node may be an electronic device such as a computer, a server, and a processor that can build the distributed platform, where the distributed platform refers to a software platform that builds data storage, data analysis, and computation and the like on a cluster composed of multiple hosts, and a basic principle of the distributed platform is to cooperatively compute by using multiple computers, and a processing problem of large-scale data can be well solved by using resource sharing and cooperative computing capabilities provided by the distributed platform, and the method has the characteristics of resource sharing, user transparency, high cost performance, high reliability, high flexibility, and the like. As shown in fig. 1, the resource scheduling method includes:
s101, obtaining operation parameters of historical tasks in a preset time period, wherein the operation parameters comprise actual resource consumption parameters and task types.
According to different application scenarios, the task types of the historical tasks may be different, and optionally, the task types may include but are not limited to a stream processing task, a batch processing task, and the like, where the stream processing task may be used to process a real-time data stream; the batch processing task can be used for processing historical data sets or data streams with low real-time performance.
For a historical task of any task type, the operation parameters of the historical task in a preset time period can be acquired, and the operation parameters can include an actual resource consumption parameter and a task type. Optionally, the preset time period may be one day, one week, one month, and the like, which is not limited herein and may be different according to the actual application scenario; the actual resource consumption parameter may be used to indicate a resource parameter actually consumed by executing the historical task, and optionally, the resource type may include, but is not limited to: CPU, memory, network, disk, etc.
S102, determining initial resource application parameters of the target task according to the actual resource consumption parameters of the historical tasks.
The target task may be the same as the historical task in task type, and the initial resource application parameter of the target task may be used to indicate the initial resource allocated to the target task.
It can be understood that after the actual resource consumption parameters of the historical tasks are obtained, the initial resource application parameters of the target tasks can be determined according to the actual resource consumption parameters of the historical tasks, so that when the target resources are allocated to the target tasks, allocation can be performed according to the actual resource consumption conditions of the historical tasks, and therefore excessive target resources can be prevented from being allocated to the target tasks, and the resource utilization rate can be improved.
In an optional embodiment, based on the actual resource consumption parameter of the historical task, a resource parameter that is greater than a certain value of the actual resource consumption parameter of the historical task may be obtained as an initial resource application parameter of the target task, and of course, the actual determination method is not limited thereto.
S103, distributing target resources for the target task and executing the target task according to the initial resource application parameters of the target task.
Based on the above embodiment, after the initial resource application parameter of the target task is obtained, the corresponding target resource may be allocated to the target task according to the initial resource application parameter, and the target task is executed.
Optionally, the historical task is a stream processing task or a batch processing task, and the distributed platform provided in the embodiment of the present application may schedule the stream and batch processing tasks based on the kubernets (K8s) technology and the Apache Flink open source stream processing framework, but is not limited thereto.
In summary, according to the resource scheduling method based on the distributed platform provided by the embodiment of the application, the operation parameters of the historical task in the preset time period are obtained, and the operation parameters include actual resource consumption parameters and task types; determining an initial resource application parameter of a target task according to an actual resource consumption parameter of a historical task, wherein the task type of the target task is the same as that of the historical task; according to the initial resource application parameters of the target task, the target resources are allocated for the target task and the target task is executed.
Fig. 2 is a schematic flowchart of another resource scheduling method based on a distributed platform according to an embodiment of the present application. Optionally, as shown in fig. 2, the determining an initial resource application parameter of the target task according to the actual resource consumption parameter of the historical task includes:
s201, obtaining initial resource application parameters of the historical tasks.
For the historical task, the historical task may include corresponding initial resource application parameters, where the initial resource application parameters may be pre-allocated to the historical task by a user according to historical experience, and according to an actual application scenario, resource types included in the initial resource application parameters may include, but are not limited to, a CPU, a memory, a network, a disk, and the like.
S202, if the initial resource application parameters and the actual resource consumption parameters of the historical tasks meet the preset conditions, the initial resource application parameters of the historical tasks are adjusted according to a preset algorithm, and the initial resource application parameters of the target tasks are obtained.
After the initial resource application parameter of the historical task is obtained, it may be determined whether the initial resource application parameter and the actual resource consumption parameter of the historical task satisfy a preset condition, optionally, the preset condition may be a preset ratio, a preset difference, or whether the initial resource application parameter of the historical task is greater than the actual resource consumption parameter, and the like, which is not limited herein.
If it is determined that the preset condition is met, the initial resource application parameters of the historical task may be adjusted according to a preset algorithm, and optionally, when the adjustment is performed, each initial resource parameter in the initial resource application parameters of the historical task may be adjusted according to a preset resource adjustment parameter or a preset adjustment coefficient, where the adjustment manner may include, but is not limited to: summing each initial resource parameter of the historical task and the corresponding preset resource adjusting parameter to obtain each initial resource parameter after the historical task is adjusted; or, multiplication operation may be performed according to each initial resource parameter of the historical task and the corresponding preset adjustment coefficient, so that each initial resource parameter after adjustment of the historical task may be obtained, or a combination of the two manners may be performed.
Of course, it should be noted that, when the initial resource application parameter of the target task is determined according to the actual resource consumption parameter of the historical task, the distribution rule of the actual resource consumption parameter of the historical task may also be statistically analyzed, and according to the statistical analysis result, the initial resource application parameter corresponding to the target task in different time periods may be determined, so as to improve the applicability. For example, statistical analysis is performed on historical tasks for processing vehicle driving information, and the generated data volume is large due to the fact that the number of online users is large in the daytime; and at night, the number of online users is small, the generated data volume is much smaller than that in the daytime, and therefore the resource consumption is much smaller than that in the daytime, corresponding initial resource application parameters can be allocated in different time periods for the target task of processing the vehicle driving information according to the statistical result, and certainly, the actual application scene is not limited to this.
Fig. 3 is a schematic flowchart of another resource scheduling method based on a distributed platform according to an embodiment of the present application. Optionally, as shown in fig. 3, allocating a target resource for a target task and executing the target task includes:
s301, if the actual resource consumption parameter in the running process of the target task is larger than the preset resource threshold corresponding to the target task, closing the target task, storing the running state of the target task, and determining whether allocable target resources exist.
For the target task, the preset resource threshold corresponding to the target task may be a maximum resource limit parameter preset for the target task, and it can be understood that, for the target task, if the target task is successfully executed, the actual resource consumption parameter of the target task should be smaller than the corresponding preset resource threshold.
Based on the above description, it can be understood that, although the target task is already allocated with the target resource according to the actual resource consumption parameter of the historical task, because the target task is a task in an actual application scenario, the target task may not be completely the same as the historical task, for example, there may be a moment when the data size processed by the target task is large, and the actual resource consumption parameter in the running process of the target task is greater than the corresponding preset resource threshold, it indicates that the target task has insufficient resources, the target task cannot be normally executed, at this moment, the target task may be closed, and the running state of the target task is saved, so that when the target task is restarted later, the running state of the target task may be quickly restored to the state before the closing based on the running state of the target task, and the execution efficiency is ensured.
After the target task is closed and the running state of the target task is saved, whether allocable target resources exist can be further determined, so that additional target resources can be added for the target task, and the target task can be guaranteed to run normally when started next time, wherein the allocable target resources can be understood as resources allocable without affecting other services.
S302, if the allocable target resources are determined, the allocation target resources are added to the target task, and the target task is restarted and executed according to the running state of the target task.
If the allocable target resource is determined, the target resource can be additionally allocated to the target task, so that the target task can be restarted and executed according to the running state of the target task based on the added target resource, and the target task can be ensured to be restarted successfully.
It should be noted that, for the target resource, after the target resource is additionally allocated to the target task, the resource that can be currently allocated to the target resource is the sum of the preset resource threshold and the added target resource.
Fig. 4 is a schematic flowchart of another resource scheduling method based on a distributed platform according to an embodiment of the present application. Optionally, as shown in fig. 4, the determining whether there are allocable target resources may include:
s401, monitoring and acquiring actual operation parameters of other current tasks, wherein the actual operation parameters comprise: actual resource consumption parameters and task types.
S402, determining whether allocable target resources exist in other tasks or not according to actual operation parameters of the current other tasks and preset resource thresholds corresponding to the other tasks.
When determining whether allocable target resources still exist, the actual operation parameters of other current tasks can be monitored and obtained, and whether other tasks still include allocable target resources is determined according to the actual operation parameters of other current tasks and the preset resource thresholds corresponding to other tasks.
If it is determined that the allocable target resource exists, increasing the allocation target resource for the target task, and restarting to execute the target task according to the running state of the target task, which may include:
and S403, if the allocable target resources are determined, dynamically allocating the allocable target resources to the target task, and restarting and executing the target task according to the running state of the target task.
Based on the above description, if it is determined that there are allocable target resources, the allocable target resources may be dynamically allocated to the target task, and the target task may be continuously executed according to the operating state of the target task.
Fig. 5 is a schematic flowchart of another resource scheduling method based on a distributed platform according to an embodiment of the present application. Optionally, as shown in fig. 5, the determining whether there are allocable target resources includes:
s501, determining whether allocable target resources exist according to the reserved allocable resources.
In some embodiments, a node may be provided with a reserved allocable resource, where the reserved allocable resource may be used to determine whether an allocable target resource still exists according to the reserved allocable resource if an actual resource consumption parameter in a target task execution process is greater than a preset resource threshold when a resource demand amount of some tasks increases.
It should be noted that, if the actual resource consumption parameter in the target task execution process is no longer greater than the preset resource threshold after the target task is executed for a certain time, the reserved allocable resource may be further recovered, so as to ensure that the actual resource consumption parameter in the new task execution process is greater than the corresponding preset resource threshold when a new task comes next time, and the reserved allocable resource may be additionally allocated to the new task, so as to ensure that the task is normally executed.
If the allocable target resource is determined, increasing the allocation target resource for the target task, and restarting and executing the target task according to the running state of the target task, wherein the steps of:
and S502, if the allocable target resources are determined, dynamically allocating the allocable target resources to the target task, and restarting and executing the target task according to the running state of the target task.
For the content of this part, reference is made to the related content mentioned above, and the description of this application is not repeated here.
Optionally, after determining whether there are allocable target resources, the method further includes:
and if no allocable target resource exists, sending alarm information to the user terminal, wherein the alarm information comprises a target task identifier.
It can be understood that, if it is determined that no allocable target resource exists, which indicates that no allocable target resource can be allocated to the target task for use, then an alarm message may be sent to the user terminal, where the alarm message includes a target task identifier, so that the user may know, through the user terminal, that the task failed to be executed, and thus facilitating expansion of the resource.
Optionally, if the target task includes a plurality of batch processing tasks, the method further includes:
and responding to the time-sharing processing confirmation instruction, and controlling the batch processing tasks to execute in a time-sharing mode according to a preset rule.
In some embodiments, when the target task includes multiple batch processing tasks, and the target task is submitted by a user, optionally, a time-sharing processing configuration interface may be presented to the user, and the time-sharing processing configuration interface may include a confirmation control, and the confirmation control may be used to indicate whether the user determines that the target task is executed in a time-sharing manner, optionally, the user may act on the confirmation control by clicking, long pressing, or the like to generate a time-sharing processing confirmation instruction, and in response to the time-sharing processing confirmation instruction, the multiple batch processing tasks may be controlled to be executed in a time-sharing manner according to a preset rule, so that consumption of resources may be maintained in a relatively smooth state, and processing pressure of the distributed cluster is reduced. Alternatively, the preset rule may be determined according to the arrival time of the batch processing task, or may be determined according to a preset priority of the batch processing task (for example, different batch processing tasks may be configured with corresponding priorities in advance), which is not limited herein.
Based on the above embodiments, as for the input data source and the output data source of the target task, optionally, the input data source and the output data source may be obtained based on databases such as tidb, mysql, postgres, hbase, kafka, and the like, which is not limited herein. It is to be understood that the type and address of the data source may be specified when the data source is created, and the output address of the data source may be specified when the data source is output.
For the creation of the target task, when the target task is created, a type of the target task (for example, a streaming task or a batch task) may be specified, and it is understood that the created task may be a task that is pre-written by a user on another device, or may be a task that is written based on the distributed platform, and if the target task is a task that is written based on the distributed platform, relevant checking (for example, syntax checking, function call checking, and the like) needs to be performed on the task, which is not limited herein. Based on the created target task, a target image file corresponding to the target task also needs to be constructed, optionally, according to a generation mode, included contents and the like of the target task, a corresponding mode can be selected to package and generate the target image file, and the target image file is pushed to an image warehouse, and the packaging mode can include but is not limited to construction by using a docker technology, construction based on a Flink technology, or construction by using an existing image packaging tool of a distributed platform.
Optionally, after the related parameters are set, optionally, the target image file can be pulled from the image warehouse based on the operator function of kubernets, then a flinkMaster is started to receive the task flow, finally the flinkMaster starts a taskManager pod to run the task flow, and the taskManager reads data according to the specified input data source and then outputs the data source after corresponding processing.
In some embodiments, the related execution data of the target task may also be saved to a log to find out which one is needed, so that the user may view the execution condition of the program and the execution effect of the task when the user needs the program (for example, the target task fails), so as to perform troubleshooting on the problem.
Fig. 6 is a functional module schematic diagram of a resource scheduling apparatus based on a distributed platform according to an embodiment of the present application, the basic principle and the generated technical effect of the apparatus are the same as those of the foregoing corresponding method embodiment, and for brief description, reference may be made to corresponding contents in the method embodiment for a part not mentioned in this embodiment. As shown in fig. 6, the resource scheduling apparatus 100 includes:
an obtaining module 110, configured to obtain operation parameters of a historical task within a preset time period, where the operation parameters include an actual resource consumption parameter and a task type;
a determining module 120, configured to determine an initial resource application parameter of a target task according to an actual resource consumption parameter of the historical task, where a task type of the target task is the same as a task type of the historical task;
the allocation module 130 is configured to allocate a target resource for the target task and execute the target task according to the initial resource application parameter of the target task.
In an optional embodiment, the determining module 120 is specifically configured to obtain an initial resource application parameter of the historical task; and if the initial resource application parameters and the actual resource consumption parameters of the historical tasks meet the preset conditions, adjusting the initial resource application parameters of the historical tasks according to a preset algorithm to obtain the initial resource application parameters of the target tasks.
In an optional embodiment, the allocating module 130 is specifically configured to close the target task, save an operating state of the target task, and determine whether an allocable target resource still exists, if an actual resource consumption parameter in the target task execution process is greater than a preset resource threshold corresponding to the target task; and if determining that the allocable target resources exist, increasing and allocating the target resources for the target task, and restarting and executing the target task according to the running state of the target task.
In an optional embodiment, the allocation module 130 is specifically configured to monitor and acquire actual operating parameters of other current tasks, where the actual operating parameters include: actual resource consumption parameters and task types; determining whether allocable target resources exist in the other tasks or not according to the actual operating parameters of the other tasks and preset resource thresholds corresponding to the other tasks; and if the allocable target resources are determined, dynamically allocating the allocable target resources to the target task, and restarting and executing the target task according to the running state of the target task.
In an optional embodiment, the allocating module 130 is specifically configured to determine whether an allocable target resource still exists according to a reserved allocable resource; and if the allocable target resources are determined, dynamically allocating the allocable target resources to the target task, and restarting and executing the target task according to the running state of the target task.
In an optional embodiment, the allocating module 130 is further configured to send warning information to the user terminal if it is determined that no allocable target resource exists, where the warning information includes a target task identifier.
In an optional implementation manner, if the target task includes a plurality of batch processing tasks, the resource scheduling apparatus further includes: and the control module is used for responding to the time-sharing processing confirmation instruction and controlling the batch processing tasks to be executed in a time-sharing mode according to a preset rule.
The above-mentioned apparatus is used for executing the method provided by the foregoing embodiment, and the implementation principle and technical effect are similar, which are not described herein again.
These above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more Application Specific Integrated Circuits (ASICs), or one or more microprocessors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), among others. For another example, when one of the above modules is implemented in the form of a Processing element scheduler code, the Processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).
Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 7, the electronic device may include: a processor 210, a storage medium 220, and a bus 230, wherein the storage medium 220 stores machine-readable instructions executable by the processor 210, and when the electronic device is operated, the processor 210 communicates with the storage medium 220 via the bus 230, and the processor 210 executes the machine-readable instructions to perform the steps of the above-mentioned method embodiments. The specific implementation and technical effects are similar, and are not described herein again.
Optionally, the present application further provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the computer program performs the steps of the above method embodiments. The specific implementation and technical effects are similar, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a unit is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.
The integrated unit implemented in the form of a software functional unit may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute some steps of the methods according to the embodiments of the present application. And the aforementioned storage medium includes: a U disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A resource scheduling method based on a distributed platform is characterized by comprising the following steps:
acquiring operation parameters of historical tasks within a preset time period, wherein the operation parameters comprise actual resource consumption parameters and task types;
determining an initial resource application parameter of a target task according to the actual resource consumption parameter of the historical task, wherein the task type of the target task is the same as the task type of the historical task;
and distributing target resources for the target task and executing the target task according to the initial resource application parameters of the target task.
2. The method according to claim 1, wherein the determining an initial resource application parameter of a target task according to an actual resource consumption parameter of the historical task comprises:
acquiring initial resource application parameters of the historical tasks;
and if the initial resource application parameters and the actual resource consumption parameters of the historical tasks meet the preset conditions, adjusting the initial resource application parameters of the historical tasks according to a preset algorithm to obtain the initial resource application parameters of the target tasks.
3. The method of claim 1, wherein the allocating target resources for the target task and executing the target task comprises:
if the actual resource consumption parameter in the target task execution process is larger than the preset resource threshold corresponding to the target task, closing the target task, saving the running state of the target task, and determining whether allocable target resources exist;
and if determining that the allocable target resources exist, increasing and allocating the target resources for the target task, and restarting and executing the target task according to the running state of the target task.
4. The method of claim 3, wherein the determining whether there are more allocable target resources comprises:
monitoring and acquiring actual operation parameters of other current tasks, wherein the actual operation parameters comprise: actual resource consumption parameters and task types;
determining whether allocable target resources exist in the other tasks or not according to the actual operating parameters of the other tasks and preset resource thresholds corresponding to the other tasks;
if the allocable target resource is determined, increasing and allocating the target resource for the target task, and restarting and executing the target task according to the running state of the target task, wherein the steps of:
and if the allocable target resources are determined, dynamically allocating the allocable target resources to the target task, and restarting and executing the target task according to the running state of the target task.
5. The method of claim 3, wherein the determining whether there are more allocable target resources comprises:
determining whether allocable target resources exist according to the reserved allocable resources;
if the allocable target resource is determined, increasing and allocating the target resource for the target task, and restarting and executing the target task according to the running state of the target task, wherein the steps of:
and if the allocable target resources are determined, dynamically allocating the allocable target resources to the target task, and restarting and executing the target task according to the running state of the target task.
6. The method of claim 4, wherein after determining whether there are more allocable target resources, further comprising:
and if no allocable target resource exists, sending alarm information to the user terminal, wherein the alarm information comprises a target task identifier.
7. The method of claim 6, wherein if the target task comprises a plurality of batch tasks, the method further comprises:
and responding to the time-sharing processing confirmation instruction, and controlling the batch processing tasks to execute in a time-sharing mode according to a preset rule.
8. A resource scheduling apparatus based on a distributed platform, comprising:
the acquisition module is used for acquiring the operation parameters of the historical tasks in a preset time period, wherein the operation parameters comprise actual resource consumption parameters and task types;
the determining module is used for determining an initial resource application parameter of a target task according to the actual resource consumption parameter of the historical task, wherein the task type of the target task is the same as the task type of the historical task;
and the allocation module is used for allocating target resources for the target task and executing the target task according to the initial resource application parameters of the target task.
9. An electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, when an electronic device runs, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to execute the steps of the distributed platform-based resource scheduling method according to any one of claims 1 to 7.
10. A storage medium having stored thereon a computer program for performing the steps of the distributed platform based resource scheduling method according to any one of claims 1 to 7 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011644671.3A CN112579304A (en) | 2020-12-30 | 2020-12-30 | Resource scheduling method, device, equipment and medium based on distributed platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011644671.3A CN112579304A (en) | 2020-12-30 | 2020-12-30 | Resource scheduling method, device, equipment and medium based on distributed platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112579304A true CN112579304A (en) | 2021-03-30 |
Family
ID=75145666
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011644671.3A Pending CN112579304A (en) | 2020-12-30 | 2020-12-30 | Resource scheduling method, device, equipment and medium based on distributed platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112579304A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113312166A (en) * | 2021-07-29 | 2021-08-27 | 阿里云计算有限公司 | Resource processing method and device |
| CN113342509A (en) * | 2021-08-03 | 2021-09-03 | 北京达佳互联信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113742059A (en) * | 2021-07-15 | 2021-12-03 | 上海朋熙半导体有限公司 | Task allocation method and device, computer equipment and storage medium |
| CN113778666A (en) * | 2021-08-13 | 2021-12-10 | 济南浪潮数据技术有限公司 | Dynamic allocation method, device and medium for resources required by monitoring equipment |
| CN113825240A (en) * | 2021-09-15 | 2021-12-21 | 阿里巴巴达摩院(杭州)科技有限公司 | Resource allocation method and base station equipment |
| CN114780170A (en) * | 2022-04-11 | 2022-07-22 | 远景智能国际私人投资有限公司 | Container resource configuration method, device, equipment and storage medium |
| CN115221211A (en) * | 2022-09-21 | 2022-10-21 | 国网智能电网研究院有限公司 | Graph calculation processing method and device, electronic equipment and storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107179945A (en) * | 2017-03-31 | 2017-09-19 | 北京奇艺世纪科技有限公司 | A kind of resource allocation methods and device |
-
2020
- 2020-12-30 CN CN202011644671.3A patent/CN112579304A/en active Pending
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107179945A (en) * | 2017-03-31 | 2017-09-19 | 北京奇艺世纪科技有限公司 | A kind of resource allocation methods and device |
Non-Patent Citations (1)
| Title |
|---|
| 陈潇潇: "《云计算与数据的应用》", 延吉：延边大学出版社, pages: 216 - 219 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113742059A (en) * | 2021-07-15 | 2021-12-03 | 上海朋熙半导体有限公司 | Task allocation method and device, computer equipment and storage medium |
| CN113742059B (en) * | 2021-07-15 | 2024-03-29 | 上海朋熙半导体有限公司 | Task allocation method, device, computer equipment and storage medium |
| CN113312166A (en) * | 2021-07-29 | 2021-08-27 | 阿里云计算有限公司 | Resource processing method and device |
| CN113312166B (en) * | 2021-07-29 | 2022-02-01 | 阿里云计算有限公司 | Resource processing method and device |
| CN113342509A (en) * | 2021-08-03 | 2021-09-03 | 北京达佳互联信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113342509B (en) * | 2021-08-03 | 2021-12-07 | 北京达佳互联信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113778666A (en) * | 2021-08-13 | 2021-12-10 | 济南浪潮数据技术有限公司 | Dynamic allocation method, device and medium for resources required by monitoring equipment |
| CN113825240A (en) * | 2021-09-15 | 2021-12-21 | 阿里巴巴达摩院(杭州)科技有限公司 | Resource allocation method and base station equipment |
| CN113825240B (en) * | 2021-09-15 | 2024-04-05 | 杭州阿里云飞天信息技术有限公司 | Resource allocation method and base station equipment |
| CN114780170A (en) * | 2022-04-11 | 2022-07-22 | 远景智能国际私人投资有限公司 | Container resource configuration method, device, equipment and storage medium |
| CN114780170B (en) * | 2022-04-11 | 2023-07-21 | 远景智能国际私人投资有限公司 | Container resource configuration method, device, equipment and storage medium |
| CN115221211A (en) * | 2022-09-21 | 2022-10-21 | 国网智能电网研究院有限公司 | Graph calculation processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112579304A (en) | Resource scheduling method, device, equipment and medium based on distributed platform | |
| CN108776934B (en) | Distributed data calculation method and device, computer equipment and readable storage medium | |
| US20160378570A1 (en) | Techniques for Offloading Computational Tasks between Nodes | |
| CN111338791A (en) | Method, device and equipment for scheduling cluster queue resources and storage medium | |
| Adhikary et al. | Quality of service aware cloud resource provisioning for social multimedia services and applications | |
| CN110990138A (en) | Resource scheduling method, device, server and storage medium | |
| US20110283292A1 (en) | Allocation of processing tasks | |
| CN114661462A (en) | Resource allocation method, system, computer readable storage medium and electronic device | |
| CN115373835A (en) | Task resource adjusting method and device for Flink cluster and electronic equipment | |
| CN113312154A (en) | Method, system, equipment and storage medium for scheduling satellite service resources | |
| CN113672345A (en) | IO prediction-based cloud virtualization engine distributed resource scheduling method | |
| CN110471769B (en) | Resource management method and device for virtual machine | |
| CN110096339B (en) | System load-based capacity expansion and contraction configuration recommendation system and method | |
| Hung et al. | Task scheduling for optimizing recovery time in cloud computing | |
| CN112565391A (en) | Method, apparatus, device and medium for adjusting instances in an industrial internet platform | |
| US10990385B1 (en) | Streaming configuration management | |
| CN112860387A (en) | Distributed task scheduling method and device, computer equipment and storage medium | |
| CN115952054A (en) | Simulation task resource management method, device, equipment and medium | |
| Tsenos et al. | Amesos: A scalable and elastic framework for latency sensitive streaming pipelines | |
| CN113626145B (en) | Dynamic capacity expansion method and system for number of business virtual machines | |
| CN115858499A (en) | Database partition processing method and device, computer equipment and storage medium | |
| CN115421920A (en) | Task management method and device for financial product, electronic equipment and storage medium | |
| CN114090201A (en) | Resource scheduling method, device, equipment and storage medium | |
| CN113391927A (en) | Method, device and system for processing business event and storage medium | |
| CN113656239A (en) | Monitoring method and device for middleware and computer program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |