CN113886069A

CN113886069A - Resource allocation method and device, electronic equipment and storage medium

Info

Publication number: CN113886069A
Application number: CN202111050492.1A
Authority: CN
Inventors: 李灏
Original assignee: Beijing QIYI Century Science and Technology Co Ltd
Current assignee: Beijing QIYI Century Science and Technology Co Ltd
Priority date: 2021-09-08
Filing date: 2021-09-08
Publication date: 2022-01-04

Abstract

The application relates to a resource allocation method, a resource allocation device, an electronic device and a storage medium, wherein the method comprises the following steps: under the condition that a task newly-added event is monitored to occur, acquiring a task configuration file of a target task for performing containerization deployment, wherein the task configuration file corresponds to the task newly-added event; pre-scheduling the application instance to a target node in at least one candidate node according to the resource occupation amount of the application instance contained in the target task indicated in the task configuration file, so that the resource of the resource occupation amount in the available resources of the target node is pre-occupied by the application instance; and under the condition that the application instances are all pre-scheduled successfully, scheduling the application instances to the target node. According to the method and the device, after the application examples in the target task can be pre-scheduled, the application examples are really scheduled, the use efficiency of resources can be effectively improved, and the throughput of a resource pool formed by all candidate nodes to the task can be improved.

Description

Resource allocation method and device, electronic equipment and storage medium

Technical Field

The present application relates to the field of resource scheduling technologies, and in particular, to a resource allocation method and apparatus, an electronic device, and a storage medium.

Background

In Kubernetes (a container group management system, abbreviated as K8s), the most basic unit of operation is Pod, which may contain multiple containers, but generally, a Pod contains only one container.

The distributed task in Kubernetes means that one task, Job (Job means a one-time task, a plurality of containers can be run in parallel through Job, when the task is executed, the container is automatically exited, and the cluster does not wake up the container again), comprises a plurality of Pods, the Pods respectively operate, and the operation results are merged according to a certain protocol. The distributed task can break through the limitation of single machine performance, and a plurality of machines serve the whole operation, so that the operation speed can be increased.

In the related art, when a user creates a Job through kubernets and resources in a resource pool only enable part of Pod under the Job to be scheduled and run, the Job cannot be started and resources already held by the Job in the resource pool cannot be used by other jobs; in addition, when there are multiple jobs to be started and resources in the resource pool can only enable all the Pod under one of the jobs to be scheduled, a situation may occur that each Job holds resources of a part of the Pod without releasing, so that each Job cannot be started; further, the above situation will result in waste of resources.

Aiming at the technical problem of resource waste during resource allocation in the related art, an effective solution is not provided at present.

Disclosure of Invention

In order to solve the technical problem of resource waste in resource allocation in the related art, the application provides a resource allocation method, a resource allocation device, an electronic device and a storage medium.

In a first aspect, an embodiment of the present application provides a resource allocation method, including:

under the condition that a task newly-added event is monitored to occur, acquiring a task configuration file of a target task which is subjected to containerization deployment and corresponds to the task newly-added event, wherein the task newly-added event is used for indicating resource allocation of the target task;

pre-scheduling the application instance to a target node in at least one candidate node according to the resource occupation amount of the application instance contained in the target task indicated in the task configuration file, so that the resource of the resource occupation amount in the available resources of the target node is pre-occupied by the application instance, wherein the available resource amount of the available resources is greater than or equal to the resource occupation amount;

and scheduling the application instances to the target node under the condition that the application instances are all pre-scheduled successfully, wherein the pre-scheduling success is used for indicating that the resource quantity of the residual resources of each candidate node is greater than or equal to 0 after the pre-scheduling is carried out on each application instance.

Optionally, as in the foregoing method, before the obtaining a task configuration file of a target task for performing containerization deployment corresponding to the task adding event, the method further includes:

acquiring the task configuration file created by the target object;

and storing the task configuration file into a target database, wherein the task newly-added event is the task configuration file stored into the target database.

Optionally, as in the foregoing method, the acquiring a task configuration file of a target task corresponding to the task adding event includes:

and acquiring the task configuration file corresponding to the task newly-added event from the target database.

Optionally, as in the foregoing method, the number of the application instances is multiple, and the number of the candidate nodes is multiple;

the pre-scheduling the application instance to a target node in at least one candidate node according to the resource occupation amount of the application instance contained in the target task indicated in the task configuration file, so that the resource occupation amount of the available resources of the target node is pre-occupied by the application instance comprises:

and under the condition that the sum of the available resource amount of each candidate node is greater than or equal to the sum of the resource occupation amount of each application instance, sequentially pre-scheduling the plurality of application instances to at least one target node in the plurality of candidate nodes so as to enable the resource occupation amount of the available resources of the target node to be pre-occupied by the application instances, wherein one application instance is scheduled to one target node.

Optionally, as in the foregoing method, the pre-scheduling, in accordance with the resource occupation amount of each application instance and the available resource amount of the target node, a plurality of application instances to at least one of the target nodes in sequence, includes:

determining a current target application instance to be processed according to the corresponding scheduling sequence of each application instance;

determining a first target node to which the target application instance is prescheduled, wherein the first target node is a node which is in the plurality of candidate nodes, has an available resource amount larger than the resource occupation amount of the target application instance and has the smallest available resource amount;

pre-scheduling the target application instance onto the first target node.

Optionally, as in the foregoing method, after the pre-scheduling of the target application instance onto the first target node, the method further includes:

and updating the available resource amount of the first target node by using the difference value of the available resource amount of the first target node and the resource occupation amount of the target application instance.

Optionally, as in the foregoing method, before the scheduling the application instance onto the target node, the method further includes:

and storing the corresponding relation between the application instance and the target node into a target storage system, so that the target node creates a target container according to the application instance after monitoring that the target node is bound with the application instance, and starts the target container to run the application instance.

In a second aspect, an embodiment of the present application provides a resource allocation apparatus, including:

the file acquisition module is used for acquiring a task configuration file of a target task for containerization deployment, corresponding to a task adding event, under the condition that the task adding event is monitored to occur, wherein the task adding event is used for indicating resource allocation of the target task;

a pre-scheduling module, configured to pre-schedule the application instance to a target node in at least one candidate node according to a resource occupancy amount of the application instance included in the target task indicated in the task configuration file, so that a resource of the resource occupancy amount in available resources of the target node is pre-occupied by the application instance, where the available resource amount of the available resource is greater than or equal to the resource occupancy amount;

and the scheduling determining module is used for scheduling the application instances to the target node under the condition that the application instances are all pre-scheduled successfully, wherein the pre-scheduling success is used for indicating that the resource quantity of the residual resources of each candidate node is more than or equal to 0 after the pre-scheduling is carried out on each application instance.

In a third aspect, an embodiment of the present application provides an electronic device, including: the system comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;

the memory is used for storing a computer program;

the processor, when executing the computer program, is configured to implement the method according to any of the preceding claims.

In a fourth aspect, the present application provides a computer-readable storage medium, which includes a stored program, where the program is executed to perform the method according to any one of the preceding claims.

Compared with the prior art, the technical scheme provided by the embodiment of the application has the following advantages:

according to the method provided by the embodiment of the application, the real scheduling is carried out on each application instance after the application instances in the target task are determined to be pre-scheduled, so that the situation that when the application instances are directly scheduled, certain application instances in the target task cannot be scheduled due to the fact that the current candidate node cannot provide enough resources for the target task, and therefore the target task cannot be executed can be avoided, and meanwhile, the situation that other tasks cannot be executed due to the fact that the scheduled application instances in the target task continuously occupy the resources and are not released easily occurs; furthermore, the method in the embodiment can effectively improve the use efficiency of the resources, and can improve the throughput of the resource pool formed by each candidate node to the task.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.

Fig. 1 is a flowchart of a resource allocation method according to an embodiment of the present application;

fig. 2 is a flowchart of a resource allocation method according to another embodiment of the present application;

fig. 3 is a flowchart of a resource allocation method according to another embodiment of the present application;

fig. 4 is a flowchart of a resource allocation method provided in the application example of the present application;

fig. 5 is a block diagram of a resource allocation apparatus according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

In the related art, kubernets provides a Job (a container for managing working classes, a set of one or more pots) for containerized deployment of computing tasks, and when a user creates a Job through kubernets, the pots under the Job may or may not be scheduled. Such as: one Job1 associates 10 Pod, of which 6 Pod can be scheduled and run, and the other 4 pods are in a waiting state, but when training of deep learning is run by using a distributed task, that is, distributed training (generally, deep learning requires operation on a large amount of training data, and a single machine cannot meet the requirement, so distributed training is the mainstream way of deep learning), 10 pods must be run to train, and the other 4 pods must be started to continue running. Clearly, this Job1 both holds resources for 6 Pod without releasing, while waiting for resources for 4 tasks to be released, before another 4 tasks run. This results in a waste of resources.

In another case, if there are 4 resources in the current resource pool, and there are 4 Pod under each of Job2 and Job3, and Job2 and Job3, in the related art, there are resources for which Job2 has obtained 1 Pod and Job3 has obtained 3 Pod, and at this time, neither of Job2 and Job3 can start normally, but the resources are occupied. If the remaining resources are not increased, the two tasks will never start normally.

Therefore, if a new scheduling mode is available for resource calling in the related art, the scheduling dilemma can be solved, and the throughput of the whole system can be greatly improved.

In order to solve the above technical problem, according to an aspect of an embodiment of the present application, a resource allocation method is provided. Alternatively, in this embodiment, the above-mentioned method for acquiring attribute information may be applied to a hardware environment formed by a terminal and a server. The server is connected with the terminal through a network, can be used for providing services for the terminal or a client installed on the terminal, and can be provided with a database on the server or independently of the server for providing data storage services for the server.

The network may include, but is not limited to, at least one of: wired networks, wireless networks. The wired network may include, but is not limited to, at least one of: wide area networks, metropolitan area networks, local area networks, which may include, but are not limited to, at least one of the following: WIFI (Wireless Fidelity), bluetooth. The terminal may not be limited to a PC, a mobile phone, a tablet computer, and the like.

The resource allocation method in the embodiment of the present application may be executed by a server, may also be executed by a terminal, and may also be executed by both the server and the terminal. The terminal executing the resource allocation method according to the embodiment of the present application may also be executed by a client installed thereon.

Taking the resource allocation method in this embodiment executed by the server as an example, fig. 1 is a resource allocation method provided in this embodiment, and includes the following steps S101 to S103:

step S101, under the condition that a task newly-added event is monitored to occur, acquiring a task configuration file of a target task which is subjected to containerization deployment and corresponds to the task newly-added event, wherein the task newly-added event is used for indicating resource allocation of the target task.

The resource allocation method is applied to the application that resources need to be allocated according to tasks, so that the tasks can obtain application scenes of the resources needed by operation; for example, kubernets may be used to allocate resources to Job, or may be used in other scenarios. Taking a scenario of resource allocation to a Job (i.e., a target task) performing containerization deployment in kubernets as an example, for a newly added Job bi, since how many resources are allocated to the Job bi needs to be determined, a task configuration file of the Job bi needs to be acquired.

The task configuration file may be a configuration file required for creating a target task JobI, and the task configuration file includes all the pods in the target task JobI and resources required by each Pod, and the types of the resources may include, but are not limited to: storage resources, memory resources, and CPU resources, among others. The task configuration file may be a file created by a user at a specific storage location, and after the creation of the task configuration file is completed, it may be regarded that a task addition event for triggering resource allocation on a target task corresponding to the task configuration file occurs.

For example, whether a newly generated or newly written task configuration file exists may be monitored according to a preset frequency (e.g., once every 5 s), when a newly written task configuration file is monitored in a certain storage location, it is determined that a task addition event occurs, and then the newly written task configuration file is read.

Step S102, according to the resource occupation amount of the application instance contained in the target task indicated in the task configuration file, the application instance is pre-scheduled to the target node in at least one candidate node, so that the resource occupied by the resource in the available resource of the target node is pre-occupied by the application instance, wherein the available resource amount of the available resource is greater than or equal to the resource occupation amount.

After the task configuration file is obtained, since the task configuration file corresponds to the target task and includes all the pods in the target task JobI and the resources required by the pods, each Pod is an application instance, all the application instances included in the target task and the resource occupation amount of each application instance can be determined based on the task configuration file.

The resource occupation amount may be a sum of resources occupied by each application instance Pod in the target task, for example, when the resources required by each application instance Pob included in the target task JobI are consistent and the resource required by each Pod is recorded as a unit resource (for example, 2GB-10 CPU operation units-20 GB storage resources), the resource occupation amount of the target task JobI is 4 unit resources. In general, a Job includes at least one Pod, and each Pod needs to be allocated to the same processing Node to operate.

The candidate nodes may be nodes for providing computational resources, the candidate nodes may include one or more nodes, a resource pool may be formed by a plurality of candidate nodes, and the computational resources that can be provided by each candidate node are limited, so that when the amount of available resources in the candidate nodes decreases with the increase of running tasks, the target node is a node to which a certain application instance in the target task is scheduled. In addition, under the condition that the corresponding resources can be allocated to each application instance in the target task is not determined, the application instances can be only pre-scheduled to the target node in at least one candidate node, so that the resources occupied by the resources in the available resources of the target node are pre-occupied by the application instances; that is, the resource occupied by the resource in the available resource on the target node is only temporarily pre-occupied by the application instance, and the resource occupied by the resource in the available resource on the target node is not used for running the application instance, so that the same resource can be prevented from being repeatedly allocated to a plurality of application instances; if the application examples contained in the last target task can be pre-scheduled to a certain target node, the application examples can occupy the corresponding pre-occupied resources; and if at least one application instance contained in the last target task cannot be prescheduled on any candidate node, all application instances of the target task are enabled to be free from pre-preoccupied with the corresponding resources, so that the pre-preoccupied resources are released and used for pre-allocating the application instances of other tasks.

For example, when only 1 application instance exists and 3 candidate nodes exist, the resource occupation amount L1 of the application instance is determined, and then the available resource amounts L2, L3 and L4 of the candidate nodes are determined; and finally, selecting candidate nodes with the available resource amount larger than L1 as target nodes according to the quantity relations between L1 and L2, L3 and L4 respectively.

And step S103, under the condition that the application instances are all pre-scheduled successfully, scheduling the application instances to the target node, wherein the pre-scheduling success is used for indicating that the resource quantity of the residual resources of each candidate node is more than or equal to 0 after each application instance is pre-scheduled.

The successful pre-scheduling may be that the remaining resource amount in each target node is still greater than or equal to 0 after the application instance in the target task is scheduled to the target node corresponding to the application instance; that is, the situation that the amount of resources required by the application instance scheduled by the target node is larger than the amount of available resources per se does not occur when the pre-scheduling is successful; further, in all the pre-schedules corresponding to the target task, each resource can only be occupied by one pre-scheduled application instance.

When each application instance is pre-scheduled successfully, it is stated that the target task can be executed by one target node or a combination of a plurality of target nodes, and thus, each application instance is scheduled to the target node corresponding to each application instance.

For example, when there are 2 application instances A, B and 3 candidate nodes a, B, c in the target task, when it is determined that application instance a can be pre-scheduled into candidate node a (i.e. the available resource amount of candidate node a is greater than or equal to the resource occupancy of application instance a) and application instance B can be pre-scheduled into candidate node B (i.e. the available resource amount of candidate node B is greater than or equal to the resource occupancy of application instance B) according to the method in the foregoing step S102, application instance a is scheduled into candidate node a and application instance B is scheduled into candidate node B, so as to complete the resource scheduling of the target task.

By the method in the embodiment, the application instances in the target task are determined to be pre-scheduled and then are really scheduled, so that the situation that when the application instances are directly scheduled, certain application instances in the target task cannot be scheduled due to the fact that the current candidate node cannot provide enough resources for the target task, and the target task cannot be executed can be avoided, and meanwhile, the situation that other tasks cannot be executed due to the fact that the scheduled application instances in the target task continuously occupy the resources and are not released easily occurs because the other tasks cannot acquire enough resources; furthermore, the method in the embodiment can effectively improve the use efficiency of the resources, and can improve the throughput of the resource pool formed by each candidate node to the task.

As shown in fig. 2, in some embodiments, as the foregoing method, before the step S101 acquires the task configuration file of the target task for performing containerization deployment corresponding to the task adding event, the method further includes the following steps S201 and S202:

in step S201, a task profile created by a target object is acquired.

The target object can be a configurator who needs to obtain operation service for the target task, and the task configuration file is a file used by the target object to create the target task.

Alternatively, a new CRD (Custom Resource Definition) Resource in kubernets needs to be created to manage the following individual Pod, because the native Job of kubernets cannot satisfy the gapscheduler. A new resource IQJob (i.e., a task profile) may be created, where the IQJob adds a podsetup field compared to the Job in the prior art, and compared to the Job profile in which a UUID (universal Unique Identifier) of a target task, the number of Pod, and the resource amount of each type of resource required by each Pod may be defined by the podsetup field.

Step S202, storing the task configuration file into a target database, wherein the task configuration file is stored into the target database by the task newly-added event.

The target database may be a database for storing the task configuration file, after the task configuration file is stored in the target database, the database may be monitored through a preset query interface according to a preset monitoring frequency, and after the task configuration file is confirmed to be newly added to the target database, it is determined that a task newly-added event occurs.

For example, after obtaining IQJob (i.e., task configuration file), kubernet API server records it into etcd (i.e., target database, etcd is a highly available Key/Value storage system, mainly used for sharing configuration and service discovery). Through the list _ and _ watch interface, a new IQJob is added in the etcd, and the entire content of the IQJob is obtained.

By the method in the embodiment, the newly added task configuration file can be obtained in real time, and the timeliness of task processing can be further improved.

In some embodiments, as in the foregoing method, the step S101 of obtaining a task configuration file of a target task corresponding to a task adding event includes:

and acquiring a task configuration file corresponding to the task newly-added event from the target database.

As can be seen from steps S201 and S202 of the foregoing embodiment, the task configuration file is stored in the target database; thus, the task profile may be retrieved from the target database.

For example, based on a task addition event, it may be determined that a task configuration file is newly written in the target database, and then the newly written file may be queried according to the writing time of each file in the database, and it may be determined that the file is the task configuration file corresponding to the task addition time.

By the method in the embodiment, the newly-added task configuration file in the target database can be quickly acquired, so that the target task can be quickly allocated with resources based on the task configuration file in the later period.

In some embodiments, as in the foregoing method, the number of application instances is multiple, and the number of candidate nodes is multiple;

the step S102 pre-schedules the application instance to a target node in the at least one candidate node according to the resource occupancy amount of the application instance included in the target task indicated in the task configuration file, so that the pre-occupancy of the resource occupancy amount in the available resources of the target node by the application instance includes:

and under the condition that the sum of the available resource amount of each candidate node is greater than or equal to the sum of the resource occupation amount of each application instance, sequentially pre-scheduling the multiple application instances to at least one target node in the multiple candidate nodes according to the resource occupation amount of each application instance, so that the resource occupying amount of the resource in the available resource of the target node is pre-occupied by the application instance, wherein one application instance is scheduled to one target node.

The sum of the available resource amounts can be obtained by accumulating the available resources of each candidate node; the sum of the resource occupation amounts can be obtained by accumulating the resource occupation amounts of the application instances of the target task, and the resource occupation amounts of the application instances can be determined based on the configuration content corresponding to the application instances in the task configuration file. Since the target task can be executed only after each application instance therein can be scheduled onto the target node, pre-scheduling is necessary only if the sum of the available resource amounts of the respective candidate nodes is greater than or equal to the sum of the resource occupation amounts of the respective application instances.

Under the condition that the sum of the available resource amounts of the candidate nodes is larger than or equal to the sum of the resource occupation amounts of the application instances, the target node pre-scheduled by each application instance can be determined based on the resource occupation amount of each application instance and the relation between the available resource amounts of the candidate nodes, so that the resource occupying amount of the available resources of the target node is pre-occupied by the application instances, the resource occupying amount of the available resources of the target node is only temporarily pre-occupied by the application instances, and the resource occupying amount of the available resources of the target node is not used for running the application instances. Further, an application instance need only be scheduled on one target node, and a target node may be scheduled with multiple application instances.

For example, after determining the resource occupation amount of each application instance Pod, each application instance Pod may be sequentially determined, a candidate Node to which each application instance Pod may be pre-scheduled in all candidate nodes is taken as a target Node, and a corresponding relationship between an application instance Pod and the target Node is established, optionally, the resource occupation amount of each application instance Pod may be sequentially compared with the available resource amount of each candidate Node, and when it is determined that the available resource amount of the candidate Node1 is greater than or equal to the resource occupation amount of the application instance Pod, the application instance Pod is scheduled on the candidate Node1 that is taken as the target Node.

By the method in the embodiment, the application instances can be scheduled according to the resource occupation amount of each application instance under the condition that the sum of the available resource amounts of each candidate node is greater than or equal to the sum of the resource occupation amounts of each application instance, so that the condition that pre-scheduling of all the application instances cannot be finished due to the fact that the sum of the available resource amounts is less than the sum of the resource occupation amounts can be avoided, and the efficiency of pre-scheduling can be effectively improved.

As shown in fig. 3, in some embodiments, as the foregoing method, the step of sequentially pre-scheduling a plurality of application instances onto at least one target node of a plurality of candidate nodes according to the resource occupation amount of each application instance and the available resource amount of the target node includes the following steps S301 to S303:

step S301, determining a current target application instance to be processed according to a scheduling order corresponding to each application instance.

The scheduling order may be a predetermined order that indicates the order in which the application instances are scheduled, each application instance having a corresponding scheduling order. Optionally, the scheduling order may be manually set in advance, or may be obtained by sequencing according to other strategies, for example: according to the resource occupation amount of each application instance, when the resource occupation amount includes a plurality of types of resources (such as a memory, a CPU, a GPU, a disk and the like), the application instances can be sequenced by determining the single resource occupation amount of the resource types for sequencing, and the scheduling order corresponding to the application instances is obtained. Typically, the scheduling orders are in sequence.

After the current application instance is prescheduled, the current application instance to be processed can be determined based on the scheduling sequence of the previous application instance, and is marked as the target application instance.

Step S302, a first target node pre-scheduled by a target application instance is determined, wherein the first target node is a node which has the available resource amount larger than the resource occupation amount of the target application instance and has the minimum available resource amount in a plurality of candidate nodes.

The candidate nodes can be sorted according to the sequence of the available resource amount from small to large, and are sequentially compared with the resource occupation amount of the target application instance according to the sorting, and when it is determined that a certain available resource amount meets the resource occupation amount of the target application instance, a first target node which has the available resource amount larger than the resource occupation amount of the target application instance and has the minimum available resource amount can be screened.

Step S303, prescheduling the target application instance to the first target node.

After the first target node satisfying the condition in step S302 is determined, the target application instance may be pre-scheduled to the first target node, so as to occupy the corresponding resource amount on the first target node by the resource occupation amount of the target application instance.

By the method in the embodiment, the node with the available resource amount larger than the resource occupation amount of the target application instance and the minimum available resource amount is selected as the first target node, so that the residual resource amount of the candidate node after the target application instance is scheduled is the minimum, and the waste of resources can be effectively reduced.

In some embodiments, as in the foregoing method, after the target application instance is pre-scheduled on the first target node, the method further includes the following step S401:

step S401, the available resource amount of the first target node is updated by using the difference value between the available resource amount of the first target node and the resource occupation amount of the target application instance.

The available resource amount of the first target node is the available resource amount before the target application instance is pre-scheduled to the first target node; and the difference is the amount of resources available to the first target node after the target application instance is pre-scheduled to the first target node.

After the difference is obtained, if an application instance needs to be allocated to the first target node in a later stage, it needs to be determined whether the first target node meets the requirement of the application instance to be allocated based on the difference, and therefore, the available resource amount of the first target node needs to be updated through the difference, otherwise, even if the pre-scheduling is successful, the situation that the pre-scheduled application instance cannot be actually scheduled to the first target node due to insufficient available resource amount can occur during actual scheduling.

In some embodiments, as in the foregoing method, before scheduling the application instance onto the target node, the method further includes the following step S501:

step S501, storing the corresponding relation between the application instance and the target node in a target storage system, so that the target node creates a target container according to the application instance after monitoring that the target node is bound with the application instance, and starts the target container to run the application instance.

The target storage system may be an etcd as described in the foregoing embodiments, and the correspondence between the application instance and the target node may be written in the etcd through the K8s scheduler.

The kubel running on the target node can monitor the application instance pod bound to the kubel running on the target node through a list and watch interface, then a target container is created according to the application instance, and the target container is started to run the application instance.

By the method in the embodiment, each target node can start the target container to run the application instance according to the corresponding relation.

Fig. 4 shows an application example of the method in any of the above embodiments:

1. and receiving the IQJob (namely, a task configuration file) created by the user, wherein the format of the parameters in the IQJob is consistent with the format of the created Job (namely, the target task).

After a request is received by a K8s API server (which provides HTTP Rest interfaces such as add, delete, check, and watch of various resource objects (Pod, RC, Service, etc.) of K8s, and is a data bus and a data center of the entire system), the request is recorded in the etcd (i.e., a target database).

The IQJob Controller can find that an IQJob is newly added in the etcd through the list _ and _ watch interface (the interface for monitoring the etcd), and then obtain the whole content of the IQJob from the etcd.

4. Firstly, the K8s API server is queried to obtain the resource use conditions of all nodes (i.e. candidate nodes) at present.

5. According to the content of the Pod (i.e. the application instance) in the IQJob, a PodSet is initialized (for determining whether the sum of the available resource amounts of the candidate nodes is greater than or equal to the sum of the resource occupation amounts of the application instances), and after determining whether the sum of the available resource amounts of the candidate nodes is greater than or equal to the sum of the resource occupation amounts of the application instances, whether each Pod can be dispatched to the Node is determined.

6. If so, then there may be multiple pods through the garg Scheduler (one Kube-batch job, and none of the pods will execute), and the K8s Scheduler will be requested to do the real scheduling.

The K8s Scheduler writes the scheduled information, i.e. the correspondence of Pod and Node, into the etcd (i.e. the target storage system consistent with the target database).

8. At this time, kubel running on the Node (main 'Node agent' running on each Node) monitors etcd through a list and watch interface, obtains the Pod bound to itself according to the corresponding relation, and then starts container to run the Pod bound to itself.

As shown in fig. 5, according to an embodiment of another aspect of the present application, there is also provided a resource allocation apparatus, including:

the file acquisition module 1 is configured to acquire a task configuration file of a target task performing containerization deployment, corresponding to a task adding event, under the condition that the occurrence of the task adding event is monitored, wherein the task adding event is used for indicating resource allocation of the target task;

the pre-scheduling module 2 is configured to pre-schedule the application instance to a target node in the at least one candidate node according to the resource occupancy amount of the application instance included in the target task indicated in the task configuration file, so that resources occupied by the resource occupancy amount in available resources of the target node are pre-occupied by the application instance, wherein the available resource occupancy amount of the available resources is greater than or equal to the resource occupancy amount;

and the scheduling determining module 3 is configured to schedule the application instances onto the target node under the condition that the application instances are all pre-scheduled successfully, where the pre-scheduling successfully is used to indicate that the resource amount of the remaining resources of each candidate node is greater than or equal to 0 after each application instance is pre-scheduled.

Specifically, the specific process of implementing the functions of each module in the apparatus according to the embodiment of the present invention may refer to the related description in the method embodiment, and is not described herein again.

In some embodiments, the apparatus as in the previous paragraph, further comprising:

an acquisition module for acquiring a task configuration file created by a target object;

and the file storage module is used for storing the task configuration file into the target database, wherein the task newly-added event is the task configuration file stored into the target database.

In some embodiments, the file obtaining module 1 is configured to obtain a task configuration file corresponding to a task adding event from a target database.

In some embodiments, as in the foregoing apparatus, the number of application instances is multiple, and the number of candidate nodes is multiple;

the pre-scheduling module 2 is configured to, when the sum of the available resource amounts of each candidate node is greater than or equal to the sum of the resource occupancy amounts of each application instance, sequentially pre-schedule the plurality of application instances onto at least one target node among the plurality of candidate nodes, so that resources occupied by resources in the available resources of the target node are pre-occupied by the application instances, where one application instance is scheduled onto one target node.

In some embodiments, as in the foregoing apparatus, the pre-scheduling module 2 includes:

the first determining unit is used for determining a target application instance to be processed currently according to the scheduling order corresponding to each application instance;

the second determining unit is used for determining a first target node pre-scheduled by a target application instance, wherein the first target node is a node which has the available resource amount larger than the resource occupation amount of the target application instance and has the minimum available resource amount in the plurality of candidate nodes;

and the pre-scheduling unit is used for pre-scheduling the target application instance to the first target node.

and the updating module is used for updating the available resource amount of the first target node by using the difference value between the available resource amount of the first target node and the resource occupation amount of the target application instance.

and the corresponding relation storage module is used for storing the corresponding relation between the application instance and the target node into the target storage system, so that the target node creates a target container according to the application instance after monitoring that the target node is bound with the application instance, and starts the target container to run the application instance.

According to another embodiment of the present application, there is also provided an electronic apparatus including: as shown in fig. 6, the electronic device may include: the system comprises a processor 1501, a communication interface 1502, a memory 1503 and a communication bus 1504, wherein the processor 1501, the communication interface 1502 and the memory 1503 complete communication with each other through the communication bus 1504.

A memory 1503 for storing a computer program;

the processor 1501 is configured to implement the steps of the above-described method embodiments when executing the program stored in the memory 1503.

The bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the electronic equipment and other equipment.

The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.

The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.

The embodiment of the present application further provides a computer-readable storage medium, where the storage medium includes a stored program, and when the program runs, the method steps of the above method embodiment are executed.

It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

The foregoing are merely exemplary embodiments of the present invention, which enable those skilled in the art to understand or practice the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for resource allocation, comprising:

2. The method according to claim 1, wherein before the obtaining of the task configuration file of the target task for containerization deployment corresponding to the task adding event, the method further comprises:

acquiring the task configuration file created by the target object;

3. The method of claim 2, wherein the obtaining a task profile for a target task corresponding to the task add event comprises:

4. The method of claim 1, wherein the number of application instances is plural and the number of candidate nodes is plural;

5. The method of claim 4, wherein said pre-scheduling a plurality of said application instances to at least one of said target nodes in turn comprises:

pre-scheduling the target application instance onto the first target node.

6. The method of claim 5, wherein after said pre-scheduling said target application instance onto said first target node, said method further comprises:

7. The method according to any of claims 1-6, wherein prior to said scheduling said application instance onto said target node, said method further comprises:

8. A resource allocation apparatus, comprising:

9. An electronic device, comprising: the system comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;

the memory is used for storing a computer program;

the processor, when executing the computer program, implementing the method of any of claims 1 to 7.

10. A computer-readable storage medium, characterized in that the storage medium comprises a stored program, wherein the program when executed performs the method of any of the preceding claims 1 to 7.