WO2017113074A1

WO2017113074A1 - Resource allocation method, device, and system

Info

Publication number: WO2017113074A1
Application number: PCT/CN2015/099258
Authority: WO
Inventors: 梁殿鹏; 赵彦荣; 刘佳; 党李飞; 彭磊
Original assignee: 华为技术有限公司
Priority date: 2015-12-28
Filing date: 2015-12-28
Publication date: 2017-07-06
Also published as: CN108293041A; CN108293041B

Abstract

Embodiments of the present invention provide a resource allocation method, device, and system, used to allocate a resource to an application. The method comprises: an application controller receives a resource allocation request from a client end, the resource allocation request carrying resource demand information of an application running on the client end, wherein the application controller is configured to manage a resource pool, the resource pool comprises at least one launched container, and the container in the resource pool is negotiated from a resource manager by the application controller and launched in advance; and the application controller selects, according to the resource demand information of the application, an idle resource container from the resource pool, and allocates the idle resource container to the client end. According to the method, an application controller can promptly allocate a resource of a resource container upon reception of a resource allocation request of an application, so that a waiting time for launching a resource container is eliminated. In addition, by reusing a resource container in a resource pool, a resource waste resulting from repeatedly creating and terminating a resource container is avoided.

Description

Method, device and system for allocating resources

Technical field

The embodiments of the present invention relate to the field of computers, and in particular, to a resource allocation method.

Background technique

Since the development of cloud computing, the scale of clusters has continued to expand, and the types of clusters have also increased. Such as the common distributed computing Hadoop cluster for parallel programming technology tasks. From the trend of the use of distributed systems in the industry and the long-term development of the Hadoop framework, the Job Tracker/Task Tracker mechanism in the first version of MapReduce MRv1 requires large-scale adjustments to fix It is flawed in scalability, memory consumption, threading model, reliability and performance.

Yet Another Resource Negotiator (yarn) is a new Hadoop resource manager. It is a universal resource management system that provides unified resource management and scheduling for upper-layer applications. The benefits of utilization, unified resource management, and data sharing have brought tremendous benefits. Yarn was originally designed to fix the apparent lack of MRv1 and to improve scalability, reliability, and cluster utilization. Yarn implements these requirements by dividing Job Tracker's two main functions (resource management and job scheduling/monitoring) into two separate service programs, one for the Global Resource Manager (RM) and for each Job Master (AM) for each job.

Each job in Yarn enables a separate AM, which solves the single point of failure and expansion bottleneck in MRv1. But this approach will introduce a new problem: the job delay is large. Each job first needs to start an AM with the RM application resource. After the AM requests the resource from the RM and starts the resource container, the job can be officially started. So the Yarn job will have a long run delay, which is not conducive to running small jobs, and because you need to apply for an AM for each job, you need more computing resources.

Summary of the invention

In view of this, an embodiment of the present invention provides a resource allocation method, device, and system, Reduce the waiting time for jobs.

In a first aspect, the present application provides a method for allocating resource containers in a distributed system. The distributed system includes a resource manager and a node manager. The resource manager manages node resources of the distributed system, and the node manager starts based on the node resources. The resource container is used to execute the task of the application. The method includes: the resource manager starts the application controller when the trigger timing is met, and configures an initial specification of the resource pool managed by the application controller, where the initial specification of the resource pool is used. After instructing the application controller to apply for the number and specification of the resource container for the first time to the resource manager, the resource manager receives the first resource request of the resource pool sent by the application controller according to the initial specification of the resource pool, according to the resource pool a resource request, allocate an initial resource container for the resource pool, and send a first resource allocation message of the resource pool to the application controller, where the first resource allocation message includes a resource pool allocation managed by the resource manager for the application controller. The information of the node where the initial resource container is located.

The resource manager actively starts the application controller and configures the initial specification of the resource pool managed by the application controller, so that the application controller applies a certain number of resource containers to the resource manager in advance according to the initial specification of the resource pool, thereby starting the application in advance. The container reduces the time that subsequent application jobs wait for the resource container to start.

With reference to the first aspect, in a first possible implementation manner of the first aspect, the resource manager starting the application controller when the triggering timing is met includes: the resource manager receiving the request for pre-launching the application controller or pre-configuring the resource pool The request is initiated, or when the system is initialized, the application controller is started.

The startup time of the application controller can be started in a variety of ways. It can be started by the administrator according to the user's requirements, or can be started by the resource manager through system configuration.

With reference to the first aspect or any one of the foregoing possible implementation manners, in the second possible implementation manner of the first aspect, the initial specification of the resource pool managed by the resource manager configuration application controller includes: the resource manager according to The expected resource requirement information of the preset application, the initial specification of the resource pool managed by the application controller is configured; or the resource manager configures the resource pool managed by the application controller according to the collected usage information of the node resource of the distributed system. The initial specifications. For example, when there are sufficient unused node resources, the resource manager can configure the initial specification of the larger resource pool; when there are fewer unused node resources, the resource manager can configure the initial of the smaller resource pool. specification.

In addition, if the application controller is started by an administrator's instruction, the initial specification of the resource pool can be carried by the administrator in the startup command of the application controller, and the resource manager starts the response. When using the controller, configure the application controller.

In a second aspect, the present application provides a computer readable medium, comprising computer executed instructions, when the processor of the computer executes the computer to execute an instruction, the computer executes the first aspect or any of the possible implementations of the first aspect Methods.

In a third aspect, the present application provides a computing device, including: a processor, a memory, a bus, and a communication interface; the memory is configured to store an execution instruction, the processor is connected to the memory through the bus, when the computing device is running The processor executes the execution instructions stored by the memory to cause the computing device to perform the method of any of the first aspect or the first aspect.

In a fourth aspect, the present application provides a method for allocating resource containers in a distributed system, where the distributed system includes a resource manager and a node manager, where the resource manager is used to manage node resources of the distributed system, and the node manager uses Starting a resource container based on the node resource, where the resource container is used to execute an application task, and the resource manager starts an application controller when the trigger timing is met, and configures an initial specification of the resource pool managed by the application controller, and according to the resource The initial specification of the pool allocates an initial resource container for the resource pool managed by the application controller, and the initial resource container of the resource pool is started;

The method includes: receiving, by the application controller, a resource allocation request from a client, where the resource allocation request is used to request a resource container for an application running on the client, where the resource allocation request carries resource requirement information of the application, and according to the application Resource requirement information, from which an idle resource container is selected and allocated to the client.

With reference to the fourth aspect, in a first possible implementation manner of the fourth aspect, before the application controller receives the resource allocation request from the client, the method further includes: the application controller according to an initial specification of the resource pool The resource manager sends a first resource request, where the first resource request carries a quantity of resources determined according to an initial specification of the resource pool, and acquires a first resource allocation message of the resource pool sent by the resource manager, where the first resource is The allocation message includes information about the node where the resource manager allocates the initial resource container for the resource pool, and sends a startup request to the node manager to request the node manager to start the initial resource container of the resource pool.

Since the resource container in the resource pool is applied to the resource manager in advance by the application controller, when the application controller receives the resource allocation request from the client, the resource container can be allocated to the client in time, thereby avoiding In the prior art, the waiting time of the AM and the resource container is started after receiving the application job.

With reference to the fourth aspect, or any one of the foregoing possible implementation manners of the fourth aspect, in the second possible implementation manner of the fourth aspect, the resource allocation request includes an identifier of the client, where the application controller is in the resource pool After the idle resource container is allocated to the client, the method further includes: the application controller sending an indication message to each resource container in the resource container allocated to the client, where the indication message carries the identifier of the client, the indication A message is used to indicate that each resource container is assigned to the client.

By carrying the identifier of the client in the indication message, indicating that the corresponding resource container is allocated to the client, so that the corresponding resource container sends a registration message to the client, so that the client can assign the task of the application to the client. Registered resource container.

With reference to the fourth aspect, or any one of the foregoing possible implementation manners, in a third possible implementation manner of the fourth aspect, the method further includes: the application controller determining the remaining idle resource containers in the resource pool The number, if the number of remaining free resource containers is less than a preset first threshold, sending a second resource request to the resource manager, and receiving a second resource allocation message of the resource pool of the resource manager, according to the The second resource allocation message of the resource pool sends a startup request to the node manager, requesting the node manager to start a new resource container allocated for the resource pool.

More specifically, after the application controller allocates the resource container in the resource pool to the client, the number of idle resource containers remaining in the resource pool may be determined, or the application controller periodically determines the resource pool. The number of remaining idle resource containers. When the number of remaining free resource containers is less than a preset first threshold, indicating that the resource container of the resource pool may not satisfy the subsequent allocation operation, the resource is re-applied to the resource manager. A container to supplement the resource container in the resource pool.

With reference to the fourth aspect, or any one of the foregoing possible implementation manners, the fourth possible implementation manner of the fourth aspect, the method further includes: determining, by the application controller, the remaining idle resource containers in the resource pool And a quantity release message, if the number of remaining idle resource containers is greater than a preset second threshold, sending a resource release message to the at least one resource container in the idle resource container, where the resource release message is used to release the at least one resource container occupation resource of.

More specifically, after the allocated resource container performs the task of the client, the number of idle resource containers remaining in the resource pool may be determined, or the application controller periodically determines remaining free resources in the resource pool. The number of containers, when the number of remaining free resource containers is greater than a preset second threshold, sending a resource release message to at least one resource container in the idle resource container. Thereby releasing the resources occupied by the resource pool and ensuring the rational use of the global resources.

With reference to the fourth aspect, or any one of the foregoing possible implementation manners, in the fifth possible implementation manner of the fourth aspect, the application controller maintains state information of each resource container in the resource pool, the state information Indicates whether the corresponding resource container is idle; the application controller selects an idle resource container from the resource pool and allocates the resource to the client. The application controller selects the resource pool according to status information of each resource container in the resource pool. An idle resource container, and the selected resource container in the selected resource pool is allocated to the client.

Wherein, if the resource container is currently allocated to perform other tasks, the state of the resource container is not idle, and if the resource container is not currently allocated, the state of the resource container is idle. Specifically, the application controller may maintain a state information table, where the state information table maintains state information of each resource container in the resource pool, so that the application controller can quickly determine the current idle resource container, and according to the idle resources. Container and resource allocation requests allocate containers for clients.

With reference to the fourth aspect, or any one of the foregoing possible implementation manners of the fourth aspect, in the sixth possible implementation manner of the fourth aspect, after the application controller selects an idle resource container from the resource pool and allocates the The method further includes the application controller setting a state of each of the resource containers allocated to the client to be not idle.

With reference to the fourth aspect, or any one of the foregoing possible implementation manners of the fourth aspect, in the seventh possible implementation manner of the fourth aspect, the application controller allocates a status of each resource container in the resource container of the client After being set to not idle, the method further includes: the application controller receiving a status update message from each of the resource containers allocated to the client, the status update message being used to indicate that the task assigned by the client is completed; The application controller sets the state of each resource container in the resource container assigned to the client to idle.

The application controller dynamically updates the state information maintained by itself according to the change of the state of the resource container, thereby ensuring that the idle resource container in the resource pool can be accurately determined when there is a resource allocation request.

With reference to the fourth aspect, or any one of the foregoing possible implementation manners, in the seventh possible implementation manner of the fourth aspect, the resource allocation request further includes user rights information; the method further includes: the application controller The user rights information is verified according to a preset user permission library, and the user rights library contains the user rights information.

The user permission library contains user rights information of different users. If the user permission information of the client is not included in the user permission library, the resource allocation request of the client is rejected or not responded; If the user rights library contains the user rights information of the client 102, the subsequent resource allocation step is performed.

In a fifth aspect, the present application provides a computer readable medium, comprising computer executed instructions, in a possible implementation manner of the fourth aspect or the fourth aspect, when the processor of the computer executes the computer to execute the instruction Methods.

In a sixth aspect, the present application provides a computing device, including: a processor, a memory, a bus, and a communication interface; the memory is configured to store an execution instruction, the processor is connected to the memory through the bus, when the computing device is running The processor executes the execution instructions stored by the memory to cause the computing device to perform the method of any of the possible implementations of the fourth aspect or the fourth aspect.

In a seventh aspect, the present application provides a resource container allocation apparatus in a distributed system, where the distributed system includes the apparatus and a node manager, where the apparatus is configured to manage node resources of the distributed system, and the node manager is configured to be based on The node resource starts a resource container, where the resource container is used to perform an application task, and the device includes: a startup unit, configured to start an application controller when the trigger timing is met, and configure an initial specification of the resource pool managed by the application controller; a receiving unit, configured to receive a first resource request of the resource pool that is sent by the application controller according to an initial specification of the resource pool; and an allocating unit, configured to allocate an initial resource container to the resource pool according to the first resource request; And a sending unit, configured to send, to the application controller, a first resource allocation message of the resource pool, where the first resource allocation message includes information about a node where the initial resource container allocated by the allocation unit is allocated to the resource pool.

The device actively starts the application controller and configures an initial specification of the resource pool managed by the application controller, so that the application controller applies for a certain number of resource containers in advance according to the initial specification of the resource pool, thereby starting the application container in advance. , reducing the time that subsequent application jobs wait for the resource container to start.

With reference to the seventh aspect, in a first possible implementation manner of the seventh aspect, the starting unit is configured to start the application controller when the trigger timing is met, including: the startup unit is configured to receive a request for pre-launching the application controller or The resource pool request is pre-configured, or the application controller is started when the system is initialized.

The startup time of the application controller can be started in a variety of ways. It can be started by the administrator according to the user's requirements, or can be started by the startup unit when the system is initialized.

With reference to the seventh aspect, or any one of the foregoing possible implementation manners of the seventh aspect, in the second possible implementation manner of the seventh aspect, the starting unit is configured to configure a resource pool managed by the application controller The initial specification includes: the startup unit is configured to configure an initial specification of a resource pool managed by the application controller according to the expected resource requirement information of the preset application; or configure according to the collected usage information of the node resource of the distributed system. The initial specification of the resource pool managed by the application controller. For example, when there are sufficient unused node resources, the startup unit can configure an initial specification of a larger resource pool; when there are fewer unused node resources, the startup unit can configure an initial specification of a smaller resource pool.

In addition, if the application controller is started by an administrator's instruction, the initial specification of the resource pool may be carried by the administrator in the startup command of the application controller, and the startup controller activates the application controller to the application controller. Configure it.

In an eighth aspect, the application provides a resource container allocation apparatus in a distributed system, where the distributed system includes a resource manager and a node manager, and the resource manager is configured to manage node resources of the distributed system, and the node manager uses Starting a resource container based on the node resource, where the resource container is used to execute an application task, the resource manager starts the device when the trigger timing is met, configures an initial specification of the resource pool managed by the device, and according to the initial of the resource pool The specification allocates an initial resource container for the resource pool managed by the device, and the initial resource container is started;

The device includes: a receiving unit, configured to receive a resource allocation request from a client, where the resource allocation request is used to request a resource container for an application running on the client, where the resource allocation request carries resource requirement information of the application; And selecting, according to the resource requirement information of the application, an idle resource container from the resource pool to be allocated to the client.

With reference to the eighth aspect, in a first possible implementation manner of the eighth aspect, the apparatus further includes: a sending unit, the receiving unit is configured to use, according to the initial of the resource pool, a resource allocation request from the client The first resource request is sent to the resource manager, where the first resource request carries the quantity of resources determined according to the initial specification of the resource pool; the receiving unit is further configured to receive the first resource allocation message sent by the resource manager, The first resource allocation message includes information about a node where the initial resource container allocated by the resource manager is allocated to the resource pool; the sending unit is further configured to send a startup request to the node manager, requesting the node manager to start the The initial resource container for the resource pool.

Since the resource container in the resource pool is applied to the resource manager in advance by the device, when the receiving unit receives the resource allocation request from the client, the allocating unit can allocate the activated resource container to the client in time, thereby avoiding In the prior art, the waiting time of the AM and the resource container is started after receiving the application job.

In combination with the eighth aspect or the eighth aspect, any one of the possible implementation manners, In a possible implementation, the device further includes a sending unit, where the resource allocation request includes an identifier of the client, and the sending unit selects an idle resource container from the resource pool and allocates the resource to the client, where the sending unit is used for An indication message is sent to each resource container in the resource container allocated to the client, where the indication message carries an identifier of the client, and the indication message is used to indicate that each resource container is allocated to the client.

Sending an indication message to the allocated resource container by the sending unit, and carrying the identifier of the client in the indication message, indicating that the corresponding resource container is allocated to the client, so that the corresponding resource container sends a registration message to the client, Enables the client to assign the application's tasks to the resource container to which it is registered.

With reference to the eighth aspect, or any one of the foregoing possible implementation manners of the eighth aspect, in a third possible implementation manner of the eighth aspect, the apparatus further includes a determining unit, and a sending unit, the determining unit is configured to determine the resource pool The number of the remaining free resource containers, if the number of remaining free resource containers is less than a preset first threshold, the sending unit is configured to send a second resource request to the resource manager, and the receiving unit is further configured to receive the resource. a second resource allocation message of the resource pool sent by the manager, the sending unit is further configured to send a startup request to the node manager according to the second resource allocation message of the resource pool, requesting the node manager to start a new allocation for the resource pool Increased resource container.

More specifically, after the allocating unit allocates the resource container in the resource pool to the client, the determining unit determines the number of free resource containers remaining in the resource pool, or periodically determines the resource pool by the determining unit. The number of remaining idle resource containers, when the number of remaining free resource containers is less than a preset first threshold, indicating that the resource container of the resource pool may not satisfy the subsequent allocation operation, and then re-send to the resource manager through the sending unit. Apply for a resource container to supplement the resource container in the resource pool.

With reference to the eighth aspect, or any one of the foregoing possible implementation manners of the eighth aspect, in a fourth possible implementation manner of the eighth aspect, the device further includes a determining unit, and a sending unit, the determining unit is configured to determine the resource pool The number of remaining idle resource containers, if the number of remaining free resource containers is greater than a preset second threshold, the sending unit is configured to send a resource release message to at least one resource container in the idle resource container, where The resource release message is used to release resources occupied by the at least one resource container.

More specifically, after the allocated resource container performs the task of the client, the determining unit determines the number of free resource containers remaining in the resource pool, or periodically determines, by the determining unit, the remaining idle in the resource pool. The number of resource containers, when the remaining free resource containers If the number is greater than the preset second threshold, the resource release message is sent by the sending unit to the at least one resource container in the idle resource container. Thereby releasing the resources occupied by the resource pool and ensuring the rational use of the global resources.

With reference to the eighth aspect, or any one of the foregoing possible implementation manners of the eighth aspect, in the fifth possible implementation manner of the eighth aspect, the allocating unit is further configured to maintain state information of each resource container in the resource pool, where The status information indicates whether the corresponding resource container is idle; the allocation unit is configured to select an idle resource container from the resource pool, and the allocation unit is configured to: according to the status information of each resource container in the resource pool, Select an idle resource container in the resource pool and assign the selected resource container in the resource pool to the client.

Wherein, if the resource container is currently allocated to perform other tasks, the state of the resource container is not idle, and if the resource container is not currently allocated, the state of the resource container is idle. Specifically, the allocating unit may maintain a state information table, where the state information table maintains state information of each resource container in the resource pool, so that the allocating unit can quickly determine the current idle resource container, and according to the idle resource container and A resource allocation request allocates a container for the client.

With reference to the eighth aspect, or any one of the foregoing possible implementation manners of the eighth aspect, in the sixth possible implementation manner of the eighth aspect, after the allocation unit selects an idle resource container from the resource pool and allocates the virtual resource container to the client, It is also used to set the state of each resource container in the resource container assigned to the client to not idle.

With reference to the eighth aspect, or any one of the foregoing possible implementation manners of the eighth aspect, in the seventh possible implementation manner of the eighth aspect, the allocating unit sets a state of each resource container in the resource container allocated to the client After not being idle, the receiving unit is further configured to receive a status update message from each of the resource containers allocated to the client, the status update message is used to indicate that the task assigned by the client is completed; Used to set the state of each resource container in the resource container assigned to the client to idle based on the status update information.

The allocation unit dynamically updates the state information maintained by itself according to the change of the state of the resource container, thereby ensuring that the idle resource container in the resource pool can be accurately determined when there is a resource allocation request.

With reference to the eighth aspect, or any one of the foregoing possible implementation manners of the eighth aspect, in the seventh possible implementation manner of the eighth aspect, the resource allocation request further includes user rights information; the receiving unit is further configured to be configured according to the preset The user permission library verifies the user rights information, and the user permission library contains the user rights information.

The user permission library contains user rights information of different users. If the user permissions are not in the library If the user rights information of the client is included, the resource allocation request of the client is rejected or not; if the user rights information of the client 102 is included in the user rights library, the subsequent resource allocation step is performed.

In a ninth aspect, the present application provides a resource container allocation system in a distributed system, where the distributed system includes a resource manager and a node manager, and the resource manager is configured to start an application controller when a trigger timing is met. Configuring an initial specification of the resource pool managed by the application controller, and receiving a first resource request of the resource pool sent by the application controller according to the initial specification of the resource pool, and initializing the resource pool according to the first resource request The resource container sends a first resource allocation message of the resource pool to the application controller; the application controller is configured to acquire a first resource allocation message of the resource pool sent by the resource manager, according to the resource pool Information indicating a node where the initial resource container allocated for the resource pool is indicated in a resource allocation message, requesting the node manager to start an initial resource container of the resource pool; the node manager is configured to use according to the application The controller's request to start the initial resource container.

With reference to the ninth aspect, in a first possible implementation manner of the ninth aspect, the resource manager is configured to start the application controller when the trigger timing is met, and the resource manager is configured to receive the request for starting the application controller in advance The application controller is started when the request for the resource pool is pre-configured.

With reference to the ninth aspect, or any one of the possible implementation manners of the ninth aspect, in the second possible implementation manner of the ninth aspect, the initial specification of the resource pool used by the resource manager to configure the application controller includes: The resource manager is configured to configure an initial specification of the resource pool managed by the application controller according to the preset resource requirement information of the application set in advance or according to the collected usage information of the node resource of the distributed system.

With reference to the ninth aspect, or any one of the foregoing possible implementation manners of the ninth aspect, the application controller is further configured to receive a resource allocation request from a client, where the resource allocation request is And configured to request a resource container for an application running on the client, and select an idle resource container from the resource pool to allocate to the client according to the resource requirement information of the application in the resource allocation request.

With reference to the ninth aspect, or any one of the possible implementation manners of the ninth aspect, in the fourth possible implementation manner of the ninth aspect, the resource allocation request includes an identifier of the client, where the application controller is in the resource pool After the idle resource container is allocated to the client, the method further includes: sending an indication message to each resource container in the resource container allocated to the client, where the indication message is carried The identifier of the client, the indication message is used to indicate that each resource container is allocated to the client.

With reference to the ninth aspect, or any one of the foregoing possible implementation manners of the ninth aspect, in the fifth possible implementation manner of the ninth aspect, the application controller is further configured to: determine the remaining idle resource containers in the resource pool a number, if the number of remaining free resource containers is less than a preset first threshold, sending a second resource request to the resource manager, and receiving a second resource allocation message of the resource pool of the resource manager, according to the resource The second resource allocation message of the pool sends a start request to the node manager, requesting the node manager to start a new resource container allocated for the resource pool.

With reference to the ninth aspect, or any one of the foregoing possible implementation manners of the ninth aspect, in the sixth possible implementation manner of the ninth aspect, the application controller is further configured to: determine the remaining idle resource containers in the resource pool a number, if the number of remaining free resource containers is greater than a preset second threshold, sending a resource release message to at least one resource container in the idle resource container, the resource release message being used to release the at least one resource container Resources.

With reference to the ninth aspect or the ninth aspect, the application controller is further configured to maintain state information of each resource container in the resource pool, in a seventh possible implementation manner of the ninth aspect, The status information is used to indicate whether the corresponding resource container is idle. The application controller selects an idle resource container from the resource pool to be allocated to the client, and is further configured to: according to status information of each resource container in the resource pool. To determine the free resource container in the resource pool.

With reference to the ninth aspect, or any one of the foregoing possible implementation manners of the ninth aspect, in the eighth possible implementation manner of the ninth aspect, after the application controller selects an idle resource container from the resource pool and allocates the Also used to: set the state of each resource container in the resource container assigned to the client to not idle.

With reference to the ninth aspect, or any one of the possible implementation manners of the ninth aspect, in the ninth possible implementation manner of the ninth aspect, the application controller allocates a status of each resource container in the resource container of the client After being set to not idle, the method is further configured to: receive a status update message from each of the resource containers allocated to the client, the status update message is used to indicate that the task assigned by the client is completed, and is assigned to the The status of each resource container in the client's resource container is set to idle.

With reference to the ninth aspect, or any one of the possible implementation manners of the ninth aspect, in the tenth possible implementation manner of the ninth aspect, the resource allocation request further includes user rights information; the system further includes: the application controller The user rights information is verified according to a preset user permission library, and the user rights library contains the user rights information.

The ninth aspect is the system implementation corresponding to the first aspect and the fourth aspect, the first aspect or any one of the possible implementation manners of the first aspect, and the feature description in any one of the possible implementation manners of the fourth aspect or the fourth aspect Applicable to any possible implementation of the ninth aspect or the ninth aspect, and details are not described herein again.

According to the technical solution disclosed by the present invention, the resource container is applied to the resource manager in advance by the application controller, and the requested resource container is started in advance. After receiving the resource allocation request of the application, the resource container resource can be allocated in time to avoid The waiting time of the resource container is started, and the resource container in the resource pool is reused, thereby avoiding resource consumption of multiple opening and closing of the resource container, and directly managing the resource container in the resource pool through the application controller, thereby realizing more Flexible management.

DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the present invention, Those skilled in the art can also obtain other drawings based on these drawings without paying any creative work.

1 is a block diagram of an exemplary networked environment of a resource allocation system in accordance with an embodiment of the present invention;

2 is a schematic structural diagram of a hardware of a computing device according to an embodiment of the invention;

FIG. 3 is a signaling diagram of a resource allocation method according to an embodiment of the present invention; FIG.

4 is a signaling diagram of a resource allocation method according to an embodiment of the present invention;

FIG. 5 is a schematic diagram showing the logical structure of a resource allocation apparatus according to an embodiment of the present invention; FIG.

FIG. 6 is a schematic diagram showing the logical structure of a resource allocation apparatus according to an embodiment of the present invention; FIG.

FIG. 7 is a schematic diagram showing the logical structure of a resource allocation apparatus according to an embodiment of the present invention.

detailed description

Embodiments of the embodiments of the present invention will be described below with reference to the accompanying drawings.

Definition of Terms:

Resource Manager (RM) is a global resource manager responsible for resource management and allocation of the entire yarn system.

A resource container is a resource abstraction in yarn that encapsulates multiple types of resources on a node. Such as memory, CPU, disk, network, etc. The resource container is used to perform the tasks of the application.

The client is a device running an application to be applied for, and the type of the application may be MapReduce, Giraph, Storm, Spark, Tez/Impala, or Message Passing Interface (MPI).

The Node Manager (NM) is a resource and task manager on each node. On the one hand, it periodically reports the resource usage on the node and the running status of each resource container to the RM; on the other hand, It receives and processes various requests such as start/stop of resource containers from the application controller.

The application controller is used to apply for and manage the resource pool to the RM. The resource pool runs the started resource container, and allocates the resource container to the client according to the resource allocation request of the client.

1 shows an exemplary networked environment block diagram of a resource allocation system 100, as shown in FIG. 1, a system 100 including a client 102, an application controller 104, a resource manager 112, and a plurality of nodes 106, each of which The node 106 includes a node manager 108 and at least one resource container 110, and the resource pool 114 contains at least one resource container 110.

The client 102 runs an application to be applied for, and the type of the application may be MapReduce, Giraph, Storm, Spark, Tez/Impala, or MPI. The client 102 can be any type of computing device, which is not limited by the embodiment of the present invention.

The resource container 110 is a resource abstraction in a node, which can encapsulate multiple types of resources on a node, such as memory, CPU, disk, network, and the like. Optionally, the resource container 110 may also encapsulate only a part of the resources on a certain node, for example, only the memory and the CPU are encapsulated, which is not limited by the embodiment of the present invention. Among them, the resource container 110 can run any type of task. For example, a MapReduce application can request a resource container 110 to initiate a map or reduce task, and a Giraph application can request a resource container 110 to run a Giraph task. Users can also implement a custom application type that runs a specific task through the resource container 110 to implement a completely new application framework.

The application controller 104 is configured to manage the resource container 110 in the resource pool 114. The resource container in the resource pool 114 is a resource container that has been started, and is applied and started by the application controller 104 to the resource manager 112 in advance. At the same time, the task of the job can be performed as soon as possible, thereby saving the startup time of the resource container 114. When the application controller 104 requests resources from the resource manager 112, the resources returned by the resource manager 112 for the application controller 104 are represented by the resource container 110.

It should be understood that different applications may correspond to different application controllers because different applications may have different requirements for resource containers, that is, resource types of resource containers required by different applications, and the number of resources of each type are different. For each type of application, there may be multiple application controllers, which are not limited by the embodiments of the present invention. For the convenience of description, the embodiment of the present invention describes only the resource application process of one application.

It should be understood that the resource pool 114 shown in FIG. 1 includes multiple resource containers 110 among the plurality of nodes 106, but does not limit the plurality of nodes 106 or the resource containers 110 of the plurality of nodes 106 all belong to the resource pool. 114. The resource pool 114 is composed of a resource container 110.

The application controller 104 is configured to receive a resource allocation request from an application on the client 102, and allocate a resource container required for the application of the application to the application of the application. When the job is executed, the client 102 is in the application's job. Each task is assigned a resource container, and the task can only use the resources described in the resource container.

The resource manager 112 is a global resource manager responsible for resource management and allocation of the entire system. When receiving a resource request from the application controller 104, the resource controller can be allocated to the application controller 104 according to the load of the entire system. .

The client 102, the application controller 104, the resource manager 112, and the node manager 108 of each node 106 can communicate through a network, where the network can be the Internet, an intranet, or a local area network (LAN). , Wireless Local Area Networks (WLANs), Storage Area Networks (SANs), etc., or a combination of the above.

It should be understood that the purpose of FIG. 1 is merely exemplary participants of the system 100 and their interrelationships. Therefore, the depicted system 100 is greatly simplified, and the embodiments of the present invention are merely described in general terms, and the implementation thereof is not limited in any way. The client 102, the application controller 104, and the node 106 in FIG. 1 may be of any architecture, which is not limited by the embodiment of the present invention.

The application controller 104 and/or resource manager 112 shown in FIG. 1 can be implemented by the computing device 200 shown in FIG. 2.

2 is a simplified logical block diagram of computing device 200. As shown in FIG. 2, computing device 200 includes a processor 202, a memory unit 204, an input/output interface 206, a communication interface 208, a bus 210, and a storage device 212. Wherein, the processor 202, the memory unit 204, and the input/output connection The port 206, the communication interface 208, and the storage device 212 implement a communication connection with each other via the bus 210.

The processor 202 is a control center of the computing device 200 for executing related programs to implement the technical solutions provided by the embodiments of the present invention. Optionally, the processor 202 includes one or more central processing units (CPUs), such as the central processing unit 1 and the central processing unit 2 shown in FIG. Optionally, the computing device 200 can also include multiple processors 202, each of which can be a single core processor (including one CPU) or a multi-core processor (including multiple CPUs). Unless otherwise stated, in the present invention, a component for performing a specific function, for example, the processor 202 or the memory unit 204, may be implemented by configuring a general-purpose component to perform a corresponding function, or may be specifically performed by a specific one. The specific components of the function are implemented, and this application does not limit this. The processor 202 can be a general-purpose central processing unit, a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits for executing related programs to implement the technology provided by the present application. Program.

Processor 202 can be coupled to one or more storage schemes via bus 210. The storage scheme can include a memory unit 204 and a storage device 212. The storage device 212 can be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory unit 204 can be a random access memory. The memory unit 204 can be integrated with or integrated with the processor 202, or it can be one or more memory units independent of the processor 202.

Program code for execution by the processor 202 or a CPU internal to the processor 202 may be stored in the storage device 212 or the memory unit 204. Alternatively, program code (eg, an operating system, an application, a resource allocation module, or a communication module, etc.) stored internal to storage device 212 is copied to memory unit 204 for execution by processor 202.

The storage device 212 can be a physical hard disk or a partition thereof (including a small computer system interface storage or a global network block device volume), a network storage protocol (including a network file system NFS or the like network or a cluster file system), a file-based virtual storage device (virtual Disk mirroring), logical volume-based storage devices. It may include high speed random access memory (RAM), and may also include non-volatile memory, such as one or more disk memories, flash memories, or other non-volatile memory. In some embodiments, the storage device may further include the one or more processors 202 separate remote storage, such as a network disk accessed through a communication interface 208 with a communication network, which may be the Internet, an intranet, a local area network (LANs), a wide area network (WLANs), a storage area network (SANs), etc., or A combination of the above networks.

Operating systems (such as Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or embedded operating systems such as Vxworks) include controls and management of general system tasks (such as memory management, storage device control, power management, etc.) And various software components and/or drivers that facilitate communication between various hardware and software components.

The input/output interface 206 is for receiving input data and information, and outputting data such as operation results.

Communication interface 208 enables communication between computing device 200 and other devices or communication networks using transceivers such as, but not limited to, transceivers.

Bus 210 may include a path for communicating information between various components of computing device 200, such as processor 202, memory unit 204, input/output interface 206, communication interface 208, and storage device 212. Optionally, the bus 210 can use a wired connection or a wireless communication mode, which is not limited in this application.

It should be noted that although the computing device 200 shown in FIG. 2 only shows the processor 202, the memory unit 204, the input/output interface 206, the communication interface 208, the bus 210, and the storage device 212, in a specific implementation process, the field Those skilled in the art will appreciate that computing device 200 also includes other devices necessary to achieve proper operation.

The computing device 200 can be a general purpose computer or a special purpose computing device, including but not limited to a portable computer, a personal desktop computer, a network server, a tablet computer, a mobile phone, a personal digital assistant (PDA), or the like. Or a combination of two or more of the above, the present application does not limit the specific implementation of the computing device 200.

Moreover, computing device 200 of FIG. 2 is merely an example of one computing device 200, which may include more or fewer components than those shown in FIG. 2, or have different component configurations. Those skilled in the art will appreciate that computing device 200 may also include hardware devices that implement other additional functions, depending on the particular needs. Those skilled in the art will appreciate that computing device 200 may also only include the components necessary to implement embodiments of the present invention, and does not necessarily include all of the devices shown in FIG. At the same time, the various components shown in Figure 2 can be implemented in hardware, software, or a combination of hardware and software.

The hardware structure shown in FIG. 2 and the foregoing description are applicable to various computing devices provided by the embodiments of the present invention, and are applicable to performing various resource allocation methods provided by the embodiments of the present invention.

As shown in FIG. 2, the memory unit 204 of the computing device 200 includes a resource allocation module, and the processor 202 executes the resource allocation module program code to implement resource management and allocation.

The resource allocation module can be comprised of one or more operational instructions to cause computing device 200 to perform one or more method steps in accordance with the above description. The specific method steps are described in detail in the following sections of this application.

3 is a signaling diagram of a resource allocation process according to an embodiment of the present invention. The distributed system includes a resource manager 112, a node manager 108, and at least two nodes 106. The resource manager 112 is configured to manage the distributed system. The node resource is used by the node manager 108 to start the resource container 110 based on the node resource. The resource container 110 encapsulates the resource of the node 106 for performing the task of the application. As shown in FIG. 3, the resource allocation process includes:

302: The resource manager 112 launches the application controller 104.

Specifically, the resource manager 112 is configured to manage resources of the at least two nodes 106, and launch the application controller 104 when the trigger timing is met, and configure an initial specification of the resource pool 114 managed by the application controller 104.

The resource manager 112 may start the application controller 104 when the system is initialized; or may be dynamically started by the user according to the requirements according to the requirements, for example, the resource manager 112 may receive the pre-launched application controller. The request of 112, or the request to pre-configure the resource pool 114, initiates the application controller 104; or is initiated by the resource manager 112 according to its own resource status, for example, when each application is frequently received When the resource is requested, the application controller 104 of the corresponding application type can be started. It should be understood that embodiments of the present invention do not limit the startup form of the application controller 104.

Specifically, when the application manager 104 is started, the resource manager 112 configures an initial specification of the resource pool 114 managed by the application controller 104, so that when the application controller 104 subsequently applies for resources to the resource manager 112, This initial specification determines the number and specifications of resource containers that are applied to the resource manager 112.

The initial specification of the resource pool 114 may be the number of resource containers initially included in the resource pool 114, and the specifications of each resource container, where the specification of the resource container is the resource type included in the resource container, and the number of each type of resource; Resource container specifications are set in advance, then resource pool The initial specification of 114 may be the number of resource containers initially included in the resource pool 114; if the specifications of the resource container are configured in advance, and the resource container has multiple specifications, the resource types in each specification and the number of resources of each type are If set in advance, the initial specification of the resource pool 114 may be the resource container specification identifier initially included in the resource pool 114, and the number of resource containers for each specification.

The initial specification of the resource pool 114 may also be the type of various resources initially included by the resource pool 114, as well as the number of each resource.

It should be understood that the embodiment of the present invention does not limit the form of the initial specification of the resource pool 114, and the representation of the number of resources may be different in different scenarios.

Specifically, the resource manager 112 may configure an initial specification of the resource pool 114 managed by the application controller 104 according to the expected resource requirement information of the preset application; the resource manager 112 may also be based on the collected node resources of the distributed system. The usage information configures the initial specifications of the resource pool 114 managed by the application controller 104. For example, resource manager 112 may configure an initial specification of a larger resource pool 114 when there are sufficient unused node resources; resource manager 112 may configure smaller resources when there are fewer unused node resources The initial specification of pool 114. Additionally, the initial specification of the resource pool 114 can be carried by the user in the startup command of the application controller 104, and the resource manager 112 configures the application controller 104 when the application controller 104 is launched.

It should be understood that the application controller 104 may have a one-to-one correspondence with the application, because different applications have different requirements for resource containers, and the types of resources and the number of each resource in the resource containers corresponding to different applications may be different, and resource pools corresponding to different applications. The initial specifications of the 114 may also be different, which is not limited by the embodiment of the present invention.

304: The application controller 104 sends a first resource request to the resource manager 112.

The application controller 104 may also register with the resource manager 112 before sending the first resource request to the resource manager 112 to facilitate subsequent resource manager 112 to manage the application controller 104.

The application controller 104 can also register with a registration server, and the address information of the registration server is distributed to the client 102 for the subsequent client 102 to query the address information of the application controller 104 through the registration server.

The registration server can store the mapping between the type of the application and the application controller 104. The type of the application can be MapReduce, Giraph, Storm, Spark, Tez/Impala, or MPI. For example, the registration server stores the Spark application and the corresponding Spark. Applied application controller 104 Corresponding relationship, when the application running on the client 102 is Spark, the client 102 can find the address information of the application controller 104 corresponding to the Spark application according to the correspondence. It is also possible to set up a separate registration server for a certain type of application. For example, a corresponding registration server can be set specifically for a Spark application. The embodiment of the present invention does not limit the specific implementation manner of the registration server.

Optionally, each type of application may correspond to multiple application controllers 104, and the multiple application controllers 104 may be used by different users, thereby facilitating the expansion of the distributed system scale.

In a specific implementation process, the correspondence between the client 102 and the application controller 104 may be maintained on the registration server, or multiple registration servers may be set, and each registration server corresponds to a different user. It should be understood that the embodiment of the present invention does not limit the number of application servers corresponding to each type of application and the implementation manner of the registration server.

Specifically, the application controller 104 carries the quantity of resources determined according to the initial specification of the resource pool in the first resource request sent to the resource manager 112.

The number of resources may be the number of resource containers requested by the application controller 104 to the resource manager 112 and the specifications of each resource container, wherein the specification of the resource container refers to the kind of resources included in the resource container and the number of each resource. The resources included in each resource container may include one or more of a processor resource, a memory resource, a network, a disk, and the like. It should be understood that, according to the type of application, the resource type in the corresponding requested resource container and the number of each resource may be different, which is not limited by the embodiment of the present invention.

If the specification of the resource container is configured in advance, that is, the resource type included in the resource container and the number of each resource are set in advance, the request that is carried in the first resource request sent by the application controller 104 to the resource manager 112 is The number of resources can be the number of resource containers. If the specification of the resource container is configured in advance, and the resource container has multiple specifications, the resource type in each specification and the resource quantity of each type are set in advance, the resource quantity may be the application controller 104 to the resource manager. The resource container specification identifier carried in the first resource request sent by 112, and the number of resource containers of each specification.

The amount of resources may also be the kind of various resources that the application controller 104 needs, as well as the number of each resource.

It should be understood that the embodiment of the present invention does not limit the form of the quantity of resources requested in the first resource request, and the representation of the quantity of resources may be different in different scenarios.

In a specific implementation process, the first resource request may further carry node information and/or rack information in which the resource container requested by the application controller 104 is located. For example, the application controller 104 may preferably host a shorter link with the application controller 104 as a node of the resource container running the request, thereby enabling more efficient control of the resource pool 114.

306: The resource manager 112 determines a node to allocate a resource container to the application controller 104 on the determined node.

The resource manager 112 is responsible for global resource management and allocation. After the resource manager 112 receives the first resource request of the resource pool 114 sent by the application controller 104 according to the initial specification of the resource pool 114, according to the first resource request, An initial resource container is allocated for the resource pool. Specifically, after receiving the first resource request, the resource manager 112 first determines an optional node, and according to the quantity of resources requested in the first resource request of the resource pool 114, the resource pool 114 is on the optional node. Assign the initial resource container.

Optionally, if the first resource request carries the requested resource container preferred host node and/or the rack, after receiving the first resource request, the resource manager 112 preferentially obtains from the host node and/or the rack The application controller 112 allocates a resource container. If the host node and/or the rack load requested in the first resource request cannot be satisfied currently, for example, the requested host cannot satisfy the request due to a load problem, it may be in the rack where the node is located. A resource container is allocated for the application controller 112. If the rack cannot satisfy the request due to load balancing, the resource controller can be allocated to the application controller 112 in the rack adjacent to the rack.

In another possible implementation manner, if the first resource request does not carry the requested resource container preferred host or rack, the resource manager 112 may allocate the resource container to the application controller 104 according to the load balancing of the node, or The application controller 104 is preferably allocated resources on a host or rack that has a short link to the application controller 104.

It should be appreciated that embodiments of the present invention do not define a policy for resource manager 112 to allocate resource containers to application controllers 104.

308: The resource manager 112 sends a first resource allocation message to the application controller 104.

Specifically, the resource manager 112 sends a first resource allocation message of the resource pool 114 to the application controller 104, where the first resource allocation message includes an initial resource allocated by the resource manager 112 to the resource pool 114 managed by the application controller 104. Information about the node where the container is located.

Specifically, the first resource allocation message may carry the resource manager 112 as an application controller. 104 The specifications of each resource container allocated and the information of the node where each resource container is located. The specification of a resource container is the kind of resource that the resource container contains, and the number of each resource.

If the specification of the resource container is configured in advance by the system, that is, the resource type included in the resource container and the quantity of each resource are set in advance, the information of the node where each resource container is located may be carried in the first resource allocation message. If the resource manager 112 allocates multiple resource containers to the application controller 104 on the same node, the first resource allocation message may also carry the number of resource containers allocated in each node.

If the resource container specification is configured in advance, and the resource container has multiple specifications, the resource type in each specification and the resource quantity of each type are set in advance, the first resource allocation message may carry the allocation. The specification of the resource container and the information of the node where each resource container is located.

In a specific implementation process, after receiving the first resource request from the application controller 104, the resource manager 112 may not immediately return the resource that meets the requirements for the application controller 104, but requires the application controller 104 to continuously and resource management. The device 112 communicates, detects the allocated resources, and pulls them past.

310: The application controller 104 launches a resource container.

The application controller 104 sends a start request to the node manager requesting the node manager to initiate the initial resource container of the resource pool 114.

Specifically, after the application controller 104 obtains the resource container allocated to the resource manager 112, it sends a startup request to each node where the resource container is located, and more specifically, to the node of the node where each resource container is located. The manager sends the start request to start the resource container for which the resource manager 112 is assigned.

The startup request also carries the specification of the resource container, and the specification of the resource container is the type of the resource container including the resource, and the quantity of each resource.

If the specification of the resource container is configured in advance, and the resource container has multiple specifications, the resource type in each specification and the number of resources in each type are set in advance, the resource container that requests the startup is also carried in the startup request. Specification of the logo.

Optionally, if the resource manager 112 allocates multiple resource containers on the same node for the application controller 104, the multiple resource containers may be initiated by carrying a request in one startup request.

After receiving the startup request, the node manager of the node where the resource container is located first performs resource localization, that is, creates a working directory of the resource container, and downloads each required for running the resource container from the distributed file system (Hadoop Distributed File System, HDFS). Resources (jar packages, executable files, etc.), etc., and then start the resource container.

After the application controller 104 starts the initial resource container allocated by the resource manager 112 for the resource pool 114 through the node manager, the activated resource container constitutes the resource pool 114.

Because the resource containers in the resource pool 114 are already in the startup state, when the task is received, the execution can be performed quickly, and the startup time before the task is avoided, thereby speeding up the execution of the application task.

312: The application controller 104 receives a resource allocation request from the client 102.

The client 102 runs an application, and the type of the application may be MapReduce, Giraph, Storm, Spark, Tez/Impala, or MPI.

When the application running on the client 102 requires resources for task processing, the client 102 sends a resource allocation request to the application controller.

In a specific implementation process, the client 102 can query the address information of the application controller 104 corresponding to the application through a registration server, and send the resource allocation request to the application controller 104 indicated by the corresponding address information. The registration server stores the corresponding information of the application controller 104 and the application, and different applications may have different application controllers 104.

Because different types of applications have different resource requirements for resource containers, the resource containers of different applications have various specifications, and an application controller 104 and a resource pool 114 are configured for each type of application, so that rational utilization of resource containers can be realized.

The Spark application is used as an example. The specific implementation process can be as follows: The user submits a spark application through the spark-submit. The user program automatically creates a SparkContext. The SparkContext is used by the Spark to provide the external interface for the Spark. Create a DAG scheduler (DAGScheduler) and a task scheduler (TaskScheduler), where DAGScheduler is used to calculate and schedule according to the application's DAG dependency, TaskScheduler to perform task calculation and schedule, and TaskScheduler to create a scheduler backend (SchedulerBackend). The schedulerBackend queries the registration server, searches for the address information of the application controller 104 corresponding to the Spark application, and sends the resource allocation request to the corresponding application controller 104.

The resource allocation request carries resource requirement information of the application requested by the client 102 to the application controller 104.

The resource requirement information of the application in the resource allocation request may be the number of resource containers requested, and the specifications of each resource container. The specification of a resource container is the kind of resource that the resource container contains, and the number of each resource.

If the specification of the resource container is configured in advance by the system, that is, the resource type included in the resource container and the quantity of each resource are set in advance, the resource requirement information of the application may be the requested resource container number.

If the specification of the resource container is configured in advance, and the resource container has multiple specifications, the resource type in each specification and the number of resources of each type are set in advance, the resource requirement information of the application may be the requested resource. The specification of the container and the number of resource containers for each specification.

The resource requirement information of the application in the resource allocation request may also be the kind of various resources requested by the client 102 to the application controller 104, and the number of each resource.

Optionally, the resource allocation request further includes user rights information of the client 102, and after receiving the resource allocation request, the application controller 104 verifies the user rights information of the client 102 according to the preset user permission library, where the user rights The library contains user permission information for different users. If the user rights information of the client 102 is not included in the user rights library, the resource allocation request of the client 102 is rejected or not; if the user rights library contains the user rights information of the client 102, the following steps are performed.

314: The application controller 104 allocates a resource container to the client 102.

After receiving the resource allocation request from the client 102, the application controller 104 selects an idle resource container from the resource pool 114 and allocates it to the client 102 according to the resource requirement information of the application in the resource allocation request.

Specifically, after receiving the resource allocation request from the client 102, the application controller 104 first determines an idle resource container in the resource pool 114, and according to the resource requirement information of the application carried in the resource allocation request, from the idle resource container. A resource container is allocated for the client 102. The idle resource container refers to a resource container that is not currently assigned to perform an application task.

Because the resource containers in the resource pool 114 controlled by the application controller 104 can be simultaneously allocated for use by the plurality of clients 102, one resource container can only perform one client 102 allocation at the same time. The task, so the application controller 104 needs to first determine the currently idle resource container before allocating the resource container to the client 102, thereby allocating the requested resource from the idle resource container to the client 102.

The application controller 104 can determine whether the current resource container is currently in an idle state by sending a query message to each resource container in the resource pool 114.

Preferably, the application controller 104 maintains state information of each resource container in the resource pool 114, the state information indicating whether the resource container is in an idle state, and if the resource container is currently allocated and other tasks are running, the resource container The status of the resource container is idle. If the resource container is not currently allocated and no other tasks are running, the status of the resource container is idle.

After receiving the resource allocation request of the client 102, the application controller 104 may determine the resource container in the resource pool 114 according to the state information of each resource container in the resource pool 114, and then request the resource requirement of the application according to the resource allocation request. Information, the resource container is allocated to the client 102 from the currently idle resource container.

The application controller 104 selects an idle resource container from the resource pool 114 and allocates it to the client 102. The application controller 104 selects an idle resource container in the resource pool 114 according to the state information of each resource container in the resource pool 114. The resource containers that are free in the selected resource pool 114 are allocated to the client 102.

For convenience of description, the resource container allocated by the application controller 104 to the client 102 is referred to as a first resource container group, and the first resource container group includes at least one resource container.

If the application controller 104 maintains state information for each resource container in the resource pool 114, the application controller 104 allocates the resource containers in the first resource container group to the client 102, and also each of the first resource container groups. The status of the resource container is set to not idle.

Optionally, the application controller 104 further determines the number of idle resource containers remaining in the resource pool 114, and sends the second resource to the resource manager 112 if the number of remaining idle resource containers is less than a preset first threshold. And requesting, and receiving the second resource allocation message of the resource pool 114 sent by the resource manager 112, sending a startup request to the node manager according to the second resource allocation message of the resource pool 114, requesting the node manager to start as the resource pool 114 The newly added resource container. Here, the second resource request may carry the requested resource quantity, and the requested resource quantity may be a difference between the first threshold and the current idle resource container number, or may be positively correlated with the difference. The form of the second resource request is similar to the form of the first resource request, and details are not described herein again. The second resource allocation message is similar to the first resource allocation message, and details are not described herein again.

During the specific implementation process, the application server 104 may determine the number of idle resource containers remaining in the resource pool 114 after allocating the resource containers in the first resource container group to the client 102 or according to a preset period. It should be appreciated that the time and manner in which the application server 104 determines the number of free resource containers remaining in the resource pool 114 is not limited by the inventive embodiment.

316: The application controller 104 sends an indication message to each resource container in the first resource container group, where the indication message carries the identifier of the client 102.

The resource allocation request sent by the client 102 to the application controller 104 further carries the identifier of the client 102, and the indication message sent by the application controller to each resource container in the first resource container group is used to indicate that the first resource is to be used. Each resource container in the container group is assigned to the client 102. In a specific implementation process, the identifier of the client 102 may be address information of the client 102 and a port number that communicates with the resource container in the first resource container group.

318: Each resource container in the first resource container group sends a registration message to the client 102.

The registration message carries the identification information of the resource container, and the identification information of the resource container is the identification information for the client to uniquely identify the resource container. Optionally, the identifier information is address information of a node where the resource container is located and information of the resource container at the node.

The registration message can also carry the specification of the resource container, that is, the kind of resources in the resource container and the number of each resource.

If the specification of the resource container is configured in advance, and the resource container has multiple specifications, the resource type in each specification and the number of resources in each type are set in advance, the registration message may carry the specification identifier of the resource container.

320: The client 102 assigns the application's job to the resource container that the application controller 114 allocates for the client 102.

After receiving the registration message of the resource container in the first resource container group, the client 102 divides the application job into at least one task, and sends each task to one resource container in the first resource container group.

322: The resource container allocated by the application controller 114 to the client 102 sends the execution result of the task to the client 102.

After the resource container in the first resource container group is assigned to its task execution, the task will be The execution result is sent to the client 102. It should be understood that the execution result here may be an actual task execution result. For a task that does not need to return a result, the execution result here may be a task completion indication message.

324: The client 102 ends the connection with the resource container in the first resource container group.

After receiving the task execution result returned by the resource container in the first resource container group, the client 102 sends an end connection message to the resource container of the first resource container group, ending the connection with the resource container in the first resource container group, and releasing Resource container resource.

326: The resource container in the first resource container group sends a status update message to the application controller 104.

After the client 102 releases the resource container resource in the first resource container group, the resource container in the first resource container group sends a status update message to the application controller 104 indicating that a new task can be received.

If the application controller 104 maintains state information of each resource container in the resource pool 114, the application controller 104 sets the state of the corresponding resource container to idle after receiving the status update message of the resource container in the first resource container group. When there is a new resource allocation request, it can be allocated to the client 102 as an alternative resource container.

Optionally, the application controller 104 further determines the number of idle resource containers remaining in the resource pool 114, and if the number of remaining free resource containers is greater than a preset second threshold, at least one of the idle resource containers The resource container sends a resource release message, and the resource release message is used to release resources occupied by the at least one resource container. The number of resource containers released may be the difference between the number of currently idle resource containers and the second threshold, or may be positively correlated with the difference.

Optionally, the application server 104 further maintains the idle time of each resource container in the resource pool 114. When the idle time of the resource container in the resource pool 114 is greater than a preset third threshold, the application server 104 provides the idle time to be greater than the preset time. The third threshold of the resource container sends a resource release message.

Optionally, when the number of remaining idle resource containers is greater than a preset second threshold, the application server 104 preferentially sends a resource release message to the resource container whose idle time is greater than a preset third threshold. Or when the number of remaining idle resource containers is greater than a preset second threshold, and the idle time of the resource container in the resource pool 114 is greater than a preset third threshold, the application server 104 is idle longer than the preset third. The threshold resource container sends a resource release message.

During the specific implementation process, the application server 104 may receive the resources in the first resource container group. The status update message of the source container, after re-setting the resource container in the first resource container group to the idle state, or determining the number of free resource containers remaining in the resource pool 114 according to a preset period. It should be appreciated that the time and manner in which the application server 104 determines the number of free resource containers remaining in the resource pool 114 is not limited by the inventive embodiment.

It should be understood that, because each type of application has different requirements on the specifications of the resource container, in the specific implementation process, the application controller 104 and the resource pool 114 are separately configured for each type of application, so that the rational use of resources can be realized. However, the embodiment of the present invention does not limit the application, and the application controller 104 and the resource pool 114 can be shared by multiple types of applications.

According to the technical solution disclosed by the embodiment of the present invention, the resource controller is applied to the resource manager in advance by the application controller, and the requested resource container is started in advance. After receiving the resource allocation request of the application, the resource container resource can be allocated in time. By avoiding the waiting time for resource container startup, and by reusing the resource container in the resource pool, resource consumption for multiple opening and closing of the resource container and resource consumption for opening and closing of multiple application managers are avoided. The application controller directly manages the resource containers in the resource pool, enabling more flexible management.

FIG. 4 is a signaling diagram of a resource allocation process according to an embodiment of the present invention. As shown in FIG. 4, the resource allocation process includes:

Steps 402-410 refer to steps 302-310 of the embodiment of FIG. 3, and details are not described herein again.

412: The client 102 sends an application job to the application controller 104.

The client runs an application of the user, and the type of the application may be MapReduce, Giraph, Storm, Spark, Tez/Impala, or MPI.

When the application running on the client needs resources for task processing, the application job is sent to the application controller through the client 102.

In a specific implementation process, the client 102 can query the address information of the application controller 104 corresponding to the application through a registration server, and send the application job to the application controller 104 indicated by the corresponding address information. The registration server stores the corresponding information of the application controller 104 and the application, and different applications may have different application controllers 104.

Because the resource requirements of the resource container are different for different types of applications, and the specifications of the resource containers of different applications are various, an application controller 104 and a resource pool are configured for each type of application, so that the resource container can be rationalized and utilized.

414: The application controller 104 allocates a resource container for the application job of the client 102.

In a specific implementation process, after receiving the application job sent by the client 102, the application controller 104 first determines an idle resource container in the resource pool 114, and divides the application job into at least one task according to the scale of the application job. And allocate a resource container for the application job of the client 102 from the idle resource container according to the number of tasks.

If there are multiple resource containers in the resource pool 114, because the resource types required by different task types or the resources of each resource are different, the application controller 104 also allocates different specifications of resources for different tasks according to the types of tasks. container. For example, if the application type is MapReduce, the task may be a Map type or a Reduce type. The Map task and the Reduce task may have different resource container resources. The application controller 104 may allocate different types of resource containers for different types of resource containers.

Because the resource containers in the resource pool 114 controlled by the application controller 104 can be simultaneously allocated to the plurality of clients 102 for use, but one resource container can only perform one task assigned by the client 102 at the same time, the application controller 104 is the client. Before allocating the resource container 102, it is necessary to first determine the currently idle resource container, thereby allocating resources for the application job of the client 102 from the idle resource container.

Preferably, the application controller 104 maintains state information of each resource container in the resource pool 114, the state information is used to indicate whether the resource container is in an idle state, and if the resource container is currently allocated and other tasks are running, The state of the resource container is not idle. If the resource container is not currently allocated and no other tasks are running, the state of the resource container is idle.

The application controller 104 can determine the resource containers that are free in the resource pool 114 based on the state information of each resource container in the resource pool 114.

For convenience of description, the resource container allocated by the application controller 104 to the application task of the client 102 is referred to as a second resource container group, and the second resource container group includes at least one resource container.

If the application controller 104 maintains state information of each resource container in the resource pool 114, the application controller 104 allocates the resource container in the second resource container group to the task of the application job of the client 102, and also the second resource. The status of each resource container in the container group is set to not idle.

Optionally, the application controller 104 further determines the number of idle resource containers remaining in the resource pool 114, and sends the second resource to the resource manager 112 if the number of remaining idle resource containers is less than a preset first threshold. Requesting, and receiving a second resource allocation message of the resource pool 114 sent by the resource manager 112, according to the second resource allocation message of the resource pool 114, the application controller 104 sends a startup request to the node manager, requesting the node manager to start A new resource container that is allocated for resource pool 114. Here, the number of resources of the second resource request request may be a difference between the first threshold and the current number of idle resource containers, or may be positively correlated with the difference.

In a specific implementation process, the application server 104 may determine the number of idle resource containers remaining in the resource pool 114 after the resource container in the second resource container group is allocated to the application job of the client 102, or according to a preset period. . It should be appreciated that the time and manner in which the application server 104 determines the number of free resource containers remaining in the resource pool 114 is not limited by the inventive embodiment.

416: The application controller 104 delivers a task to each resource container in the second resource container group.

The application controller 104 divides the job of the client 102 into at least one task, and selects a resource container for each task, and then delivers the corresponding task to the resource container in the second resource container group.

418: The application controller 104 receives the task execution result returned by each resource container in the second resource container group.

After the resource container in the second resource container group finishes executing the task issued by the application controller 104, the execution result of the task is returned to the application controller 104. It should be understood that the execution result here may be an actual task execution result. For a task that does not need to return a result, the execution result here may be a task completion indication message.

If the application controller 104 maintains the state information of each resource container in the resource pool 114, the application controller 104 receives the task execution result returned by the resource container in the second resource container group, and sets the state of the corresponding resource container to Idle, when there is a new application job, it can be assigned to the new application job as an alternate resource container.

Optionally, the application controller 104 further determines the number of idle resource containers remaining in the resource pool 114, and if the number of remaining free resource containers is greater than a preset second threshold, at least one of the idle resource containers The resource container sends a resource release message, and the resource release message is used to release resources occupied by the at least one resource container. The number of resource containers released can be current The difference between the number of idle resource containers and the second threshold, or a positive correlation with the difference.

During the specific implementation process, the application server 104 may receive the execution result returned by each resource container in the second resource container group, reset the resource container of the second resource container group to the idle state, or follow the preset. The period determines the number of free resource containers remaining in resource pool 114. It should be appreciated that the time and manner in which the application server 104 determines the number of free resource containers remaining in the resource pool 114 is not limited by the inventive embodiment.

420: The application controller 104 returns the execution result of the application job of the client 102 to the client 102.

In a specific implementation, the application controller 104 may merge the execution results returned by each resource container in the second resource container group, and then send the merged execution result to the client 102.

According to the technical solution disclosed in the embodiment of the present invention, the resource controller is applied to the resource manager in advance by the application controller, and the requested resource container is started in advance. After receiving the application job of the client, the resource container resource can be allocated in time. The waiting time of the resource container startup is avoided, and the resource consumption of the resource container in the resource pool is avoided, and the resource consumption of the resource container being opened and closed multiple times and the resource consumption of multiple application managers being turned on and off are avoided. The controller directly manages the resource containers in the resource pool, enabling more flexible management.

FIG. 5 is a schematic diagram of a logical structure of a resource container allocation apparatus 500 in a distributed system according to an embodiment of the present invention. The distributed system includes a device 500 and a node manager, where the apparatus 500 is used. The node resource of the distributed system is managed, the node manager is configured to start the resource container based on the node resource, and the resource container is used to perform the task of the application. As shown in FIG. 5, the device 500 includes a starting unit 502, a receiving unit 504, an allocating unit 506, and processing. Unit 508, wherein

The startup unit 502 is configured to start the application controller when the trigger timing is met, and configure an initial specification of the resource pool managed by the application controller.

The activation unit 502 can start the application controller upon receiving a request to start the application controller in advance or receiving a request for pre-configuring the resource pool. The startup unit 502 can be started when the system is initialized, or can be dynamically started by the user according to the requirements according to the requirements, or can be started by the resource manager according to the resource status of the user, and the embodiment of the present invention does not Define the startup form of the application controller.

Specifically, the initiating unit 502 may configure an initial specification of the resource pool managed by the application controller according to the expected resource requirement information of the preset application; the initiating unit 502 may further use the node resource of the distributed system collected by the device 500. Information, configure the initial specifications of the resource pool managed by the application controller. For example, when there are sufficient unused node resources, the startup unit 502 can configure an initial specification of a larger resource pool; when there are fewer unused node resources, the startup unit 502 can configure an initial of a smaller resource pool. specification. In addition, the initial specification of the resource pool may be carried by the user in the startup command of the application controller, and the startup controller 502 configures the application controller when the application controller is started.

In a specific implementation process, the receiving unit 502 can be implemented by the processor 202, the memory unit 204, and the communication interface 208 shown in FIG. More specifically, the communication module and resource allocation module in memory unit 204 can be executed by processor 202 to cause communication interface 208 to launch the application server by instruction.

The receiving unit 504 is configured to receive a first resource request of the resource pool that is sent by the application controller according to an initial specification of the resource pool.

In a specific implementation process, the receiving unit 504 can be implemented by the processor 202, the memory unit 204, and the communication interface 208 shown in FIG. More specifically, the communication module in memory unit 204 can be executed by processor 202 to cause communication interface 208 to receive a resource request from a resource pool that the application controller sent according to the initial specification of the resource pool.

The allocating unit 506 is configured to allocate an initial resource container to the resource pool according to the first resource request.

In a specific implementation process, the allocating unit 506 can be implemented by the processor 202 and the memory unit shown in FIG. 2. More specifically, resource allocation in memory unit 204 can be performed by processor 202. A module that allocates node resources for a resource pool to an application controller.

The sending unit 508 is configured to send, to the application controller, a first resource allocation message of the resource pool, where the first resource allocation message includes information about a node where the initial resource container allocated by the allocation unit 506 is allocated to the resource pool.

In a specific implementation process, the sending unit 508 can be implemented by the processor 202, the memory unit 204, and the communication interface 208 shown in FIG. 2. More specifically, the communication module in memory unit 204 can be executed by processor 202 to cause communication interface 208 to send a first resource allocation message for the resource pool to the application controller.

The embodiment of the present invention is an apparatus embodiment of the resource manager in the embodiment of FIG. 3 and FIG. 4, and the feature description of the embodiment of FIG. 3 and FIG. 4 is applicable to the embodiment of the present invention, and details are not described herein again.

FIG. 6 is a schematic diagram showing the logical structure of a resource container allocation apparatus 600 in a distributed system according to an embodiment of the present invention. The distributed system includes a resource manager and a node manager, and the resource manager is used to manage node resources and nodes of the distributed system. The manager is configured to start a resource container based on the node resource, where the resource container is used to execute the task of the application, and the resource manager starts the device 600 when the triggering time is met, and configures the initial specification of the resource pool managed by the device 600, and according to the initial specification of the resource pool, The resource pool allocates an initial resource container, and the initial resource container in the resource pool is started;

Apparatus 600 includes:

The receiving unit 602 is configured to receive a resource allocation request from the client, where the resource allocation request is used to request a resource container for the application running on the client, where the resource allocation request carries resource requirement information of the application.

In a specific implementation process, the receiving unit 602 can be implemented by the processor 202, the memory unit 204, and the communication interface 208 shown in FIG. More specifically, the communication module in memory unit 204 can be executed by processor 202 to cause communication interface 208 to receive a resource allocation request from the client.

The allocating unit 604 is configured to select, from the resource pool, an idle resource container to be allocated to the client according to the resource requirement information of the application.

In a specific implementation process, the allocating unit 604 can be implemented by the processor 202 and the memory unit shown in FIG. 2. More specifically, the resource allocation module in the memory unit 204 can be executed by the processor 202 to allocate the node resources of the resource pool to the application controller.

As shown in FIG. 7, the apparatus 600 further includes a sending unit 606. Before the receiving unit 602 receives the resource allocation request from the client, the sending unit 606 is configured to send the first resource request to the resource manager according to the initial specification of the resource pool, where The resource request carries the initial specification according to the resource pool. The receiving unit 602 is further configured to receive a first resource allocation message of the resource pool sent by the resource manager, where the first resource allocation message includes an initial resource container allocated by the resource manager for the resource pool. The information of the node where the node is located; the sending unit 606 is further configured to send a startup request to the node manager, requesting the node manager to start the initial resource container of the resource pool.

Specifically, the resource allocation request includes the identifier of the client, and after the allocating unit 604 selects the idle resource container from the resource pool and allocates the data to the client, the sending unit 606 is configured to send an indication message to each resource container in the resource container allocated to the client. The indication message carries the identifier of the client, and the indication message is used to indicate that each resource container is allocated to the client.

As shown in FIG. 7, the apparatus 600 further includes a determining unit 608, configured to determine the number of free resource containers remaining in the resource pool, and if the number of remaining free resource containers is less than a preset first threshold, send The unit 606 is configured to send a second resource request to the resource manager, and the receiving unit 602 is further configured to receive the second resource allocation message of the resource pool sent by the resource manager, where the sending unit 606 is further configured to use the second resource according to the resource pool. Assign a message, send a start request to the node manager, and request the node manager to start the new resource container allocated for the resource pool.

If the number of remaining free resource containers is greater than a preset second threshold, the sending unit 606 is configured to send a resource release message to the at least one resource container in the idle resource container, where the resource release message is used to release at least one resource container occupation resource of.

Optionally, the allocating unit 604 is further configured to maintain state information of each resource container in the resource pool, where the state information indicates whether the corresponding resource container is idle; and the allocating unit 604 is configured to select an idle resource container from the resource pool. The allocation to the client includes: an allocating unit 604, configured to select an idle resource container in the resource pool according to status information of each resource container in the resource pool, and allocate the selected resource container in the selected resource pool to The client.

After the allocation unit 604 selects an idle resource container from the resource pool and allocates it to the client, it also sets the state of each resource container in the resource container allocated to the client to be not idle.

After the allocating unit 604 sets the state of each resource container in the resource container allocated to the client to not idle, the receiving unit 602 is further configured to receive a status update message from each of the resource containers allocated to the client, the status update. The message is used to indicate that the task of the client allocation is completed; the allocating unit 604 is further configured to set the state of each resource container in the resource container allocated to the client to be idle according to the status update information.

Optionally, the resource allocation request further includes user rights information; the receiving unit 602 is further configured to verify user rights information according to the preset user rights library, where the user rights library includes user rights information. The embodiment of the present invention is an apparatus embodiment of the application controller in the embodiment of FIG. 3, and the feature description of the embodiment of FIG. 3 is applicable to the embodiment of the present invention, and details are not described herein again.

In the several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and may be implemented in another manner, for example, multiple modules or components may be combined or may be Integrate into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or module, and may be electrical, mechanical or otherwise.

The modules described as separate components may or may not be physically separated. The components displayed as modules may or may not be physical modules, that is, may be located in one place, or may be distributed to multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of hardware plus software function modules.

The above-described integrated modules implemented in the form of software function modules can be stored in a computer readable storage medium. The software functional modules described above are stored in a storage medium and include instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform some of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: a mobile hard disk, a read-only memory (English: Read-Only Memory, ROM for short), a random access memory (English: Random Access Memory, RAM for short), a magnetic disk or an optical disk, and the like. The medium of the code.

It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not limited thereto; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that The technical solutions described in the foregoing embodiments are modified, or some of the technical features are equivalently replaced; and the modifications or substitutions do not deviate from the technical scope of the embodiments of the present invention.

Claims

A distributed system, characterized in that the distributed system comprises a resource manager and a node manager;

The resource manager is configured to start an application controller when a triggering time is met, configure an initial specification of a resource pool managed by the application controller, and receive a location that is sent by the application controller according to an initial specification of the resource pool. The first resource request of the resource pool is allocated to the resource pool according to the first resource request, and the first resource allocation message of the resource pool is sent to the application controller;

The application controller is configured to acquire a first resource allocation message of the resource pool sent by the resource manager, and the initial allocation of the resource pool indicated in the first resource allocation message of the resource pool is Information about a node where the resource container is located, requesting the node manager to start an initial resource container of the resource pool;

The node manager is configured to start the initial resource container according to a request of the application controller.
The system according to claim 1, wherein the resource manager is configured to start an application controller when a trigger timing is met, comprising: the resource manager receiving a request to start the application controller in advance, or pre- The application controller is started when the request for the resource pool is configured.
The system according to claim 1 or 2, wherein the resource manager is configured to configure an initial specification of a resource pool managed by the application controller, the resource manager configured to use the application according to a preset The expected resource requirement information, or the initial specification of the resource pool managed by the application controller, according to the collected usage information of the node resource of the distributed system.
The system according to any one of claims 1 to 3, wherein the application controller is further configured to: receive a resource allocation request from a client, where the resource allocation request is used for running on the client The application requests the resource container, and selects an idle resource container from the resource pool to allocate to the client according to the resource requirement information of the application in the resource allocation request.
The system according to claim 4, wherein the resource allocation request includes an identifier of the client, and after the application controller selects an idle resource container from the resource pool to allocate to the client, Also used for: each resource in a resource container allocated to the client The source container sends an indication message, where the indication message carries the identifier of the client.
The system according to any one of claims 1 to 5, wherein the application controller is further configured to: determine the number of free resource containers remaining in the resource pool, if the remaining idle resource containers are Sending a second resource request to the resource manager, and receiving a second resource allocation message of the resource pool sent by the resource manager, and according to the second resource pool, the number is less than a preset first threshold. a resource allocation message, sending a startup request to the node manager, requesting the node manager to start an additional resource container allocated for the resource pool.
The system according to any one of claims 1 to 6, wherein the application controller is further configured to: determine the number of free resource containers remaining in the resource pool, if the remaining idle resource containers are If the number is greater than the preset second threshold, the resource release message is sent to at least one resource container in the idle resource container.
A method for allocating resource containers in a distributed system, characterized in that the distributed system comprises a resource manager for managing node resources of the distributed system, and a node manager for Starting a resource container based on the node resource, the resource container for performing a task of an application;

The method includes:

The resource manager starts the application controller when the trigger timing is met, and configures an initial specification of the resource pool managed by the application controller;

Receiving, by the resource manager, a first resource request of the resource pool that is sent by the application controller according to an initial specification of the resource pool;

The resource manager allocates an initial resource container to the resource pool according to the first resource request;

The resource manager sends a first resource allocation message of the resource pool to the application controller, where the first resource allocation message includes an initial resource container allocated by the resource manager for the resource pool. Node information.
The method according to claim 8, wherein the starting the application controller when the resource manager meets the trigger timing comprises:

The resource manager starts the application controller when receiving a request to start the application controller in advance, or pre-configuring the resource pool.
The method according to claim 8 or 9, wherein the resource manager configures an initial specification of a resource pool managed by the application controller, including:

The resource manager configures an initial specification of a resource pool managed by the application controller according to the expected resource requirement information of the application that is set in advance; or

The resource manager configures an initial specification of the resource pool managed by the application controller according to the collected usage information of the node resource of the distributed system.
A method for allocating resource containers in a distributed system, characterized in that the distributed system comprises a resource manager for managing node resources of the distributed system, and a node manager for The node resource starts the resource container, and the resource manager starts the application controller when the triggering time is met, configures an initial specification of the resource pool managed by the application controller, and uses the initial specification of the resource pool as the resource pool. Allocating an initial resource container that has been started;

The method includes:

The application controller receives a resource allocation request from a client, where the resource allocation request is used to request a resource container for an application running on the client, where the resource allocation request carries resource requirement information of the application;

The application controller selects an idle resource container from the resource pool to allocate to the client according to the resource requirement information of the application.
The method according to claim 11, wherein before the application controller receives the resource allocation request from the client, the method further includes:

The application controller sends a first resource request to the resource manager according to an initial specification of the resource pool;

The application controller acquires a first resource allocation message of the resource pool sent by the resource manager, where the first resource allocation message includes an initial resource container allocated by the resource manager for the resource pool. Information about the node;

The application controller sends a start request to the node manager, requesting the node manager to initiate an initial resource container of the resource pool.
The method according to claim 11 or 12, wherein the resource allocation request includes an identifier of the client, and the application controller selects an idle resource container from the resource pool and allocates the identifier to the client. Thereafter, the method further includes:

The application controller sends an indication message to each resource container in the resource container allocated to the client, where the indication message carries the identifier of the client.
Method according to any of claims 11-13, characterized in that the method further include:

The application controller determines the number of idle resource containers remaining in the resource pool, and if the number of remaining free resource containers is less than a preset first threshold, sending a second resource request to the resource manager;

The application controller receives a second resource allocation message of the resource pool sent by the resource manager;

And the application controller sends a startup request to the node manager according to the second resource allocation message of the resource pool, and requests the node manager to start a new resource container allocated for the resource pool.
The method according to any one of claims 11 to 14, wherein the method further comprises:

The application controller determines the number of idle resource containers remaining in the resource pool, and if the number of remaining free resource containers is greater than a preset second threshold, sending to at least one resource container in the idle resource container Resource release message.
The method according to any one of claims 11 to 15, wherein the application controller maintains state information of each resource container in the resource pool, and the state information indicates whether the corresponding resource container is idle;

The selecting, by the application controller, the idle resource container from the resource pool to be allocated to the client includes:

The application controller selects an idle resource container in the resource pool according to status information of each resource container in the resource pool, and allocates the selected resource container in the selected resource pool to the client. .
A device for allocating resource containers in a distributed system, characterized in that the distributed system comprises the device and a node manager, the device is for managing node resources of the distributed system, and the node manager is used for The node resource starts a resource container, and the resource container is used to perform an application task,

The device includes:

a startup unit, configured to start an application controller when a trigger timing is met, and configure an initial specification of a resource pool managed by the application controller;

a receiving unit, configured to receive a first resource request of the resource pool that is sent by the application controller according to an initial specification of the resource pool;

An allocating unit, configured to allocate an initial resource container to the resource pool according to the first resource request;

a sending unit, configured to send, to the application controller, a first resource allocation message of the resource pool, where the first resource allocation message includes a node where an initial resource container allocated by the allocation unit is configured by using the resource pool Information.
The device according to claim 17, wherein the activation unit is configured to start the application controller when the trigger timing is met, including:

The startup unit is configured to start the application controller when the receiving unit receives a request to start the application controller in advance, or pre-configures the request of the resource pool.
The device according to claim 17 or 18, wherein the activation unit is configured to configure an initial specification of a resource pool managed by the application controller, including:

The startup unit is configured to configure an initial specification of a resource pool managed by the application controller according to the expected resource requirement information of the application that is set in advance; or

The startup unit is configured to configure an initial specification of a resource pool managed by the application controller according to the collected usage information of the node resource of the distributed system.
A device for allocating resource containers in a distributed system, characterized in that the distributed system comprises a resource manager for managing node resources of the distributed system, and a node manager for The node resource starts a resource container, and the resource manager starts the device when the triggering time is met, configures an initial specification of the resource pool managed by the device, and allocates an initial allocation to the resource pool according to an initial specification of the resource pool. Resource container, the initial resource container has been started;

The device includes:

a receiving unit, configured to receive a resource allocation request from a client, where the resource allocation request is used to request a resource container for an application running on the client, where the resource allocation request carries resource requirement information of the application;

And an allocating unit, configured to select, from the resource pool, an idle resource container to be allocated to the client according to the resource requirement information of the application.
The apparatus according to claim 20, wherein said apparatus further comprises a transmitting unit, said transmitting unit is configured to use an initial specification of said resource pool before receiving said resource allocation request from said client Sending a first resource request to the resource manager;

The receiving unit is further configured to: receive the first resource of the resource pool sent by the resource manager a source allocation message, where the first resource allocation message includes information of a node where the initial resource container allocated by the resource manager for the resource pool is located;

The sending unit is further configured to: send a startup request to the node manager, requesting the node manager to start an initial resource container of the resource pool.
The device according to claim 20 or 21, wherein the device further comprises a sending unit, wherein the resource allocation request includes an identifier of the client, and the allocating unit selects an idle resource from the resource pool. After the container is allocated to the client, the sending unit is configured to: send an indication message to each resource container in the resource container allocated to the client, where the indication message carries the identifier of the client.
The apparatus according to any one of claims 20 to 22, wherein the apparatus further comprises a determining unit, and a transmitting unit, the determining unit is configured to determine the number of free resource containers remaining in the resource pool The sending unit is configured to send a second resource request to the resource manager, if the number of remaining free resource containers is less than a preset first threshold;

The receiving unit is further configured to: receive a second resource allocation message of the resource pool sent by the resource manager;

The sending unit is further configured to: send, according to the second resource allocation message of the resource pool, a startup request to the node manager, requesting the node manager to start a new resource container allocated for the resource pool.
The apparatus according to any one of claims 20 to 23, characterized in that the apparatus further comprises a determining unit, and a transmitting unit, the determining unit for determining the number of free resource containers remaining in the resource pool And if the number of remaining idle resource containers is greater than a preset second threshold, the sending unit is configured to send a resource release message to at least one resource container in the idle resource container.
The device according to any one of claims 20 to 24, wherein the allocating unit is further configured to maintain state information of each resource container in the resource pool, the state information indicating whether the corresponding resource container is idle. ;

The allocating unit is configured to select an idle resource container from the resource pool, and the allocation unit is configured to select the resource pool according to status information of each resource container in the resource pool. An idle resource container, and the selected resource container in the selected resource pool is allocated to the client.
A computer readable medium, comprising: a computer executing instructions The computer executes the method of any of claims 8-10 when the processor of the computer executes the computer-executed instructions.
A computing device, comprising: a processor, a memory, a bus, and a communication interface;

The memory is configured to store execution instructions, the processor is coupled to the memory via the bus, and when the computing device is running, the processor executes the execution instructions stored by the memory to cause the The apparatus performs the method of any of claims 8-10.
A computer readable medium, comprising computer executed instructions for performing the method of any of claims 11-16 when a processor of a computer executes the computer to execute an instruction.
A computing device, comprising: a processor, a memory, a bus, and a communication interface;

The memory is configured to store execution instructions, the processor is coupled to the memory via the bus, and when the computing device is running, the processor executes the execution instructions stored by the memory to cause the The apparatus performs the method of any of claims 11-16.