CN109783225B - Tenant priority management method and system of multi-tenant big data platform - Google Patents
Tenant priority management method and system of multi-tenant big data platform Download PDFInfo
- Publication number
- CN109783225B CN109783225B CN201811514328.XA CN201811514328A CN109783225B CN 109783225 B CN109783225 B CN 109783225B CN 201811514328 A CN201811514328 A CN 201811514328A CN 109783225 B CN109783225 B CN 109783225B
- Authority
- CN
- China
- Prior art keywords
- tenant
- priority
- resource
- manager
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a tenant priority management method of a multi-tenant big data platform, which comprises the following steps: initializing a tenant priority manager, wherein the tenant priority manager consists of a resource request waiting queue and a radial basis function neural network; when the cluster load is lighter, the resource request of each tenant can be satisfied, and the tenant priority manager is not started at the moment; when the resource request of the tenant cannot be satisfied, collecting the resource request of the tenant through a tenant priority manager and putting corresponding information into a resource request waiting queue; if the tenant needs to improve the resource request priority, an application is sent to a tenant priority manager, the tenant priority manager calls a radial basis function neural network to calculate the tenant resource request priority, and then a judgment document is generated to be transmitted to a system manager for auxiliary judgment. Compared with the traditional resource allocation strategy of the multi-tenant big data platform, the method and the system can effectively overcome the defects in the aspect of tenant priority processing of the multi-tenant big data platform.
Description
Technical Field
The invention relates to the field of big data platforms, in particular to a tenant priority management method and system of a multi-tenant big data platform.
Background
To date, big data platform technology has been applied to aspects of large and small enterprises. More and more enterprises have also built their own big data platforms for running big data applications inside the enterprise. Conventional large data platforms within an enterprise typically use private clusters of the department for each department to meet computing or storage requirements, which presents three challenges: firstly, the resource utilization rate of the whole cluster is poor; secondly, the enterprise manager has a heavy management load on the whole cluster; third, the sharing of data between departments can become more complex.
In light of the above three problems, multi-tenant (multi-tenant) technology for Hadoop big data platforms has been developed. Hadoop serves as an open source big data framework because of its strong load balancing mechanism and scalability, which occupies a very large market share. Although a multi-tenant system facing the Hadoop big data platform can integrate the advantages of Hadoop: the data is processed, such as by deploying big data processing component Hive, spark, storm, and the operations of the job are performed using a MapReduce distributed computing framework. Meanwhile, the system provides possibility for a plurality of tenants to share cluster resources: a complete multi-tenant system allows each tenant with corresponding rights to process the same piece of data uniformly, interactively and in real time across the entire cluster, or to run the tenant's personal jobs with authorized partial or whole computing resources (CPU, memory, etc.). The multi-tenant technology solves the problems of data sharing and isolation and resource sharing and isolation in a large data cluster to a certain extent, however, under the condition of limited cluster resources, how to ensure the computing capacity of important tenants, so that the large data platform can more efficiently complete tasks of each tenant, namely the priority management of the cluster resources, becomes a critical problem.
In general, in a multi-tenant system facing to a Hadoop big data platform, hadoop YARN is used for uniformly scheduling and managing resources of the whole cluster. YARN is implemented as a resource manager integrated within Hadoop. Depending on resource requirements and availability, YARN provides three scheduling strategies: a first-in first-out scheduling policy (FIFO Scheduler), a capacity scheduling policy (Capacity Scheduler), and a Fair scheduling policy (Fair Scheduler). The capacity scheduling strategy and the fairness scheduling strategy are widely applied to the multi-tenant system because the capacity scheduling strategy and the fairness scheduling strategy cannot cause starvation of other jobs due to long time-consuming jobs. In the multi-tenant system of the enterprise-level large data platform, the YARN is used to uniformly schedule and manage the resources of the whole cluster, but the problem of the priority of the jobs among different tenants cannot be solved (although under the fair scheduling policy, the tenants also need to wait for a period of time to smoothly allocate the required resources), and the tasks to be completed in an urgent need cannot be timely delivered due to insufficient resources. Second, because of the YARN's resource reservation mechanism, each tenant has the right to upload the respective jobs to YARN management, regardless of whether the current resources are available, and according to YARN's mechanism, accept and suspend these jobs. When the suspended operation is too much, the working efficiency of the whole cluster is extremely reduced, and even part of nodes are down when the working efficiency is severe. The drawbacks of using YARN as a total scheduler for cluster resources are becoming increasingly apparent.
In addition, in modern enterprises, because the development of complex products involves the collaboration of different personnel of different departments when completing a series of tasks, the resource requirements and rights between the departments (tenants) are greatly changed. The use of a fixed scheduling policy is not sufficient to cope with all emergencies, and therefore an efficient mechanism is needed to analyze the tenant resource requests and then prioritize the resource requests of individual tenants based on their current timeliness characteristics. In the above conventional resource request priority management, most of the analysis work is done by engineers based on experience obtained in previous projects. Such implicit knowledge-based management strategies can have a significant impact due to enterprise personnel variations.
For multi-tenant systems of one enterprise-level big data platform, how to maximize limited computing resource benefits is a problem that the cluster owners must solve. Currently, many scholars, research institutions and large IT companies have proposed methods that do not take into account the complex application environment of current enterprise-level big data platforms: it is difficult to trade off the problem of resource demand priority for jobs at various departments in an enterprise in real time. Dynamic allocation of resources cannot effectively solve the priority problem, and a fixed priority resource allocation strategy is particularly heavy. If the resource request priority of each tenant is determined by the engineer according to the previous project experience, not only the workload of the engineer is increased, but also the work of the whole project is greatly influenced by personnel mobilization. In summary, in today's enterprise-level Hadoop multi-tenant system, there is no suitable solution to solve the problem of job resource application priority of different tenants.
Disclosure of Invention
The invention aims to overcome the defects and shortcomings of the prior art and provides a tenant priority management method of a multi-tenant large data platform. On the basis of dynamically distributing cluster resources to multiple tenants, the method of the radial basis function neural network helps a system administrator manage the priorities of the tenants. Compared with the traditional multi-tenant large data platform resource allocation strategy, the method provided by the invention can effectively solve the tenant priority management problem in the large data platform.
Another object of the present invention is to provide a tenant priority management system for a multi-tenant big data platform.
The aim of the invention is achieved by the following technical scheme:
a tenant priority management method of a multi-tenant big data platform is based on a radial basis function neural network and comprises the following steps in sequence:
s1, initializing a tenant priority manager;
s2, the tenant sends a resource request to a cluster (big data platform), and the cluster allocates resources to the tenant;
s3, when the cluster resources are insufficient to meet the resource requests of the current tenant, starting a tenant priority manager, and adding the resource requests of the tenant into a resource application waiting queue;
s4, when the tenant needs to improve the resource request priority of the tenant, the tenant sends an application to the tenant priority manager through the system internal mailbox;
s5, the tenant priority manager calculates tenant resource application priority through a radial basis function neural network, generates a judgment document and gives the judgment document to a system manager;
s6, the system administrator decides whether to agree to promote the resource request priority of the tenant according to the judgment document;
s7, for the resource request with high priority, a system administrator is authorized to control the system administrator to preempt the resources of other tenants;
s8, after all the jobs of a certain tenant are completed, the cluster resource manager recovers the resources occupied by the cluster resource manager; when the tenant has a new task, the resource request will be reissued.
The step S1 specifically comprises the following steps:
s101, assigning a fixed tenant ID for a tenant in a system, wherein each tenant ID is a unique identifier of the tenant and cannot be changed and repeated;
s102, collecting relevant data for training a radial basis function neural network, wherein input vectors required to be provided are as follows: tenant ID, resource application amount, job type, job predicted completion time, resource application waiting time; the output vector is: 3. 2, 1; wherein 3 represents a low priority, 2 represents a medium priority, and 1 represents a high priority;
s103, training a radial basis function neural network by using the collected information until an output result reaches an expected value;
s104, deploying a tenant priority manager.
The tenant priority manager consists of a radial basis function neural network and a resource request waiting queue.
In step S2, the tenant sends a resource request to the cluster, where the request information includes: the current tenant ID, job type, resource application amount, job predicted completion time.
In step S2, if the load of the cluster is low, the tenant priority manager is not started under the condition that the resource request of each tenant job can be satisfied.
In step S2, the cluster allocates resources to the tenant by modifying a resource manager of the cluster: when the cluster allocates resources for a tenant, a resource queue is created in the resource manager for the tenant, and resources are allocated to the resource queue.
The resource manager of the cluster includes a resource manager in YARN.
In step S3, the request content of the resource request of the tenant includes the current tenant ID, the job type, the resource application amount, the job predicted completion time, and the resource application waiting time; wherein the resource application wait time is timed by the resource application wait queue when the request joins the queue.
In step S4, the tenant issues an application to the tenant priority manager through the system internal mailbox, and may have a remark document, where the content of the remark document is determined by the tenant.
In step S5, the priority manager receives an input vector from the tenant and obtains the input vector from the resource request waiting queue.
In step S5, the judging document is composed of the corresponding tenant job priority calculated by the radial basis function neural network and the tenant self-defined remark content.
In step S7, after the resources of the other tenants are preempted, the resource manager of the cluster temporarily retrieves the resources occupied by the resource queues, and does not delete the resource queues of the tenants, so that the jobs in the queues are temporarily suspended, and then the jobs of the advanced tenants are redistributed after the completion of the jobs of the advanced tenants or when the cluster has idle resources.
In step S8, the cluster resource manager will reclaim the resources occupied by the tenant, and the specific method is as follows:
by modifying the resource manager of the cluster, each time when a job is completed in the big data platform, the resource queue of the job is checked, and if no other job exists in the resource queue, the resources occupied by the resource queue are recovered.
The radial basis function neural network is specifically designed as follows:
input layer (first layer): to predict the priority of each resource request, the input vector is built up of five variables, respectively: tenant ID, tenant resource application amount, tenant job type, tenant job predicted completion time and tenant resource application waiting time;
hidden layer: the hidden layer performs space mapping transformation on the input layer vector; by linear combination of the nonlinear basis functions (Gaussian functions are adopted in the invention), RBFNN can obtain good performance in approximate nonlinear relation; adopting a Gaussian kernel function as a neuron kernel function, wherein the number of nodes is determined according to the need; the Gaussian kernel function is shown in a formula (1):
wherein x is an input layer vector, and n is the number of input layer units; sigma is a width vector initialized by a neural network, the smaller the width is, the narrower the shape of a corresponding hidden layer neuron action function is, and the smaller the response of information near the center of other neurons in the neuron is; x' is a center vector; k is the output value of the radial basis function;
output layer: the output nodes of the output layer are set as one, and the output results are three types of high priority, medium priority and low priority; the formula for calculating the output value of the neural network is shown as formula (2):
from the above equation, the parameter learning of RBFNN includes two parts: the first one involves parameters in the hidden layer, including a center vector x' and an initialization width vector σ, and the other involves a weight parameter ωi in the hidden layer; the position of any radial basis function in the input space is uniquely specified by its center and width; the output of RBFNN is a weighted sum of each activated hidden layer neural node.
Another object of the invention is achieved by the following technical scheme:
a tenant priority management system of a multi-tenant big data platform comprises a big data platform (i.e. a cluster), a tenant priority manager and a system manager; the tenant priority manager is used for generating a priority judgment document to assist a system administrator in making priority judgment, and collecting unallocated resource requests outside the cluster, so that the resource manager of the big data platform is focused on resource allocation and tenant inner resource scheduling, and the risk of cluster downtime is reduced to a certain extent; the tenant uses the computing resource of the big data platform; the tenant priority manager includes two components: radial basis function neural networks and resource request wait queues.
Compared with the prior art, the invention has the following advantages and beneficial effects:
(1) The method is applicable to an enterprise-level multi-tenant system based on a big data platform, and can solve the problem of tenant operation on resource allocation priority through the tenant priority manager when the cluster is heavy in load, unlike a common multi-tenant system.
(2) Because the tenant job resource application priority has complex nonlinear relation with tenant ID, resource application amount, job type, job predicted completion time, resource application waiting time and the like, the tenant priority manager constructed by the invention adopts Radial Basis Function Neural Network (RBFNN) to generate priority judgment documents. RBFNN can approach any nonlinear function, can process the regularity mapping which is difficult to analyze in a system, has good generalization capability and has quick learning convergence rate. The method can be well applied to the priority mode classification required by the invention.
(3) The manager is assisted by the tenant priority manager to judge the tenant resource request priority, so that the problem that the resource request priority of each tenant in the multi-tenant system of the current enterprise-level big data platform is difficult to judge due to personnel variation of the manager is solved.
(4) The invention adopts the priority judging method controlled by the administrator, thereby avoiding the problem of improper priority processing caused by automatic judgment of the system on sensitive tasks. The reliability of the system is ensured to a certain extent.
Drawings
Fig. 1 is a schematic structural diagram of a tenant priority management system of a multi-tenant big data platform according to the present invention.
Fig. 2 is a flowchart of a tenant priority management method of a multi-tenant big data platform according to the present invention.
Fig. 3 is a flowchart of an initializing tenant priority manager according to the present invention.
FIG. 4 is a schematic diagram of a radial basis function neural network according to the present invention.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but embodiments of the present invention are not limited thereto.
Example 1
A tenant priority management method of a multi-tenant big data platform based on a radial basis function neural network is realized by constructing a tenant priority manager on the big data platform. The function of the tenant priority manager is mainly to generate a priority judgment document to assist a system administrator in making priority judgment, and to collect unallocated resource requests outside the cluster, so that the resource manager of the big data platform is focused on resource allocation and tenant inner resource scheduling, and the cluster downtime risk is reduced to a certain extent. Wherein the tenant priority manager comprises two components: radial basis function neural networks and resource request wait queues.
Wherein, a radial basis function neural network model is shown in fig. 4, and the module assists in generating a priority judgment document through a complex nonlinear relation between a resource request and a priority of a computing tenant. The resource request waiting queue in the tenant priority manager is used for storing tenant resource requests which are not successfully allocated with resources when the big data platform is in use peak period. Among the non-prioritized resource requests, the resource requests in the resource request wait queue conform to a first-in-first-out (FIFO) scheduling order.
In the multi-tenant big data platform, after a tenant applies for required resources to the platform, a resource manager of the big data platform allocates the required resources to the tenant. The specific distribution method comprises the following steps: the resource manager of the big data platform will create a resource queue for the tenant and divide certain resources into the queue. This means that the tenant is exclusive to the resources it allocates, and after the tenant applies for the resources, jobs can be uploaded to the resource queue, and then the resource manager will schedule these jobs according to its scheduling method. Since the amount of resources of the big data platform is not infinite, when all jobs of a tenant are completed, the cluster resource manager will reclaim the resources occupied by the cluster resource manager. The specific judging method comprises the following steps: by modifying the resource manager of the big data platform, the invention checks the resource queue of the job whenever the job is completed in the big data platform, and if no other job exists in the resource queue, the invention retrieves the resources occupied by the resource queue.
As shown in fig. 2, a tenant priority management method of a multi-tenant big data platform based on a radial basis function neural network may be implemented as follows:
(1) The tenant priority manager is initialized. Initializing the tenant priority manager mainly comprises training a radial basis function neural network and establishing a connection between the tenant priority manager and a big data platform. Since the tenant priority manager is a functional middleware built on top of the big data platform, intimate data communication with the resource manager of the big data platform is required. The specific steps of initializing the tenant priority manager are as follows:
(1-1) first assigning a fixed tenant ID to each tenant within the multi-tenant big data platform. The tenant ID is a unique identity of a tenant, and cannot be changed and repeated.
(1-2) a system administrator collecting relevant data for training the radial basis function neural network. The input vectors that need to be provided are: tenant ID, resource application amount, job type, job predicted completion time, resource application waiting time. The output vector is: 3 (low priority), 2 (medium priority), 1 (high priority).
(1-3): the radial basis neural network is trained with the collected information. Until the output result reaches the expected value.
(1-4): the tenant priority manager is deployed. And establishing connection between the tenant priority manager and the resource manager of the big data platform by using an RPC protocol.
(2) And the tenant sends a resource request to the big data platform, and the big data platform allocates resources to the tenant. When the cluster load is lower, the cluster resources can meet the resource requests of all tenants, and the tenant priority management system is not started at this time. If the current cluster resource is insufficient to meet the resource request of the current tenant, a tenant priority manager is started to add the resource request of the tenant into a resource application waiting queue. The tenant resource request content placed in the resource application waiting queue should include: current tenant ID, job type, resource application amount, job predicted completion time, resource application waiting time. Wherein the resource request waiting time is counted by the resource application waiting queue when the request joins the queue
(3) Under the default condition, the tenant resource request in the resource request waiting queue is subjected to resource allocation in a FIFO (first in first out) mode, and if the current tenant job is urgently needed to be completed, a request for increasing the priority can be sent to the tenant priority manager so as to allocate the required resource as soon as possible.
When the tenant needs to improve the resource request priority of the tenant, an application is sent to the tenant priority manager through the system internal mailbox. In the application sent by the tenant to the tenant priority manager, remark information is submitted at the same time, and the content of the remark information is determined by the tenant. The tenant priority manager controls the radial basis function neural network to acquire corresponding resource request input vectors from the resource request waiting queue, and then calculates the resource request priority of the tenant. And the tenant priority manager generates tenant priority judgment documents according to the calculation results of the radial basis function neural network and the remark information submitted by the tenant, and gives the tenant priority judgment documents to a system administrator.
(4) And the system administrator decides the resource request priority of the tenant according to the judgment document. By adopting the method, the problem of mishandling of the priority on some sensitive tasks due to automatic judgment of the system is avoided. The reliability of the system is ensured to a certain extent.
(5) If the system administrator increases the tenant's resource request, for tenant resource requests that require priority allocation of resources, the system administrator will adjust the resource request to the queue head of the resource request wait queue for priority scheduling.
(6) For the urgent tenant resource request which needs to be completed urgently, a system administrator is authorized to control the resource manager of the big data platform to conduct preemptive allocation. When the resources need to be preempted, the system administrator controls the preempted tenant resources to be temporarily recovered by the resource manager of the big data platform to occupy the resources of the resource queue (the resource queue of the tenant is not deleted, the jobs in the queue are temporarily suspended), and the corresponding resources are allocated to the resource queue of the preempted tenant again after the operation of the advanced tenant is completed or when the cluster has free resources.
(7) When an unallocated resource request exists in the resource request waiting queue, the resource manager of the big data platform inquires whether the queue head resource request of the resource request waiting queue is met or not every time after the exclusive resource of a certain tenant is recovered.
As shown in fig. 1, a tenant priority management system of a multi-tenant big data platform includes a big data platform (i.e. cluster), a tenant priority manager, and a system administrator; the tenant priority manager is used for generating a priority judgment document to assist a system administrator in making priority judgment, and collecting unallocated resource requests outside the cluster, so that the resource manager of the big data platform is focused on resource allocation and tenant inner resource scheduling, and the risk of cluster downtime is reduced to a certain extent; the tenant uses the computing resource of the big data platform; the tenant priority manager includes two components: radial basis function neural networks and resource request wait queues.
Example 2
The method is applied to a multi-tenant system based on a Hadoop big data platform to realize the tenant priority management method of the multi-tenant big data platform based on a radial basis function neural network.
As shown in fig. 1, the working of the present invention is schematically illustrated. The tenant priority manager constructed by the invention is a functional component positioned between the Hadoop big data platform and a system administrator. The tenant priority manager has the function of collecting some resource request information of the big data platform. By default, the resource request waiting queue in the tenant priority manager will allocate resources for blocked resource requests according to the FIFO policy. When the tenant needs to give priority to its resource request so as to allocate resources in time, the tenant priority manager generates a judgment document for calculating priority information of the tenant-related job, and then gives the judgment document to a system administrator for auxiliary judgment.
As shown in fig. 2, a workflow diagram of the present invention is shown. When the Hadoop platform applies this method, the tenant priority manager is first initialized (fig. 3). And allocating unique IDs for all tenants in the Hadoop big data platform multi-tenant system. The radial basis function neural network in the data training tenant priority manager is then provided. The following table shows:
TABLE 1 input data
TABLE 2 output data
| 3 (Low priority) |
| 2 (Medium priority) |
| 1 (Low priority) |
The deployment of the system of the present invention is then completed in one (or more) servers for implementing resource request priority management for the entire cluster. In Hadoop-based multi-tenant systems, the collection and allocation of resource requests can be achieved by modifying the resource manager (a resource manager integrated in Hadoop) in the yacn. The tenant priority manager may establish a connection between the tenant priority manager and the big data platform via an RPC protocol.
The tenant priority manager is already deployed successfully. When the cluster load is normal, the resource requests of all tenants can be distributed, and at the moment, the tenant priority management system does not need to be started. When the cluster is overloaded, part of the resource requests cannot be responded in time, and at this time, the resource requests in the tenant priority manager wait for the queue to collect the unallocated resource requests. Under the default condition, resource requests in a resource request queue are distributed in sequence in a FIFO (first in first out) mode, and when important tenants in the cluster need to use resources in time, an application can be sent to a tenant priority manager. When the tenant priority manager receives a priority request of a certain tenant, the input vector of the tenant is collected and delivered to the radial basis function neural network to calculate the priority of the tenant. The collection of input vectors includes: the tenant ID, the resource request amount and the resource application waiting time of the tenant are collected from the resource request waiting queue, and the type of the tenant job and the expected completion time of the tenant job are collected from the priority request of the tenant. After the RBFNN calculates the priority of the resource request of the tenant, the tenant priority manager generates a priority judgment document according to the calculation result of the RBFNN and gives the priority judgment document to the manager, and the manager subsequently decides the preemption or the priority allocation of the resource request.
In addition, the invention modifies the resource manager of the big data platform, so that the resource allocation of the system to the tenant is not permanent resource allocation: when a resource manager in the cluster allocates resources to a tenant, a resource queue is created in the resource manager for the tenant, and the job of the tenant is uploaded to the resource queue for resource scheduling. Each time a job in the cluster is completed, the resource manager will issue a query to the resource queue of the tenant, and if the resource queue of the tenant is empty, the resource manager will delete the resource queue and reclaim the resources allocated thereto.
The above examples are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above examples, and any other changes, modifications, substitutions, combinations, and simplifications that do not depart from the spirit and principle of the present invention should be made in the equivalent manner, and the embodiments are included in the protection scope of the present invention.
Claims (6)
1. The tenant priority management method of the multi-tenant big data platform is characterized by comprising the following steps in sequence based on a radial basis function neural network:
s1, initializing a tenant priority manager;
s2, the tenant sends a resource request to the cluster, and the cluster allocates resources to the tenant;
s3, when the cluster resources are insufficient to meet the resource requests of the current tenant, starting a tenant priority manager, and adding the resource requests of the tenant into a resource application waiting queue;
s4, when the tenant needs to improve the resource request priority of the tenant, the tenant sends an application to the tenant priority manager through the system internal mailbox;
s5, the tenant priority manager calculates tenant resource application priority through a radial basis function neural network, generates a judgment document and gives the judgment document to a system manager;
s6, the system administrator decides whether to agree to promote the resource request priority of the tenant according to the judgment document;
s7, for the resource request with high priority, a system administrator is authorized to control the system administrator to preempt the resources of other tenants;
s8, after all the jobs of a certain tenant are completed, the cluster resource manager recovers the resources occupied by the cluster resource manager; when the tenant has a new task, re-issuing a resource request;
the tenant priority manager consists of a radial basis function neural network and a resource request waiting queue, wherein the resource request waiting queue collects unallocated resource requests outside a cluster;
the step S1 specifically comprises the following steps:
s101, assigning a fixed tenant ID for a tenant in a system, wherein each tenant ID is a unique identifier of the tenant and cannot be changed and repeated;
s102, collecting relevant data for training a radial basis function neural network, wherein input vectors required to be provided are as follows: tenant ID, resource application amount, job type, job predicted completion time, resource application waiting time; the output vector is: 3. 2, 1; wherein 3 represents a low priority, 2 represents a medium priority, and 1 represents a high priority;
s103, training a radial basis function neural network by using the collected information until an output result reaches an expected value;
s104, deploying a tenant priority manager;
the radial basis function neural network, namely RBFNN, is specifically designed as follows:
input layer: to predict the priority of each resource request, the input vector is built up of five variables, respectively: tenant ID, tenant resource application amount, tenant job type, tenant job predicted completion time and tenant resource application waiting time;
hidden layer: the hidden layer performs space mapping transformation on the input layer vector; adopting a Gaussian kernel function as a neuron kernel function, wherein the number of nodes is determined according to the need; the Gaussian kernel function is shown in a formula (1):
wherein x is an input layer vector, and n is the number of input layer units; sigma is a width vector initialized by a neural network, the smaller the width is, the narrower the shape of a corresponding hidden layer neuron action function is, and the smaller the response of information near the center of other neurons in the neuron is; x' is the corresponding center vector of the input layer vector; k is the output value of the radial basis function;
output layer: the output nodes of the output layer are set as one, and the output results are three types of high priority, medium priority and low priority; the formula for calculating the output value of the neural network is shown as formula (2):
wherein y is an implicit layer vector, and p is the number of implicit layer units; y' is the center vector corresponding to the hidden layer vector, σ is the initialization width vector, ω p Is a weight parameter in the hidden layer; the position of any radial basis function in the input space is uniquely specified by its center and width; the output of RBFNN is a weighted sum of each activated hidden layer neural node.
2. The tenant priority management method of the multi-tenant big data platform according to claim 1, wherein in step S2, the cluster allocates resources to the tenant by modifying a resource manager of the cluster: when the cluster allocates resources for a tenant, a resource queue is created for the tenant in a resource manager, and resources are allocated to the resource queue; the resource manager of the cluster includes a resource manager in YARN.
3. The tenant priority management method of the multi-tenant big data platform according to claim 1, wherein in step S5, the priority manager receives an input vector from a tenant and obtains the input vector from a resource request waiting queue; in step S5, the judging document is composed of the corresponding tenant job priority calculated by the radial basis function neural network and the tenant self-defined remark content.
4. The method for managing tenant priorities of the multi-tenant large data platform according to claim 1, wherein in step S7, after the resources of the other tenants are preempted, the resource manager of the cluster temporarily retrieves the resources occupied by the resource queues thereof, while the resource queues of the tenants are not deleted, the jobs in the queues are temporarily suspended, and the jobs of the higher tenants are reassigned after the completion of the jobs or when the cluster has free resources.
5. The method for managing tenant priorities of the multi-tenant big data platform according to claim 1, wherein in step S8, the cluster resource manager will reclaim resources occupied by tenants, and the specific method is as follows:
by modifying the resource manager of the cluster, each time when a job is completed in the big data platform, the resource queue of the job is checked, and if no other job exists in the resource queue, the resources occupied by the resource queue are recovered.
6. A tenant priority management system of a multi-tenant big data platform is characterized in that: the system comprises a big data platform, a tenant priority manager and a system administrator; the tenant priority manager is used for generating a priority judgment document to assist a system administrator in making priority judgment and collecting unallocated resource requests outside the cluster; the tenant uses the computing resource of the big data platform; the tenant priority manager includes two components: a radial basis function neural network and a resource request waiting queue;
the step of initializing the tenant priority manager comprises:
s101, assigning a fixed tenant ID for a tenant in a system, wherein each tenant ID is a unique identifier of the tenant and cannot be changed and repeated;
s102, collecting relevant data for training a radial basis function neural network, wherein input vectors required to be provided are as follows: tenant ID, resource application amount, job type, job predicted completion time, resource application waiting time; the output vector is: 3. 2, 1; wherein 3 represents a low priority, 2 represents a medium priority, and 1 represents a high priority;
s103, training a radial basis function neural network by using the collected information until an output result reaches an expected value;
s104, deploying a tenant priority manager;
the radial basis function neural network is specifically designed as follows:
input layer: to predict the priority of each resource request, the input vector is built up of five variables, respectively: tenant ID, tenant resource application amount, tenant job type, tenant job predicted completion time and tenant resource application waiting time;
hidden layer: the hidden layer performs space mapping transformation on the input layer vector; adopting a Gaussian kernel function as a neuron kernel function, wherein the number of nodes is determined according to the need; the Gaussian kernel function is shown in a formula (1):
wherein x is an input layer vector, and n is the number of input layer units; sigma is a width vector initialized by a neural network, the smaller the width is, the narrower the shape of a corresponding hidden layer neuron action function is, and the smaller the response of information near the center of other neurons in the neuron is; x' is the corresponding center vector of the input layer vector; k is the output value of the radial basis function;
output layer: the output nodes of the output layer are set as one, and the output results are three types of high priority, medium priority and low priority; the formula for calculating the output value of the neural network is shown as formula (2):
wherein y is an implicit layer vector, and p is the number of implicit layer units; y' is the corresponding center direction of the hidden layer vectorQuantity, σ is the initialized width vector, ω p Is a weight parameter in the hidden layer; the position of any radial basis function in the input space is uniquely specified by its center and width; the output of RBFNN is a weighted sum of each activated hidden layer neural node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811514328.XA CN109783225B (en) | 2018-12-12 | 2018-12-12 | Tenant priority management method and system of multi-tenant big data platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811514328.XA CN109783225B (en) | 2018-12-12 | 2018-12-12 | Tenant priority management method and system of multi-tenant big data platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109783225A CN109783225A (en) | 2019-05-21 |
| CN109783225B true CN109783225B (en) | 2023-09-08 |
Family
ID=66496707
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811514328.XA Active CN109783225B (en) | 2018-12-12 | 2018-12-12 | Tenant priority management method and system of multi-tenant big data platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109783225B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112286673A (en) * | 2019-07-22 | 2021-01-29 | 北京车和家信息技术有限公司 | Node resource allocation method and device |
| CN112445569B (en) * | 2019-09-02 | 2023-01-17 | 阿里巴巴集团控股有限公司 | Deployment method, device, electronic equipment and storage medium |
| CN110941681B (en) * | 2019-12-11 | 2021-02-23 | 南方电网数字电网研究院有限公司 | Multi-tenant data processing system, method and device of power system |
| CN115878310A (en) * | 2021-09-30 | 2023-03-31 | 中兴通讯股份有限公司 | Virtual resource information synchronization method, orchestrator, electronic device, and readable medium |
| CN115994019B (en) * | 2023-01-10 | 2023-06-06 | 杭州比智科技有限公司 | Strategy method and system for dynamic computation of multi-tenant resources based on big data cluster |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106502792A (en) * | 2016-10-20 | 2017-03-15 | 华南理工大学 | A kind of multi-tenant priority scheduling of resource method towards dissimilar load |
| CN106506670A (en) * | 2016-11-21 | 2017-03-15 | 北京永信至诚科技股份有限公司 | A kind of cloud platform virtual resource high speed dispatching method and system |
| WO2017140237A1 (en) * | 2016-02-19 | 2017-08-24 | Huawei Technologies Co., Ltd. | Method and system for multi-tenant resource distribution |
| CN108009023A (en) * | 2017-11-29 | 2018-05-08 | 武汉理工大学 | Method for scheduling task based on BP neural network time prediction in mixed cloud |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8782224B2 (en) * | 2011-12-29 | 2014-07-15 | Joyent, Inc. | Systems and methods for time-based dynamic allocation of resource management |
-
2018
- 2018-12-12 CN CN201811514328.XA patent/CN109783225B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017140237A1 (en) * | 2016-02-19 | 2017-08-24 | Huawei Technologies Co., Ltd. | Method and system for multi-tenant resource distribution |
| CN106502792A (en) * | 2016-10-20 | 2017-03-15 | 华南理工大学 | A kind of multi-tenant priority scheduling of resource method towards dissimilar load |
| CN106506670A (en) * | 2016-11-21 | 2017-03-15 | 北京永信至诚科技股份有限公司 | A kind of cloud platform virtual resource high speed dispatching method and system |
| CN108009023A (en) * | 2017-11-29 | 2018-05-08 | 武汉理工大学 | Method for scheduling task based on BP neural network time prediction in mixed cloud |
Non-Patent Citations (1)
| Title |
|---|
| 潘泉等著.《现代目标跟踪与信息融合》.国防工业出版社,2009,(第1版),第242页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109783225A (en) | 2019-05-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109783225B (en) | Tenant priority management method and system of multi-tenant big data platform | |
| US11243805B2 (en) | Job distribution within a grid environment using clusters of execution hosts | |
| CN104580396B (en) | A kind of method for scheduling task, node and system | |
| CN109582448B (en) | Criticality and timeliness oriented edge calculation task scheduling method | |
| CN110825520B (en) | Cluster extremely-fast elastic telescoping method for realizing efficient resource utilization | |
| CN108762896A (en) | One kind being based on Hadoop cluster tasks dispatching method and computer equipment | |
| US20200174844A1 (en) | System and method for resource partitioning in distributed computing | |
| CN107066319A (en) | A kind of multidimensional towards heterogeneous resource dispatches system | |
| CN104657221A (en) | Multi-queue peak-alternation scheduling model and multi-queue peak-alteration scheduling method based on task classification in cloud computing | |
| US20230012487A1 (en) | Machine learning workload orchestration in heterogeneous clusters | |
| CN105373426B (en) | A kind of car networking memory aware real time job dispatching method based on Hadoop | |
| CN104123182A (en) | Map Reduce task data-center-across scheduling system and method based on master-slave framework | |
| CN103491024A (en) | Job scheduling method and device for streaming data | |
| CN106095581B (en) | Network storage virtualization scheduling method under private cloud condition | |
| WO2024021489A1 (en) | Task scheduling method and apparatus, and kubernetes scheduler | |
| CN106201681B (en) | Method for scheduling task based on pre-release the Resources list under Hadoop platform | |
| CN107483364A (en) | A kind of scheduling of Hadoop Yarn network bandwidth resources, partition method and device | |
| CN113946431B (en) | Resource scheduling method, system, medium and computing device | |
| Janagoudar et al. | Multi-objective scheduling using logistic regression for openstack-based cloud | |
| CN113672391B (en) | Parallel computing task scheduling method and system based on Kubernetes | |
| Nivodhini et al. | Algorithms to improve scheduling techniques in IaaS cloud | |
| CN115562841A (en) | Cloud video service self-adaptive resource scheduling system and method | |
| CN111522637B (en) | Method for scheduling storm task based on cost effectiveness | |
| Nzanywayingoma et al. | Task scheduling and virtual resource optimising in Hadoop YARN-based cloud computing environment | |
| Wang et al. | A parallel tasks Scheduling heuristic in the Cloud with multiple attributes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |