CN112579265A - Task permission using method and device, storage medium, electronic equipment and big data platform - Google Patents
Task permission using method and device, storage medium, electronic equipment and big data platform Download PDFInfo
- Publication number
- CN112579265A CN112579265A CN201910945164.4A CN201910945164A CN112579265A CN 112579265 A CN112579265 A CN 112579265A CN 201910945164 A CN201910945164 A CN 201910945164A CN 112579265 A CN112579265 A CN 112579265A
- Authority
- CN
- China
- Prior art keywords
- account node
- task
- resource allocation
- account
- allocation request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000013468 resource allocation Methods 0.000 claims abstract description 101
- 230000007246 mechanism Effects 0.000 claims description 34
- 238000007726 management method Methods 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 12
- 238000010586 diagram Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 8
- 238000004364 calculation method Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Storage Device Security (AREA)
Abstract
The disclosure relates to a task permission using method and device, a storage medium, electronic equipment and a big data platform, wherein the method comprises the following steps: acquiring a resource allocation request of a first account node, wherein the resource allocation request is used for requesting allocation of system resources; and allocating the system resource to the first account node according to the resource allocation request, and setting authority information of the first account node on the allocated system resource, wherein the system resource is used for completing a task created by the first account node.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a task permission using method and apparatus, a storage medium, an electronic device, and a big data platform.
Background
The existing data calculation mode is difficult to meet the current calculation requirement, so that a large data calculation platform is produced at the same time. Since the big data platform is popular, Apache-Yarn is widely applied to common big data computing frameworks such as Hive, storm, spark, blink and the like as a resource management and task scheduling framework of a big data platform system.
Apache-Yarn manages and distributes the resources of the platform in a unified way in the form of task queues, and different task queues can be distributed to different users for use in a file configuration mode. However, if there are too many users, the manual configuration of the files may cause inconvenience and inefficiency in management. Or, management can be performed by introducing a security component, but the method can only allocate task rights of an existing task queue, and cannot complete operations such as creation and application of the task queue, and moreover, under the Hive computing framework, the method has a problem that task rights management is not effective.
Disclosure of Invention
The purpose of the present disclosure is to provide a task permission using method and apparatus, a storage medium, an electronic device, and a big data platform, so as to solve the above technical problems.
In order to achieve the above object, in a first aspect of the present disclosure, a task permission using method for a big data platform is provided, where the method is applied to a big data platform built by an Apache-yann resource management framework, and the method includes: acquiring a resource allocation request of a first account node, wherein the resource allocation request is used for requesting allocation of system resources; and allocating the system resource to the first account node according to the resource allocation request, and setting authority information of the first account node on the allocated system resource, wherein the system resource is used for completing a task created by the first account node.
Optionally, the obtaining a resource allocation request of the first account node includes: receiving the resource allocation request forwarded by a second account node, wherein the resource allocation request is forwarded after the second account node approves the resource allocation request to be approved from the first account node, and if the resource allocation request passes the approval, the second account node adds approval passing information to the resource allocation request after the approval; before the allocating the system resource for the first account node according to the resource allocation request, the method further includes: determining whether approval passing information exists in the resource allocation request from the second account node, if so, allowing the first account node to be allocated with the system resource, otherwise, forbidding the first account node to be allocated with the resource, and the authority level of the second account node is higher than that of the first account node.
Optionally, the method further comprises: acquiring a task query request of the first account node, wherein the task query request comprises account information of the first account node, and the task query request is used for querying state information of a task to be processed and/or a processed task of the big data platform submitted by the first account node; determining whether the first account node has the inquiry authority or not according to the authority information corresponding to the account information; and if the first account node has the query authority, returning a query result according to the task query request.
Optionally, the resource allocation request is further configured to request the big data platform to set a scheduling mechanism for the task submitted by the first account node, where the scheduling mechanism includes a fair scheduling mechanism and a first-in-first-out scheduling mechanism.
Optionally, the system resources include at least one of: CPU resources, memory resources, and task number resources.
In a second aspect of the present disclosure, a task permission using apparatus for a big data platform is provided, where the apparatus is applied to a big data platform built by an Apache-yann resource management framework, and the apparatus includes:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a resource allocation request of a first account node, and the resource allocation request is used for requesting allocation of system resources; and the allocation module is used for allocating the system resources to the first account node according to the resource allocation request and setting authority information of the first account node on the allocated system resources, wherein the system resources are used for completing tasks created by the first account node.
Optionally, the obtaining module includes: the receiving submodule is used for receiving the resource allocation request forwarded by a second account node, wherein the resource allocation request is sent after the second account node approves the resource allocation request to be approved from the first account node, and the second account node can add approval passing information in the approved resource allocation request under the condition of passing approval; the device further comprises: the determining module is used for determining whether approval passing information exists in the resource allocation request from the second account node, if so, the system resource is allowed to be allocated to the first account node, otherwise, the resource is forbidden to be allocated to the first account node, and the authority level of the second account node is higher than that of the first account node.
Optionally, the apparatus further comprises: the query module is used for acquiring a task query request of the first account node, wherein the task query request comprises account information of the first account node, and the task query request is used for querying state information of a task to be processed and/or a processed task of the big data platform submitted by the first account node; determining whether the first account node has the inquiry authority or not according to the authority information corresponding to the account information; and if the first account node has the query authority, returning a query result according to the task query request.
Optionally, the resource allocation request is further configured to request the big data platform to set a scheduling mechanism for the task submitted by the first account node, where the scheduling mechanism includes a fair scheduling mechanism and a first-in-first-out scheduling mechanism.
Optionally, the system resources include at least one of: CPU resources, memory resources, and task number resources.
In a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which program is characterized in that the program is executed by a processor to perform the steps of the method according to any one of the first aspect of the present disclosure.
A fourth aspect of the present disclosure provides an electronic device, including: a memory having a computer program stored thereon; a processor for executing the computer program in the memory to implement the steps of the method of any one of the first aspect of the present disclosure.
Through the technical scheme, the following technical effects can be at least achieved:
the task authority distribution of the big data platform constructed by the Apache-Yarn resource management framework can be completed, so that the application of an Apache-Yarn task queue, authority management and other operations are realized, and the problem of low management efficiency of the big data platform under the Apache-Yarn framework on task authority is solved.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows.
Drawings
The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure without limiting the disclosure. In the drawings:
FIG. 1 is a flow chart illustrating a method for task permission usage for large data platforms in accordance with an exemplary disclosed embodiment.
FIG. 2 is a flow diagram illustrating a task rights assignment process according to an exemplary disclosed embodiment.
FIG. 3 is a flow chart illustrating a method for task rights usage for large data platforms in accordance with an exemplary disclosed embodiment.
Fig. 4 is a flow diagram illustrating an authentication flow of task rights according to an exemplary disclosed embodiment.
FIG. 5 is a block diagram illustrating a task rights usage device for large data platforms in accordance with an exemplary disclosed embodiment.
FIG. 6 is a block diagram illustrating an electronic device according to an exemplary disclosed embodiment.
Detailed Description
The following detailed description of specific embodiments of the present disclosure is provided in connection with the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the present disclosure, are given by way of illustration and explanation only, not limitation.
FIG. 1 is a flowchart illustrating a task permission usage method for a big data platform, which is applied to a big data platform built by an Apache-Yarn resource management framework, according to an exemplary disclosed embodiment, and the method includes the following steps:
s11, acquiring a resource allocation request of the first account node, wherein the resource allocation request is used for requesting allocation of system resources.
The account node is a node where computing equipment accessed to the big data platform is located, and can be associated with the account in a login mode and be cancelled in a logout mode.
The resource allocation request of the first account node may be sent when the user logs in to the first account node, and before sending the resource allocation request, the first account node may further select different tenants, and the computing resources owned by the different tenants may be different.
Optionally, the resource allocation request of the first account node may be a resource allocation request for the first account node directly sent by the first account node, or may be a resource allocation request for the first account node forwarded by the second account node. Before the second account node forwards the resource allocation request of the first account node, the resource allocation request to be approved sent by the first account node needs to be received, and after the second account node approves the resource allocation request, the second account node sends the resource allocation request to the resource manager of the big data platform. Before allocating resources to the first account node according to the resource allocation request, it may be determined that approval pass information of a second account node exists in the resource allocation request, and the permission level of the second account node is higher than that of the first account node.
It should be noted that the distinction between the first account node and the second account node is to explain the authority level of the second account node and the first account node, and for two account nodes with authority dependency, the account node with higher authority level is the second account node, and the account node with lower authority level is the first account node.
For example, two existing account nodes are an account node a and an account node B in order from high to low according to the authority level, wherein when the account node a needs to request the platform to allocate the computing resource, the resource allocation request may be directly sent to the platform, but when the account node B needs to request the platform to allocate the computing resource, the resource allocation request needs to be sent to the account node a first, and after the account node a passes the approval, the account node a sends the resource allocation request with the approval information to the platform, so as to implement allocation of the resource. Alternatively, since account node B (the first account node) is subordinate to account node a (the second account node), when allocating system computing resources to account node B, it may be allocated from the computing resources of account node a.
Optionally, the resource allocation request is further configured to request the big data platform to set a scheduling mechanism for the task submitted by the first account node, where the scheduling mechanism includes a fair scheduling mechanism and a first-in-first-out scheduling mechanism.
Under the fair scheduling mechanism, all jobs can obtain equivalent resources evenly over time, when a single job runs, the single job can use the resources of the whole cluster, and when other jobs are submitted, the system can allocate idle resources to the new jobs, so that each job can obtain approximately the same amount of resources.
Under the first-in first-out scheduling mechanism, the system can preferentially ensure the completion of the first-submitted job, the first-submitted job can be relatively allocated with more resources, and the later-submitted job can be allocated with the idle resources after the first-submitted job is completed.
The scheduling mechanism may be specified by the big data platform or may be applied by the first account node and approved by the big data platform.
As shown in fig. 2, specifically, task authority may be assigned according to the following process: when a user logs in through a first account node, the user can check the tenant where the user is located and account information of the first account node (namely information of a role to which the user belongs); after logging in, the user can select a tenant for the first account node and can view information of a big data platform tool born owned by the tenant; a user puts forward a yarn component task queue application for a first account node; the system administrator approves the application as required; and if the application is passed, creating resources for the application and distributing corresponding authority.
S12, allocating resources for the first account node according to the resource allocation request, and setting authority information of the first account node for the allocated resources.
The system resource is a computing resource of a big data platform for completing the task created by the first account node, and may be any one or any combination of a CPU resource, a memory resource, and a task number resource.
After obtaining the allocated resources, the first account node may submit the computing task to the big data platform based on the amount of resources available.
By the technical scheme, task authority distribution of a big data platform constructed by the Apache-Yarn resource management framework can be completed, so that operations such as application of an Apache-Yarn task queue, authority management and the like are realized, and the problem of low management efficiency of the big data platform under the Apache-Yarn framework on task authority is solved.
FIG. 3 is a flowchart illustrating a task permission usage method for big data platforms, applied to big data platforms consisting of an Apache-Yarn resource management framework and a Hive computing framework, according to an exemplary disclosed embodiment, and as shown in FIG. 3, the method includes the following steps:
s31, acquiring a resource allocation request of the first account node, wherein the resource allocation request is used for requesting allocation of system resources.
The account node is a node where computing equipment accessed to the big data platform is located, and can be associated with the account in a login mode and be cancelled in a logout mode.
The resource allocation request of the first account node may be sent when the user logs in to the first account node, and before sending the resource allocation request, the first account node may further select different tenants, and the computing resources owned by the different tenants may be different.
Optionally, the resource allocation request of the first account node may be a resource allocation request for the first account node directly sent by the first account node, or may be a resource allocation request for the first account node forwarded by the second account node. Before the second account node forwards the resource allocation request of the first account node, the resource allocation request to be approved sent by the first account node needs to be received, and after the second account node approves the resource allocation request, the second account node sends the resource allocation request to the resource manager of the big data platform. Before allocating resources to the first account node according to the resource allocation request, it may be determined that approval pass information of a second account node exists in the resource allocation request, and the permission level of the second account node is higher than that of the first account node.
It should be noted that the distinction between the first account node and the second account node is to explain the authority level of the second account node and the first account node, and for two account nodes with authority dependency, the account node with higher authority level is the second account node, and the account node with lower authority level is the first account node.
For example, two existing account nodes are an account node a and an account node B in order from high to low according to the authority level, wherein when the account node a needs to request the platform to allocate the computing resource, the resource allocation request may be directly sent to the platform, but when the account node B needs to request the platform to allocate the computing resource, the resource allocation request needs to be sent to the account node a first, and after the account node a passes the approval, the account node a sends the resource allocation request with the approval information to the platform, so as to implement allocation of the resource. Alternatively, since account node B (the first account node) is subordinate to account node a (the second account node), when allocating system computing resources to account node B, it may be allocated from the computing resources of account node a.
Optionally, the resource allocation request is further configured to request the big data platform to set a scheduling mechanism for the task submitted by the first account node, where the scheduling mechanism includes a fair scheduling mechanism and a first-in-first-out scheduling mechanism.
Under the fair scheduling mechanism, all jobs can obtain equivalent resources evenly over time, when a single job runs, the single job can use the resources of the whole cluster, and when other jobs are submitted, the system can allocate idle resources to the new jobs, so that each job can obtain approximately the same amount of resources.
Under the first-in first-out scheduling mechanism, the system can preferentially ensure the completion of the first-submitted job, the first-submitted job can be relatively allocated with more resources, and the later-submitted job can be allocated with the idle resources after the first-submitted job is completed.
The scheduling mechanism may be specified by the big data platform or may be applied by the first account node and approved by the big data platform.
As shown in fig. 2, specifically, task authority may be assigned according to the following process: when a user logs in through a first account node, the user can check the tenant where the user is located and account information of the first account node (namely information of a role to which the user belongs); after logging in, the user can select a tenant for the first account node and can view information of a big data platform tool born owned by the tenant; a user puts forward a yarn component task queue application for a first account node; the system administrator approves the application as required; and if the application is passed, creating resources for the application and distributing corresponding authority.
S32, allocating resources for the first account node according to the resource allocation request, and setting authority information of the first account node for the allocated resources.
The system resource is a computing resource of a big data platform for completing the task created by the first account node, and may be any one or any combination of a CPU resource, a memory resource, and a task number resource. After obtaining the allocated resources, the first account node may submit the computing task to the big data platform based on the amount of resources available.
S33, acquiring a task query request of the first account node, wherein the task query request comprises account information of the first account node, and the task query request is used for querying state information of a task to be processed and/or a processed task of the big data platform submitted by the first account node.
The first account node may send a task query request to the big data platform, where an object of the task query request may be a completion state or a completion result of a computing task submitted to the big data platform by the first account node, or a completion state or a completion result of a computing task submitted to the big data platform by another account node.
If the task query request is in the Hive computing framework, the task query request can be intercepted by the Hive-hook and then sent to the security system of the big data platform, and the security system completes the authentication of the query request.
And S34, determining whether the first account node has the inquiry authority or not according to the authority information corresponding to the account information.
The query authority may be set according to a predetermined rule, for example, it may be set that the first account node may query status information of a task submitted by itself, or may query status information of a task submitted by an account node whose authority level is lower than that of itself.
And S35, if the first account node has the query authority, returning a query result according to the task query request.
The query result may be status information of the unprocessed task and/or the processed task, and may also include a processing result of the processed task, and the like.
Fig. 4 is a flowchart illustrating an authentication process of a possible task right, and as shown in fig. 4, the authentication process under the Hive computing framework is as follows: after a user submits a query request through a first account node, the Hive-hook intercepts the query request and sends the query request to a security system of a big data platform for authentication; the security system judges whether a first account node of the request sent by the user has the task queue authority, if so, the first account node is inquired and returns a result, and if not, the inquiry request is intercepted.
By the technical scheme, task authority distribution of a big data platform constructed by an Apache-Yarn resource management framework can be completed, so that operations such as application of an Apache-Yarn task queue and authority management are realized, the problem that the management efficiency of the big data platform under the Apache-Yarn framework on task authority is low is solved, and the problem that task authority management submitted under a Hive calculation framework is not effective when a security component takes effect is solved in a mode of authenticating a task query request.
Fig. 5 is a block diagram illustrating a task right utilization apparatus for a big data platform, which is applied to the big data platform built by the Apache-yann resource management framework, according to an exemplary disclosed embodiment, and as shown in fig. 5, the apparatus 500 includes an obtaining module 510 and an allocating module 520.
The obtaining module 510 is configured to obtain a resource allocation request of a first account node, where the resource allocation request is used to request allocation of a system resource.
The allocating module 520 is configured to allocate the system resource to the first account node according to the resource allocation request, and set authority information of the first account node on the allocated system resource, where the system resource is used to complete a task created by the first account node.
Optionally, the obtaining module includes: the receiving submodule is used for receiving the resource allocation request forwarded by a second account node, wherein the resource allocation request is sent after the second account node approves the resource allocation request to be approved from the first account node, and the second account node can add approval passing information in the approved resource allocation request under the condition of passing approval; the device further comprises: the determining module is used for determining whether approval passing information exists in the resource allocation request from the second account node, if so, the system resource is allowed to be allocated to the first account node, otherwise, the resource is forbidden to be allocated to the first account node, and the authority level of the second account node is higher than that of the first account node.
Optionally, the apparatus further comprises: the query module is used for acquiring a task query request of the first account node, wherein the task query request comprises account information of the first account node, and the task query request is used for querying state information of a task to be processed and/or a processed task of the big data platform submitted by the first account node; determining whether the first account node has the inquiry authority or not according to the authority information corresponding to the account information; and if the first account node has the query authority, returning a query result according to the task query request.
Optionally, the resource allocation request is further configured to request the big data platform to set a scheduling mechanism for the task submitted by the first account node, where the scheduling mechanism includes a fair scheduling mechanism and a first-in-first-out scheduling mechanism.
Optionally, the system resources include at least one of: CPU resources, memory resources, and task number resources.
The task permission using device comprises a processor and a memory, wherein the acquisition module, the distribution module, the determination module, the query module, the receiving submodule and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.
The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. The kernel can be set to be one or more than one, and corresponding functions are realized by adjusting kernel parameters so as to solve corresponding technical problems.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
An embodiment of the present invention provides a storage medium, on which a program is stored, which, when executed by a processor, implements the task permission using method.
The embodiment of the invention provides a processor, which is used for running a program, wherein the task permission using method is executed when the program runs.
The embodiment of the invention provides a big data platform which comprises the device in the embodiment.
The embodiment of the invention provides an electronic device 600, wherein the electronic device 600 comprises at least one processor 601, at least one memory 602 connected with the processor, and a bus 603; the processor 601 and the memory 602 complete communication with each other through the bus 603; the processor 601 is used to call the program instructions in the memory 602 to execute the task right usage method described above. The electronic device herein may be a server, a big data platform, etc.
The present application further provides a computer program product adapted to perform a program for initializing the following method steps when executed on a data processing device:
acquiring a resource allocation request of a first account node, wherein the resource allocation request is used for requesting allocation of system resources;
and allocating the system resource to the first account node according to the resource allocation request, and setting authority information of the first account node on the allocated system resource, wherein the system resource is used for completing a task created by the first account node.
Optionally, the obtaining a resource allocation request of the first account node includes:
receiving the resource allocation request forwarded by a second account node, wherein the resource allocation request is forwarded after the second account node approves the resource allocation request to be approved from the first account node, and if the resource allocation request passes the approval, the second account node adds approval passing information to the resource allocation request after the approval;
before the allocating the system resource for the first account node according to the resource allocation request, the method further includes:
determining whether approval passing information exists in the resource allocation request from the second account node, if so, allowing the first account node to be allocated with the system resource, otherwise, forbidding the first account node to be allocated with the resource, and the authority level of the second account node is higher than that of the first account node.
Optionally, the method further comprises:
acquiring a task query request of the first account node, wherein the task query request comprises account information of the first account node, and the task query request is used for querying state information of a task to be processed and/or a processed task of the big data platform submitted by the first account node;
determining whether the first account node has the inquiry authority or not according to the authority information corresponding to the account information;
and if the first account node has the query authority, returning a query result according to the task query request.
Optionally, the resource allocation request is further configured to request the big data platform to set a scheduling mechanism for the task submitted by the first account node, where the scheduling mechanism includes a fair scheduling mechanism and a first-in-first-out scheduling mechanism.
Optionally, the system resources include at least one of: CPU resources, memory resources, and task number resources.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a device includes one or more processors (CPUs), memory, and a bus. The device may also include input/output interfaces, network interfaces, and the like.
The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A task permission using method for a big data platform is applied to the big data platform built by an Apache-Yarn resource management framework, and the method comprises the following steps:
acquiring a resource allocation request of a first account node, wherein the resource allocation request is used for requesting allocation of system resources;
and allocating the system resource to the first account node according to the resource allocation request, and setting authority information of the first account node on the allocated system resource, wherein the system resource is used for completing a task created by the first account node.
2. The method of claim 1, wherein obtaining the resource allocation request of the first account node comprises:
receiving the resource allocation request forwarded by a second account node, wherein the resource allocation request is forwarded after the second account node approves the resource allocation request to be approved from the first account node, and if the resource allocation request passes the approval, the second account node adds approval passing information to the resource allocation request after the approval;
before the allocating the system resource for the first account node according to the resource allocation request, the method further includes:
determining whether approval passing information exists in the resource allocation request from the second account node, if so, allowing the first account node to be allocated with the system resource, otherwise, forbidding the first account node to be allocated with the resource, and the authority level of the second account node is higher than that of the first account node.
3. The method according to claim 1 or 2, characterized in that the method further comprises:
acquiring a task query request of the first account node, wherein the task query request comprises account information of the first account node, and the task query request is used for querying state information of a task to be processed and/or a processed task of the big data platform submitted by the first account node;
determining whether the first account node has the inquiry authority or not according to the authority information corresponding to the account information;
and if the first account node has the query authority, returning a query result according to the task query request.
4. The method according to claim 1 or 2, wherein the resource allocation request is further used for requesting the big data platform to set a scheduling mechanism for the task submitted to the first account node, and the scheduling mechanism comprises a fair scheduling mechanism and a first-in-first-out scheduling mechanism.
5. The method of claim 1 or 2, wherein the system resources comprise at least one of: CPU resources, memory resources, and task number resources.
6. A task permission use device for a big data platform, wherein the device is applied to the big data platform built by an Apache-Yarn resource management framework, and the device comprises:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a resource allocation request of a first account node, and the resource allocation request is used for requesting allocation of system resources;
and the allocation module is used for allocating the system resources to the first account node according to the resource allocation request and setting authority information of the first account node on the allocated system resources, wherein the system resources are used for completing tasks created by the first account node.
7. The apparatus of claim 6, wherein the obtaining module comprises:
the receiving submodule is used for receiving the resource allocation request forwarded by a second account node, wherein the resource allocation request is sent after the second account node approves the resource allocation request to be approved from the first account node, and the second account node can add approval passing information in the approved resource allocation request under the condition of passing approval;
the device further comprises:
the determining module is used for determining whether approval passing information exists in the resource allocation request from the second account node, if so, the system resource is allowed to be allocated to the first account node, otherwise, the resource is forbidden to be allocated to the first account node, and the authority level of the second account node is higher than that of the first account node.
8. The apparatus of claim 6 or 7, further comprising:
the query module is used for acquiring a task query request of the first account node, wherein the task query request comprises account information of the first account node, and the task query request is used for querying state information of a task to be processed and/or a processed task of the big data platform submitted by the first account node; determining whether the first account node has the inquiry authority or not according to the authority information corresponding to the account information; and if the first account node has the query authority, returning a query result according to the task query request.
9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 5.
10. An electronic device, comprising:
a memory having a computer program stored thereon;
a processor connected to the processor via a bus for executing the computer program in the memory to implement the steps of the method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910945164.4A CN112579265A (en) | 2019-09-30 | 2019-09-30 | Task permission using method and device, storage medium, electronic equipment and big data platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910945164.4A CN112579265A (en) | 2019-09-30 | 2019-09-30 | Task permission using method and device, storage medium, electronic equipment and big data platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112579265A true CN112579265A (en) | 2021-03-30 |
Family
ID=75117065
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910945164.4A Pending CN112579265A (en) | 2019-09-30 | 2019-09-30 | Task permission using method and device, storage medium, electronic equipment and big data platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112579265A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103365713A (en) * | 2012-04-01 | 2013-10-23 | 华为技术有限公司 | Resource dispatch and management method and device |
| CN107864211A (en) * | 2017-11-17 | 2018-03-30 | 中国联合网络通信集团有限公司 | Cluster resource dispatching method and system |
| CN108132775A (en) * | 2016-11-30 | 2018-06-08 | 新华三技术有限公司 | A kind of tenant manages system and method |
| CN108846635A (en) * | 2018-05-30 | 2018-11-20 | 郑州云海信息技术有限公司 | A kind of method and system of resource bid |
| US20180375784A1 (en) * | 2017-06-27 | 2018-12-27 | Microsoft Technology Licensing, Llc | Throttling queue for a request scheduling and processing system |
| CN109408196A (en) * | 2018-09-27 | 2019-03-01 | 中国科学院电子学研究所 | A kind of virtual management system, construction method and computer readable storage medium based on Xen |
-
2019
- 2019-09-30 CN CN201910945164.4A patent/CN112579265A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103365713A (en) * | 2012-04-01 | 2013-10-23 | 华为技术有限公司 | Resource dispatch and management method and device |
| CN108132775A (en) * | 2016-11-30 | 2018-06-08 | 新华三技术有限公司 | A kind of tenant manages system and method |
| US20180375784A1 (en) * | 2017-06-27 | 2018-12-27 | Microsoft Technology Licensing, Llc | Throttling queue for a request scheduling and processing system |
| CN107864211A (en) * | 2017-11-17 | 2018-03-30 | 中国联合网络通信集团有限公司 | Cluster resource dispatching method and system |
| CN108846635A (en) * | 2018-05-30 | 2018-11-20 | 郑州云海信息技术有限公司 | A kind of method and system of resource bid |
| CN109408196A (en) * | 2018-09-27 | 2019-03-01 | 中国科学院电子学研究所 | A kind of virtual management system, construction method and computer readable storage medium based on Xen |
Non-Patent Citations (1)
| Title |
|---|
| 何美斌，胡精英等: "基于Hadoop的大数据平台多租户管理策略研究", 《电脑编程技巧与维护》, 31 December 2017 (2017-12-31), pages 56 - 57 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112513813B (en) | Performing auxiliary functions in an on-demand network code execution system | |
| US10817331B2 (en) | Execution of auxiliary functions in an on-demand network code execution system | |
| US11575748B2 (en) | Data storage method and apparatus for combining different data distribution policies | |
| CN108399101B (en) | Method, device and system for scheduling resources | |
| US20180253244A1 (en) | Method and system for memory allocation in a disaggregated memory architecture | |
| US9444636B2 (en) | Message broadcasting in a clustered computing environment | |
| US9513947B2 (en) | Adaptive virtual machine request approver | |
| US11509693B2 (en) | Event-restricted credentials for resource allocation | |
| CN108021400B (en) | Data processing method and device, computer storage medium and equipment | |
| US20200104177A1 (en) | Resource allocation system, management device, method, and program | |
| Song et al. | A two-stage approach for task and resource management in multimedia cloud environment | |
| CN110941681B (en) | Multi-tenant data processing system, method and device of power system | |
| CN106533961B (en) | Flow control method and device | |
| US20130305245A1 (en) | Methods for managing work load bursts and devices thereof | |
| CN110971646A (en) | Cluster control device, system and method | |
| US10990519B2 (en) | Multi-tenant cloud elastic garbage collector | |
| CN112241319A (en) | Method, electronic device and computer program product for balancing load | |
| CN105357239A (en) | Method and device for providing service, and method and device for acquiring service | |
| CN112003931B (en) | Method and system for deploying scheduling controller and related components | |
| US10956228B2 (en) | Task management using a virtual node | |
| US10979359B1 (en) | Polling resource management system | |
| CN103841200A (en) | Method and device for controlling software licensing | |
| CN112579265A (en) | Task permission using method and device, storage medium, electronic equipment and big data platform | |
| US12095774B2 (en) | Control of usage of computing services based on dynamic grouping | |
| CN110929269B (en) | System authority management method, device, medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |