CN114168292A - Resource scheduling method, device, equipment and medium - Google Patents
Resource scheduling method, device, equipment and medium Download PDFInfo
- Publication number
- CN114168292A CN114168292A CN202111501983.3A CN202111501983A CN114168292A CN 114168292 A CN114168292 A CN 114168292A CN 202111501983 A CN202111501983 A CN 202111501983A CN 114168292 A CN114168292 A CN 114168292A
- Authority
- CN
- China
- Prior art keywords
- resource
- container
- job
- resources
- type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000013486 operation strategy Methods 0.000 claims abstract description 17
- 230000015654 memory Effects 0.000 claims description 55
- 238000005516 engineering process Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 5
- 238000007726 management method Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 241001481833 Coryphaena hippurus Species 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000002955 isolation Methods 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The present application relates to the field of big data technologies, and in particular, to a resource scheduling method, apparatus, device, and medium. When the type of the operation is a container operation type, converting the configured resource group into a physical resource amount of the container so as to enable the operation to be operated in the container of the limited resource; and when the type of the job is a non-container operation type, converting the configured resource group into the physical resource amount of each working node according to the number of the working nodes and the operation strategy so as to enable the job to operate on the nodes with limited resources. Therefore, the operation of the container operation type and the operation of the non-container operation type are reasonably packaged, and different resource distribution modes are provided according to the operation types, so that the container resources and the non-container resources are reasonably set, and the utilization rate of the resources is improved.
Description
Technical Field
The present application relates to the field of big data technologies, and in particular, to a resource scheduling method, apparatus, device, and medium.
Background
The main function of job scheduling is to examine whether the system can meet the resource requirement of the user job according to the information in the job control block, and to select some jobs from the external backup queue to call into the memory according to a certain algorithm, and to create processes and allocate necessary resources for them.
However, the existing resource scheduling can only provide concurrency control in a logic level, cannot achieve resource isolation in a physical level, is easy to cause disordered resource competition, and leads to low resource utilization rate.
Disclosure of Invention
The application provides a resource scheduling method, device, equipment and medium, which are used for solving the problem of low resource utilization rate.
In a first aspect, the present application provides a resource scheduling method, where the method is used for a job manager, and includes:
acquiring a job to be executed, and judging whether the type of the job is a container operation type;
if so, converting the configured resource group into the physical resource amount of the container so as to enable the operation to run in the container with limited resources;
if not, converting the configured resource group into the physical resource quantity of each working node according to the number of the working nodes and the operation strategy so as to enable the operation to be operated on the nodes with limited resources.
Optionally, the configured resource group includes a maximum resource amount of the CPU;
the converting the configured resource group into the physical resource amount of the container specifically includes:
and converting the maximum CPU resource amount into a container operation application resource amount.
Optionally, the converting the maximum resource amount of the CPU into the resource amount of the container job application specifically includes:
and compressing the maximum resource quantity of the CPU according to a preset proportion, and converting the maximum resource quantity of the CPU into a container operation application resource quantity.
Optionally, the configured resource group includes a maximum amount of resources in the memory, and the operation policy includes: CPU priority or memory priority;
if the operation strategy comprises CPU priority, the method for converting the configured resource group into the physical resource quantity of each working node according to the number of the working nodes and the operation strategy specifically comprises the following steps:
calculating the number of tasks operated by a single working node according to the maximum CPU resource amount, the number of working nodes and the CPU resource amount of the main node;
and calculating the CPU resource quantity of each task on each working node according to the task quantity operated by each working node, the number of the working nodes, the CPU resource quantity of a single task and the CPU resource quantity of the main node, wherein the memory of the single task is fixed to be 4G.
Optionally, if the operation policy includes memory priority, the converting the configured resource group into the physical resource amount of each working node according to the number of the working nodes and the operation policy specifically includes:
and calculating the memory resource amount of each working node according to the memory resource amount of a single task, the number of the working nodes and the memory resource amount of the main node, wherein the CPU of the single task is fixed to be 1 core.
Optionally, determining whether the type of the job is a container operation type specifically includes:
judging whether the type of the operation is a command line type, wherein the command line type comprises the following steps: python, perl, shell or java.
In a second aspect, the present application provides a resource scheduling method, where the method is used for a resource manager, and includes:
allocating resources to a container corresponding to the to-be-executed job according to the physical resource amount of the container, so that the job runs in the container of the limited resources, or allocating resources to a node corresponding to the to-be-executed job according to the physical resource amount of each working node, so that the job runs on the node of the limited resources;
the physical resource amount of the container is obtained by converting the configured resource group of the job when the job manager determines that the type of the job is a container operation type;
and the physical resource quantity of each working node is obtained by converting the configured resource group of the operation according to the number of the working nodes and the operation strategy when the operation manager determines that the type of the operation is a non-container operation type.
Optionally, the Kubernetes is configured to allocate execution resources to the container corresponding to the job according to the physical resource amount of the container;
and the yarn is used for distributing execution resources to the nodes corresponding to the jobs according to the physical resource quantity of each working node.
In a third aspect, the present application provides a resource scheduling apparatus, including:
the first acquisition module is used for acquiring the operation to be executed;
the judging module is used for judging whether the type of the operation is a container operation type;
the second acquisition module is used for converting the configured resource group into the physical resource quantity of the container when the type of the operation is the container operation type, so that the operation can be operated in the container with limited resources; and if the configured resource group is converted into the physical resource quantity of each working node according to the number of the working nodes and the operation strategy, so that the operation is operated on the nodes with limited resources.
In a fourth aspect, the present application provides an electronic device comprising: a memory and a processor;
the memory is used for storing instructions; the processor is configured to call instructions in the memory to perform the resource scheduling method in any one of the possible designs of the first aspect and the first aspect or the resource scheduling method in any one of the possible designs of the second aspect and the second aspect.
In a fifth aspect, the present application provides a computer-readable storage medium, where computer instructions are stored in the computer-readable storage medium, and when at least one processor of an electronic device executes the computer instructions, the electronic device executes a resource scheduling method in any one of the possible designs of the first aspect and the first aspect or a resource scheduling method in any one of the possible designs of the second aspect and the second aspect.
In a sixth aspect, the present application provides a computer program product comprising computer instructions that, when executed by at least one processor of an electronic device, cause the electronic device to perform the method for scheduling resources in any one of the possible designs of the first aspect and the first aspect or the method for scheduling resources in any one of the possible designs of the second aspect and the second aspect.
The resource scheduling method provided by the application obtains the operation to be executed, judges whether the type of the operation is a container operation type, converts the configured resource group into the physical resource quantity of the container if the type of the operation is the container operation type, so that the operation operates in the container of the limited resource, and converts the configured resource group into the physical resource quantity of each working node according to the number of the working nodes and the operation strategy if the type of the operation is not the container operation type, so that the operation operates on the node of the limited resource. Therefore, the operation of the container operation type and the operation of the non-container operation type are reasonably packaged, and different resource distribution modes are provided according to the operation types, so that the container resources and the non-container resources are reasonably set, and the utilization rate of the resources is improved. And the operation of the container operation type can be conveniently operated in the container environment, and the operation of the non-container operation type can be conveniently operated on each node.
Drawings
In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic view of a scenario of a resource scheduling method according to an embodiment of the present application;
fig. 2 is a flowchart of a resource scheduling method according to an embodiment of the present application;
fig. 3 is a flowchart of a resource scheduling method according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a resource scheduling apparatus according to an embodiment of the present application;
fig. 5 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application.
Detailed Description
To make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be clearly and completely described below with reference to the drawings in the present application, and it is obvious that the described embodiments are some, but not all embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The traditional resource scheduling method comprises the steps of carrying out resource scheduling by a Control-M-based scheduling system, a Dolphin Scheduler-based scheduling system or an Airflow-based scheduling system.
The Control-M-based scheduling system realizes the separation of three functions of management, scheduling and job execution through a three-layer architecture. The three-layer architecture is respectively Control-M/EM, Control-M/Server and Control-M/Agent. The Control-M/EM is a management module and is responsible for uploading operation, monitoring operation and the like; the Control-M/Server is the core of the whole scheduling system and is responsible for instantiation of a job flow, scheduling of jobs, resource allocation, job operation management and the like; the Control-M/Agent is a job running node and is responsible for running the job issued by the Control-M/Server, saving the job running state and synchronizing the job running state to the Control-M/Server. Control-M provides resource management at the logical level and controls according to job concurrency, does not have a resource scheduling method at the physical resource level, and cannot directly interface with kubernets and Hadoop platforms.
The Dolphin Scheduler-based scheduling system is used for providing a special spark plug-in for spark engine type jobs, and task submission is carried out through spark-submblimt at the bottom layer. In the specific resource management control, the resource parameters of spark are completely opened to the user, and the resource parameters include, for example, the number of Driver cores, the number of Driver memories, the number of executors, the number of memories used by a single Executor, and the number of CPU cores. However, all resource parameters of spark require the user to fill in manually, and no container type plug-in is provided, nor is there a resource control policy for the container.
The Airflow-based scheduling system provides Spark Submit Operator and kubernets Pod Operator plug-ins for container-type and Spark-type jobs, respectively. For Spark Submit Operator, the user is required to set the resource parameters, and for Kubernets Pod Operator, the parameters of part of Kubernets Yaml file are opened to the user for setting, and parameter control of CPU and memory resources is not provided.
Therefore, the traditional resource scheduling method cannot achieve resource isolation on a physical layer, and disordered resource competition is easily caused, so that the utilization rate of resources is low.
In order to solve the above problems, the present application provides a resource scheduling method, which obtains a job to be executed, determines whether the type of the job is a container operation type, if so, converts a configured resource group into a physical resource amount of a container to enable the job to operate in the container of a limited resource, and if not, converts the configured resource group into a physical resource amount of each working node according to the number of the working nodes and an operation policy to enable the job to operate on the node of the limited resource. Therefore, the operation of the container operation type and the operation of the non-container operation type are reasonably packaged, and different resource distribution modes are provided according to the operation types, so that the container resources and the non-container resources are reasonably set, and the utilization rate of the resources is improved. And the operation of the container operation type can be conveniently operated in the container environment, and the operation of the non-container operation type can be conveniently operated on each node.
The technical solution of the present application will be described in detail below with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.
Fig. 1 is a scene diagram illustrating a resource scheduling method according to an embodiment of the present application. The job manager 101 acquires a job to be executed uploaded by the client 102, and determines whether the type of the job is a container operation type. When the type of the job to be executed is a container operation type, acquiring a resource group configured by a user through a client 102, converting the configured resource group into a physical resource amount of a container, and sending the physical resource amount of the container to a resource manager 103, where the resource manager 103 allocates resources for a container corresponding to the job according to the physical resource amount of the container, so that the job operates in the container of the limited resources. When the type of the job to be executed is a non-container operation type, the job manager 101 obtains a resource group configured by a user through the client 102, converts the configured resource group into a physical resource amount of each working node according to the number of the working nodes and the operation policy, and sends the physical resource amount of each node to the resource manager 103, and the resource manager 103 allocates resources to each node according to the node corresponding to the job and the physical resource amount of each node, so that the job runs on the node of the limited resources.
Fig. 2 shows a flowchart of a resource scheduling method according to an embodiment of the present application. On the basis of the embodiment shown in fig. 1, as shown in fig. 2, with the job manager as the execution subject, the method of this embodiment may include the following steps:
s101, acquiring a job to be executed, and judging whether the type of the job is a container operation type.
A job is a logical unit defined by a user to accomplish certain work. The jobs to be executed may include one or more jobs, and the plurality of jobs may include a group of jobs having similar functions, similar frequencies, or having a dependency relationship.
The job manager determines whether the type of the job is a container run type after acquiring the job to be executed. The container run type job may be, for example, a command type job, which may include, for example, python, perl, shell, executable java, or other executable commands. A non-container run type job may be, for example, a run engine dependent spark type job.
After determining whether the type of the job is the container operation type, if the type of the job is the container operation type, step S102 is performed, and if the type of the job is the non-container operation type, step S103 is performed.
And S102, converting the configured resource group into the physical resource amount of the container so as to enable the operation to be operated in the container with limited resources.
And when the type of the operation is the container operation type, the operation manager acquires the resource group configured by the user from the client and converts the configured resource group into the physical resource amount of the container. The configured resource group can comprise the maximum resource amount of the operation, and the user only needs to set the maximum resource amount of the operation without paying attention to resource parameters and the like, so that the experience of the user is improved. The maximum amount of resources can be characterized by the DCU, which is used to abstract the units of container resources. The maximum amount of resources may be, for example, 9 DCUs, with 1DCU representing 1core 4G. The maximum resource amount includes the maximum resource amount of the CPU and the maximum resource amount of the memory. The physical resource amount of the container may include a container job application resource amount, a maximum memory resource, and the like.
After acquiring the maximum CPU resource amount of the job set by the user, the job manager converts the maximum CPU resource class into a container job application resource amount, so that the job runs in a container with limited resources, and the container job application resource amount can be understood as a running resource required for running the job in the container.
Due to the fact that
For the jobs operated by the containers, each job is operated in an independent pod, the pod is composed of one or more containers, that is, the pod is a set of a group of containers, and the setting of the physical resources of a single job is mainly to set the ratio of the limit and the request of the used CPU and the ratio of the limit and the request of the memory respectively. The request refers to the amount of resources necessary to satisfy the scheduling algorithm of the resource manager, i.e., the amount of container job resources, the resource manager may include kubernets, and the limit refers to the amount of resources necessary to satisfy the scheduling algorithm of kubernets. The Request is the upper limit of the resource used by a single job, i.e., the maximum amount of resource. The applicant finds that the eviction of the pod can happen when the resource of a Kubernetes single node cannot meet the requirement of creating the pod, and preferentially evicts the pod with inconsistent ratio of limit and request of the memory. Therefore, the maximum resource amount of the CPU in the maximum resource amount can be converted, and the maximum resource amount of the memory is not converted.
As an implementation manner, the maximum amount of the CPU resource may be compressed according to a preset ratio, so as to convert the maximum amount of the CPU resource into an amount of the container job application resource. For example, for shell operation, when the maximum resource amount set by the user is 9DCU, the preset ratio may be 0.5, and the resource amount of the non-container operation obtained after the maximum resource amount of the CPU is converted is 4.5 DCU.
S103, converting the configured resource group into physical resource amount of each working node according to the number of the working nodes and the operation strategy so as to enable the operation to be operated on the nodes with limited resources.
And when the type of the job is a non-container operation type, the job manager acquires the resource group configured by the user from the client, and converts the resource group into the physical resource amount of each working node according to the number of the working nodes and the operation strategy. The configured resource group can include a maximum amount of resources, which can be characterized by the CU, the CU to abstract units of non-container resources, a number of working nodes, and an operating policy. The maximum amount of resources may be, for example, 3CU, with 1CU representing 1core 4G. The maximum resource amount includes the maximum resource amount of the CPU and the maximum resource amount of the memory. The number of working nodes refers to the number of nodes which can execute the operation in the cluster managed by the resource manager, and the number of the working nodes does not exceed the number of the working nodes in the cluster. The working mode comprises CPU priority or memory priority, when the CPU priority is given, the number of tasks (executors) which can be started by each working node is (N-1)/the number of the working nodes, wherein N represents the maximum resource quantity of the CPU set by a user, the number of the tasks represents the number of the tasks which can be simultaneously operated by each node, but the memory value of each task is fixed to be 4G, when the memory priority is given, each working node starts one task, and the memory used by the task is ((N-1) multiplied by 4)/the number of the working nodes. Generally, a CPU-priority mode may be adopted for CPU-intensive spark jobs, a memory-priority mode may be adopted for memory-intensive spark jobs, and the maximum resource amount set by the user is at least 3 CUs because the master node occupies at least 1 CU.
And converting the maximum resource quantity in the configured resource group into the physical resource quantity of each working node according to the number of the working nodes in the configured resource group and the operation strategy. The physical resource amount comprises a main node resource and a working node resource, and for spark type jobs (spark on horn), the physical resource amount mainly comprises two parts: one part is a master node resource, i.e., a spare driver resource, and the other part is a work node resource, i.e., a spare worker resource, and the work node resource may include a CPU resource and a memory resource.
The jobs of the non-container operation type are, for example, jobs of which the operation engine depends on spark, and physical resources of the jobs include a master node resource (spark drive) and a worker node resource (spark worker), and the maximum resource amount N is the master node resource amount + the worker node resource amount. The maximum resource amount includes the maximum resource amount of the CPU and the maximum resource amount of the memory, the maximum resource amount of the CPU is the master node CPU resource amount + the work node CPU resource amount + the number of the work nodes, and the maximum resource amount of the memory is the master node memory resource amount + the number of the work nodes. The master node resources can be fixed to 1CU (1core 4G), the number of the working nodes does not exceed the number of the working nodes in the horn cluster, and the master node resources can be set by a user.
As an implementation manner, if the working manner includes CPU priority, the maximum CPU resource amount is N CU, the number of tasks that each work node can start may be (N-1)/the number of work nodes, for example, N is 9, the number of work nodes is 2, and the number of tasks that each work node can start is 2, then the CPU resource of a single node is calculated to be 8G according to the maximum CPU resource amount, the number of work nodes, and the master node CPU resource amount, and since a single node can start two tasks, the CPU resource amount of a single task started on a single node is 4G, which may specifically refer to table 1. When a job to be executed includes a plurality of jobs, a task may be understood as a job, and when a job to be executed includes one job, a task may be understood as a part of the job.
As an implementation manner, if the working manner includes memory priority, the memory resource amount of a single node is calculated according to the memory resource amount of a single task, the number of working nodes, and the memory resource amount of the main node. For example, the maximum memory resource amount is 9CU, the master node CPU resource amount is 1CU ═ 4G, and the number of working nodes is 4, the memory resource amount of a single task is ((9-1) × 4)/4 ═ 8, as shown in specific reference table 1.
TABLE 1
The resource scheduling method provided by the application packages the operation of the container operation type and the operation of the non-container operation type reasonably, and provides different resource distribution modes according to the operation types, so that the container resources and the non-container resources are set reasonably, and the utilization rate of the resources is improved. And the operation of the container operation type can be conveniently operated in the container environment, and the operation of the non-container operation type can be conveniently operated on each node.
Fig. 3 shows a flowchart of a resource scheduling method according to an embodiment of the present application. As shown in fig. 3, with the resource manager as the execution subject, the method of this embodiment may include the following steps:
s201, receiving the physical resource amount of the container or the physical resource amount of each working node sent by the job manager;
the physical resource quantity of the container is obtained by converting the configured resource group of the operation when the operation manager determines that the type of the operation is the container operation type;
and the physical resource quantity of each working node is obtained by converting the configured resource group of the job according to the number of the working nodes and the operation strategy when the job manager determines that the type of the job is a non-container operation type.
S202, distributing resources for the jobs to be executed according to the physical resource amount of the container or distributing resources for the jobs to be executed according to the physical resource amount of each working node.
When the job manager sends the physical resource amount of the container, the resource manager allocates resources for the container according to the physical resource amount of the container, so that the job runs in the container of the limited resources. When the job manager transmits the physical resource amount of each worker node, the resource manager allocates resources to each worker node according to the physical resource amount of each worker node, so that the job runs on each node that defines the resources.
As one implementation, resource managers include kubernets and yarn. Kubernetes is used for allocating execution resources to a container corresponding to a job according to the physical resource amount of the container when the physical resource amount of the container is sent by the job manager, and yarn is used for allocating resources to each working node according to the physical resource amount of each working node when the physical resource amount of each node is sent by the job manager, so that the job runs on the node with limited resources.
According to the resource scheduling method, the resource manager allocates resources according to the physical resource amount of the container or the physical resource amount of each node, container resources and non-container resources are reasonably set, and the utilization rate of the resources is improved. And the container type operation can be conveniently operated in the container environment, and the non-container type operation can be conveniently operated in the corresponding node.
Fig. 4 is a schematic structural diagram of a resource scheduling apparatus according to an embodiment of the present application, and as shown in fig. 4, a resource scheduling apparatus 10 according to this embodiment is used to implement operations corresponding to a job manager in any of the method embodiments described above, where the resource scheduling apparatus 10 according to this embodiment includes:
a first obtaining module 11, configured to obtain a job to be executed;
a judging module 12, configured to judge whether the type of the job is a container operation type;
a second obtaining module 13, configured to, when the type of the job is a container operation type, convert the configured resource group into a physical resource amount of the container, so that the job operates in the container of the limited resource; and if the configured resource group is converted into the physical resource quantity of each working node according to the number of the working nodes and the operation strategy, so that the operation is operated on the nodes with limited resources.
The resource scheduling apparatus 10 provided in the embodiment of the present application may implement the method embodiment, and for specific implementation principles and technical effects, reference may be made to the method embodiment, which is not described herein again.
Fig. 5 shows a hardware structure diagram of an electronic device according to an embodiment of the present application. As shown in fig. 5, the electronic device 30 is configured to implement the operations corresponding to the electronic device in any of the method embodiments described above, where the electronic device 30 of this embodiment may include: memory 31, processor 32 and communication interface 33.
A memory 31 for storing computer instructions. The Memory 31 may include a Random Access Memory (RAM), a Non-Volatile Memory (NVM), at least one disk Memory, a usb disk, a removable hard disk, a read-only Memory, a magnetic disk or an optical disk.
A processor 32 for executing the computer instructions stored in the memory to implement the resource scheduling method in the above embodiments. Reference may be made in particular to the description relating to the method embodiments described above. The Processor 32 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present invention may be embodied directly in a hardware processor, or in a combination of the hardware and software modules within the processor.
Alternatively, the memory 31 may be separate or integrated with the processor 32.
A communication interface 33 may be connected to the processor 32. Processor 32 may control communication interface 33 to perform the functions of receiving and transmitting information signals.
The electronic device provided in this embodiment may be configured to execute the resource scheduling method, and the implementation manner and the technical effect thereof are similar, and details are not described herein again.
The present application also provides a computer readable storage medium, in which computer instructions are stored, and the computer instructions are executed by a processor to implement the methods provided by the above-mentioned various embodiments.
The present application also provides a computer program product comprising computer instructions stored in a computer readable storage medium. The computer instructions may be read by at least one processor of the device from a computer-readable storage medium, and execution of the computer instructions by the at least one processor causes the device to perform the methods provided by the various embodiments described above.
The embodiment of the present application further provides a chip, which includes a memory and a processor, where the memory is used to store computer instructions, and the processor is used to call and execute the computer instructions from the memory, so that a device in which the chip is installed executes the method described in the above various possible embodiments.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: it is also possible to modify the solutions described in the previous embodiments or to substitute some or all of them with equivalents. And the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.
Claims (11)
1. A resource scheduling method, for a job manager, comprising:
acquiring a job to be executed, and judging whether the type of the job is a container operation type;
if so, converting the configured resource group into the physical resource amount of the container so as to enable the operation to run in the container with limited resources;
if not, converting the configured resource group into the physical resource quantity of each working node according to the number of the working nodes and the operation strategy so as to enable the operation to be operated on the nodes with limited resources.
2. The method of claim 1, wherein the configured set of resources comprises a CPU maximum amount of resources;
the converting the configured resource group into the physical resource amount of the container specifically includes:
and converting the maximum CPU resource amount into a container operation application resource amount.
3. The method according to claim 2, wherein the converting the maximum amount of resources of the CPU into the amount of resources of the container job application specifically comprises:
and compressing the maximum resource quantity of the CPU according to a preset proportion, and converting the maximum resource quantity of the CPU into a container operation application resource quantity.
4. The method of claim 2, wherein the configured set of resources comprises a maximum amount of resources in memory, and wherein the operating policy comprises: CPU priority or memory priority;
if the operation strategy comprises CPU priority, the method for converting the configured resource group into the physical resource quantity of each working node according to the number of the working nodes and the operation strategy specifically comprises the following steps:
calculating the number of tasks operated by a single working node according to the maximum CPU resource amount, the number of working nodes and the CPU resource amount of the main node;
and calculating the CPU resource quantity of each task on each working node according to the task quantity operated by each working node, the number of the working nodes, the CPU resource quantity of a single task and the CPU resource quantity of the main node, wherein the memory of the single task is fixed to be 4G.
5. The method according to claim 4, wherein if the operation policy includes memory priority, the converting the configured resource group into the physical resource amount of each working node according to the number of the working nodes and the operation policy specifically includes:
and calculating the memory resource amount of each working node according to the memory resource amount of a single task, the number of the working nodes and the memory resource amount of the main node, wherein the CPU of the single task is fixed to be 1 core.
6. The method according to claim 1, wherein determining whether the type of the job is a container operation type specifically includes:
judging whether the type of the operation is a command line type, wherein the command line type comprises the following steps: python, perl, shell or java.
7. A resource scheduling method, used for a resource manager, includes:
receiving the physical resource amount of the container or the physical resource amount of each working node sent by the job manager;
allocating resources to a container corresponding to the to-be-executed job according to the physical resource amount of the container, so that the job runs in the container of the limited resources, or allocating resources to a node corresponding to the to-be-executed job according to the physical resource amount of each working node, so that the job runs on the node of the limited resources;
the physical resource amount of the container is obtained by converting the configured resource group of the job when the job manager determines that the type of the job is a container operation type;
and the physical resource quantity of each working node is obtained by converting the configured resource group of the operation according to the number of the working nodes and the operation strategy when the operation manager determines that the type of the operation is a non-container operation type.
8. The method of claim 7, wherein the resource manager comprises kubernets and yarn;
the Kubernetes is used for allocating execution resources to the container corresponding to the operation according to the physical resource amount of the container;
and the yarn is used for distributing execution resources to the nodes corresponding to the jobs according to the physical resource quantity of each working node.
9. An apparatus for scheduling resources, the apparatus comprising:
the first acquisition module is used for acquiring the operation to be executed;
the judging module is used for judging whether the type of the operation is a container operation type;
the second acquisition module is used for converting the configured resource group into the physical resource quantity of the container when the type of the operation is the container operation type, so that the operation can be operated in the container with limited resources; and if the configured resource group is converted into the physical resource quantity of each working node according to the number of the working nodes and the operation strategy, so that the operation is operated on the nodes with limited resources.
10. An electronic device, comprising: a processor, and a memory communicatively coupled to the processor;
the memory stores computer-executable instructions;
the processor executes computer-executable instructions stored by the memory to implement the resource scheduling method of any one of claims 1 to 7 or the resource scheduling method of claim 8 or 9.
11. A computer-readable storage medium, having stored thereon computer instructions, which, when executed by a processor, are adapted to implement the resource scheduling method of any one of claims 1 to 7 or the resource scheduling method of claim 8 or 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111501983.3A CN114168292A (en) | 2021-12-09 | 2021-12-09 | Resource scheduling method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111501983.3A CN114168292A (en) | 2021-12-09 | 2021-12-09 | Resource scheduling method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114168292A true CN114168292A (en) | 2022-03-11 |
Family
ID=80485209
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111501983.3A Pending CN114168292A (en) | 2021-12-09 | 2021-12-09 | Resource scheduling method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114168292A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105589751A (en) * | 2015-11-27 | 2016-05-18 | 杭州华三通信技术有限公司 | Physical resource scheduling method and device |
| CN109471727A (en) * | 2018-10-29 | 2019-03-15 | 北京金山云网络技术有限公司 | A kind of task processing method, apparatus and system |
| CN110278279A (en) * | 2019-06-27 | 2019-09-24 | 苏宁消费金融有限公司 | A kind of big data of dynamic resource scheduling mechanism dispatches development platform and method offline |
| CN112506650A (en) * | 2020-11-27 | 2021-03-16 | 中国建设银行股份有限公司 | Resource allocation method, system, computer device and storage medium |
| WO2021063339A1 (en) * | 2019-09-30 | 2021-04-08 | 星环信息科技(上海)股份有限公司 | Cluster resource scheduling method, apparatus, device and storage medium |
| CN113419839A (en) * | 2021-07-20 | 2021-09-21 | 北京字节跳动网络技术有限公司 | Resource scheduling method and device for multi-type jobs, electronic equipment and storage medium |
| CN113448716A (en) * | 2020-03-24 | 2021-09-28 | 中兴通讯股份有限公司 | Resource adjusting method, computer equipment and storage medium |
-
2021
- 2021-12-09 CN CN202111501983.3A patent/CN114168292A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105589751A (en) * | 2015-11-27 | 2016-05-18 | 杭州华三通信技术有限公司 | Physical resource scheduling method and device |
| CN109471727A (en) * | 2018-10-29 | 2019-03-15 | 北京金山云网络技术有限公司 | A kind of task processing method, apparatus and system |
| CN110278279A (en) * | 2019-06-27 | 2019-09-24 | 苏宁消费金融有限公司 | A kind of big data of dynamic resource scheduling mechanism dispatches development platform and method offline |
| WO2021063339A1 (en) * | 2019-09-30 | 2021-04-08 | 星环信息科技(上海)股份有限公司 | Cluster resource scheduling method, apparatus, device and storage medium |
| CN113448716A (en) * | 2020-03-24 | 2021-09-28 | 中兴通讯股份有限公司 | Resource adjusting method, computer equipment and storage medium |
| CN112506650A (en) * | 2020-11-27 | 2021-03-16 | 中国建设银行股份有限公司 | Resource allocation method, system, computer device and storage medium |
| CN113419839A (en) * | 2021-07-20 | 2021-09-21 | 北京字节跳动网络技术有限公司 | Resource scheduling method and device for multi-type jobs, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8793695B2 (en) | Information processing device and information processing method | |
| US6438553B1 (en) | Distributed job integrated management system and method | |
| US9319281B2 (en) | Resource management method, resource management device, and program product | |
| EP3567829B1 (en) | Resource management method and apparatus | |
| CN109710416B (en) | Resource scheduling method and device | |
| US10606650B2 (en) | Methods and nodes for scheduling data processing | |
| CN112948050A (en) | Method and device for deploying pod | |
| CN115220921A (en) | Resource scheduling method, image processor, image pickup device, and medium | |
| CN114138434A (en) | Big data task scheduling system | |
| CN112860383A (en) | Cluster resource scheduling method, device, equipment and storage medium | |
| CN111240824B (en) | CPU resource scheduling method and electronic equipment | |
| CN114968601A (en) | Scheduling method and scheduling system for AI training jobs with resources reserved according to proportion | |
| CN113419839A (en) | Resource scheduling method and device for multi-type jobs, electronic equipment and storage medium | |
| CN114168292A (en) | Resource scheduling method, device, equipment and medium | |
| CN109039694A (en) | A kind of the global network resource allocation methods and device of service-oriented | |
| CN117056080A (en) | Distribution method and device of computing resources, computer equipment and storage medium | |
| CN112068960A (en) | CPU resource allocation method, device, storage medium and equipment | |
| CN112003931A (en) | Method and system for deploying scheduling controller and related components | |
| CN115063282A (en) | GPU resource scheduling method, device, equipment and storage medium | |
| CN115168057B (en) | Resource scheduling method and device based on k8s cluster | |
| CN116974762A (en) | Method and system for scheduling and managing server resources | |
| CN114185689B (en) | Medical artificial intelligence and high-performance computing resource scheduling system and scheduling method | |
| KR20190061241A (en) | Mesos process apparatus for unified management of resource and method for the same | |
| CN116149804A (en) | Task scheduling method, device and server | |
| CN117311990B (en) | Resource adjustment method and device, electronic equipment, storage medium and training platform |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |