CN115033375A - Distributed task scheduling method, device, equipment and storage medium in cluster mode - Google Patents
Distributed task scheduling method, device, equipment and storage medium in cluster mode Download PDFInfo
- Publication number
- CN115033375A CN115033375A CN202210549648.9A CN202210549648A CN115033375A CN 115033375 A CN115033375 A CN 115033375A CN 202210549648 A CN202210549648 A CN 202210549648A CN 115033375 A CN115033375 A CN 115033375A
- Authority
- CN
- China
- Prior art keywords
- task
- job
- scheduling
- tasks
- thread
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a distributed task scheduling method, a distributed task scheduling device, distributed task scheduling equipment and a distributed task scheduling storage medium in a cluster mode, which are used for solving the technical problems of inflexible scheduling and low scheduling efficiency of a task scheduling framework. The method monitors the number of tasks by adding the thread monitor, and dynamically adjusts the configuration parameters of the thread pool according to the information such as the number of tasks and the like; the task monitor is added to monitor the execution state of the job task, and an interface for stopping the job task is provided to realize the capability of stopping the job task as required; the task monitor monitors the execution condition of the periodically scheduled job tasks and provides a dynamic adjustment interface of the task scheduling strategy so as to realize the function of dynamically adjusting the scheduling strategy of the periodically scheduled job tasks according to the resource load. The invention can improve the scheduling flexibility and scheduling efficiency of the job tasks of the distributed task scheduling framework, avoid the waste of resources, improve the utilization rate of the resources and prevent the downtime of the service and the database.
Description
Technical Field
The invention relates to the technical field of communication and cloud computing, in particular to a distributed task scheduling method, device, equipment and storage medium in a cluster mode.
Background
With the development of network and communication technology, software projects become more and more complex, and the application of micro-service architecture becomes more and more extensive. In a distributed business system based on microservice, a plurality of business systems have a requirement for executing tasks at regular time, and if all job tasks are scheduled in the business systems of the business systems, on one hand, the complexity of the business systems is increased, and on the other hand, the management of the systems is inconvenient.
Quartz, xxl-jobi, Elastic-Job, etc. are task scheduling frameworks/software/components/modules/systems that can implement distributed task scheduling and management. Taking Quartz as an example, Quartz supports clustering and transactions, Quartz can be used to execute timing tasks, and it has three core modules, namely scheduler (task scheduler), trigger (task trigger), and job (task executor). The jobs is used for executing tasks of users, the trigger is used for setting the execution time and the execution times of the tasks, and the scheduler can bind the jobs and the trigger together to realize the execution of the specified tasks at the specified time, so that certain service is executed at fixed time.
The task scheduling software described above has some drawbacks. Taking Quartz as an example, firstly, in the Quartz cluster mode, the size of the thread pool cannot be dynamically adjusted, which is likely to cause resource waste. Second, in the quartz cluster mode, threads in a single dispatch task cannot be stopped. In addition, in the quartt cluster mode, dynamic switching of thread sequential execution or concurrent execution cannot be realized for the tasks scheduled periodically.
Disclosure of Invention
In view of this, the present invention provides a distributed task scheduling method, apparatus, device and storage medium in a cluster mode, which are used to solve the technical problems of inflexible scheduling and low scheduling efficiency of a task scheduling framework.
Based on one aspect of the embodiments of the present invention, the present invention provides a distributed task scheduling method in a cluster mode, which is applied to a task scheduling node that provides a task scheduling function for a distributed computing cluster, and the method includes:
monitoring the number of tasks in a task scheduling information base through a thread monitor, and dynamically adjusting configuration parameters of a thread pool according to the number of tasks;
providing a job task stopping interface, responding to a job task stopping instruction through a task monitor, acquiring thread information for executing a job task, and stopping a corresponding job task;
and monitoring the execution condition of the periodically scheduled job tasks through the task monitor, and dynamically adjusting the scheduling strategy of the periodically scheduled job tasks according to the execution condition of the job tasks and the resource load condition.
Further, the method for dynamically adjusting the configuration parameters of the thread pool according to the number of tasks includes:
if the current task number exceeds the task number of the previous statistical period and is smaller than a preset maximum task number threshold value, increasing the thread number of the thread pool; and if the current task number is smaller than the preset proportion of the previous statistical period task number and is larger than a preset minimum task number threshold value, reducing the thread number of the thread pool.
Further, the step of obtaining thread information for executing the job task by the task monitor in response to the job task stop instruction, and stopping the corresponding job task includes:
and the task monitor inquires thread information for executing the corresponding job task in a task scheduling information base according to the job task identifier carried in the job task stopping instruction, and sends the job task stopping instruction to the computing node where the thread is located so as to enable the computing node to stop executing the job task.
Further, the method for monitoring the execution condition of the periodically scheduled job task by the task monitor and dynamically adjusting the scheduling policy of the periodically scheduled job task according to the execution condition of the job task and the resource load condition includes:
periodically acquiring the execution condition of the periodically scheduled job tasks and the resource load condition of the computing nodes and the service database for executing the periodically scheduled job tasks by a task monitor;
under the condition that the resources of the computing nodes and the service database are sufficient, adjusting the scheduling strategy of the job tasks, and allowing the periodically scheduled job tasks to be executed in parallel;
and under the condition that the resources of the computing node and the service database are insufficient, adjusting the scheduling strategy of the job task to enable the job task scheduled periodically to be executed in series.
Further, the execution condition of the periodically scheduled job task includes whether the job task is completed or not, and the estimated completion duration of the uncompleted job task;
under the condition that the former periodically scheduled job task is not finished, the task monitor dynamically adjusts the scheduling strategy of the subsequent periodically scheduled job task according to the resource conditions of the computing node and the service database;
and the task scheduler performs task scheduling according to the modified task scheduling strategy.
Based on another aspect of the embodiments of the present invention, the present invention further provides a distributed task scheduling device in a cluster mode, where the device is applied to a task scheduling node that provides a task scheduling function for a distributed computing cluster, and the device includes:
the thread monitor is used for monitoring the number of tasks in the task scheduling information base and dynamically adjusting the configuration parameters of the thread pool according to the number of the tasks;
the task monitor is used for providing a task stopping interface, responding to a task stopping instruction, acquiring thread information for executing a task and stopping a corresponding task; and the scheduling policy module is also used for monitoring the execution condition of the periodically scheduled job tasks and dynamically adjusting the scheduling policy of the periodically scheduled job tasks according to the execution condition of the job tasks and the resource load condition.
Further, when the current task number exceeds the task number of the previous statistical period and is less than a preset maximum task number threshold value, the task monitor increases the thread number of the thread pool; and when the current task number is smaller than the preset proportion of the previous statistical period task number and is larger than a preset minimum task number threshold value, reducing the thread number of the thread pool.
Further, the task monitor includes:
the task stopping module is used for inquiring thread information for executing the corresponding job task in the task scheduling information base according to the job task identifier carried in the job task stopping instruction, and sending the job task stopping instruction to the computing node where the thread is located so as to enable the computing node to stop the execution of the job task;
the state resource acquisition module is used for periodically acquiring the execution condition of the periodically scheduled job task and the resource load condition of a computing node and a service database which execute the periodically scheduled job task;
the scheduling strategy adjusting module is used for adjusting the scheduling strategy of the job task under the condition that the resources of the computing node and the service database are sufficient, and allowing the periodically scheduled job task to be executed in parallel; and under the condition that the resources of the computing node and the service database are insufficient, adjusting the scheduling strategy of the job task to enable the job task scheduled periodically to be executed in series.
The method monitors the number of tasks by adding the thread monitor, and dynamically adjusts the configuration parameters of the thread pool according to the information such as the number of tasks and the like; the task monitor is added to monitor the execution state of the job task, and an interface for stopping the job task is provided to realize the capability of stopping the job task as required; the task monitor monitors the execution condition of the periodically scheduled job tasks and provides a dynamic adjustment interface of the task scheduling strategy so as to realize the function of dynamically adjusting the scheduling strategy of the periodically scheduled job tasks according to the resource load. According to the invention, the scheduling flexibility and scheduling efficiency of the job tasks of the distributed task scheduling framework can be improved by adding the thread monitor and the task monitor, the waste of resources is avoided, the utilization rate of the resources is improved, and the downtime of the service and the database is prevented.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments of the present invention or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present invention, and for those skilled in the art, other drawings may be obtained according to the drawings of the embodiments of the present invention.
FIG. 1 is a schematic diagram of a distributed computing architecture to which the distributed task scheduling method in the cluster mode provided by the present invention is applied;
FIG. 2 is a flowchart illustrating steps of a distributed task scheduling method in a cluster mode according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device for implementing the distributed task scheduling method in the cluster mode according to the present invention.
Detailed Description
The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of the invention. As used in this embodiment of the invention, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The term "and/or" as used herein is meant to encompass any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used to describe various information in embodiments of the present invention, the information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information, without departing from the scope of embodiments of the present invention. Depending on the context, moreover, the word "if" as used may be interpreted as "at … …" or "when … …" or "in response to a determination".
The invention aims to solve the technical problems that a task scheduling framework in a distributed application scene in a cluster mode cannot dynamically adjust the size of a thread pool, cannot stop scheduling tasks according to needs, cannot dynamically adjust a periodically scheduled job task scheduling strategy and the like. In order to achieve the purpose of the invention, the invention provides a distributed task scheduling method under a cluster mode, and the basic idea of the invention is as follows: increasing the number of tasks monitored by a thread monitor, and dynamically adjusting configuration parameters of a thread pool according to information such as the number of tasks; a task monitor is added to monitor the execution state of the job task and provide an interface for stopping the job task so as to realize the capability of stopping the job task as required; the task monitor monitors the execution condition of the periodically scheduled job tasks and provides a dynamic adjustment interface of the task scheduling strategy so as to realize the function of dynamically adjusting the scheduling strategy of the periodically scheduled job tasks according to the resource load. The invention can improve the scheduling flexibility and the scheduling efficiency of the Job tasks of the distributed task scheduling framework such as Quartz, xxl-joba, Elastic-Job and the like through adding the thread monitor and the task monitor, avoids the waste of resources, improves the utilization rate of the resources and prevents the downtime of the service and the database.
Based on the basic idea of the invention, the following describes a specific implementation process of the invention with reference to the accompanying drawings and specific embodiments.
Fig. 1 is a schematic diagram of a distributed computing architecture to which the distributed task scheduling method in the cluster mode provided by the present invention is applied. The system comprises a task scheduling cluster 100, a distributed computing cluster 130 and a service database 140, wherein the task scheduling cluster 100 may comprise one or more task scheduling nodes 110, and the information related to task scheduling generated and managed and maintained by the task scheduling nodes 110 is stored in a task scheduling information base 120. The distributed computing cluster 130 is used to actually execute job tasks, and the business data processed by the job tasks are stored in the business database 140.
The task scheduling node 110 includes basic components including a task Scheduler (Scheduler), a task Trigger (Trigger), and a task executor (Job), and the present invention adds a thread monitor component and a task monitor component on the basis of the Quartz basic component.
The task scheduler is used to bind the trigger and the job task, and the job task information, the trigger information, and the binding relationship between the job task and the trigger are stored in the task scheduling information base 120. And the task executor is used for acquiring the thread from the thread pool and distributing the job task to the computing nodes in the distributed computing cluster according to the binding relationship between the job task and the trigger to execute the job task.
A thread pool refers to a collection of threads created to perform a job task. The thread in the thread pool is created when the system is started, when the task executor allocates the job task to one thread for execution, the thread is switched into a working state from an idle state, when the job task is executed, the task executor releases the used thread back to the thread pool, the thread in the idle state in the thread pool cannot be killed, and the next job task is waited to be scheduled for execution.
And the thread monitor is used for monitoring the number of the operation tasks and dynamically adjusting the size of the thread pool according to the number of the operation tasks. The thread monitor monitors the number of job tasks by registering events, an event generator (event source) sends a message to a monitor (thread monitor) registering the event to inform the monitor of certain information, and the monitor can also obtain an event object and can obtain relevant attributes and execute relevant operations according to the object.
The task monitor is used for monitoring the execution condition of the job task and closing the specified job task in response to the job task stopping instruction; and the scheduling strategy is also used for dynamically adjusting the periodically scheduled job tasks according to the execution condition of the job tasks. The task monitor is mainly used for timely clearing fault operation tasks, adjusting a scheduling strategy according to the resource use condition of the computing node and improving the high availability of services.
The task scheduling cluster 100 provides scheduling functionality for a large number of job tasks for the distributed computing cluster 130. In a big data service application scenario in which a distributed computing architecture is used to process a large amount of data, the data volume in the service database 140 of a user is often huge, for example, the number of data records may reach more than 1 hundred million, the task scheduling cluster 100 schedules job tasks to process massive service data, and efficient, flexible and highly available scheduling of the job tasks is a core performance index of a service system.
In the service application scenario provided in an embodiment of the present invention, a task is used to perform data quality check and verification (referred to as "check" for short) on a large amount of service data stored in a service database, and a task scheduling framework is implemented by using Quartz. The following describes a specific embodiment by taking the application scenario as an example, but the scope of the present invention is not limited to the application scenario, and is also applicable to other similar application scenarios and other task scheduling frameworks.
Fig. 2 is a flowchart illustrating steps of a distributed task scheduling method in a cluster mode according to an embodiment of the present invention. The method of the embodiment is applied to the task scheduling node 110 as illustrated in fig. 1, and comprises the following steps:
step 201, monitoring the task quantity in a task scheduling information base through a thread monitor, and dynamically adjusting configuration parameters of a thread pool according to the task quantity;
taking a checking service providing data quality service as an example, a thread monitor monitors checking tasks of the data quality service by registering a monitoring event, if the number of the checking tasks is increased, the number of threads in a thread pool is dynamically increased, so that the thread pool can contain more threads for completing checking task; when the checking task is reduced, the number of threads in the thread pool is dynamically reduced, thereby reducing the waste of resources and the shortage of resource use.
The method for dynamically expanding and contracting the specific thread pool can be controlled based on a preset dynamic adjustment policy, for example, the dynamic adjustment policy can be: if the number of the tasks of the current operation exceeds the number of the tasks of the previous statistical period and is less than a preset maximum task number threshold value, increasing the number of the threads of the thread pool by 10 percent; and if the current task number is less than 50% of the previous statistical cycle task number and greater than a preset minimum task number threshold value, reducing the thread number of the thread pool by 10%.
The expansion and contraction of the thread pool only considers the factor of the number of tasks, and in actual service, the decision can be comprehensively made according to the factors of task parallelism, the amount of resources required by the tasks, the length of task execution time and the like. The setting of the maximum bearable task quantity threshold value and the minimum task quantity threshold value needs to be comprehensively judged and configured by considering factors such as hardware resource configuration of the distributed computing cluster, the quantity of scheduling tasks, time consumed by executed services and the like.
An application scenario of the checking service is that a task scheduling cluster schedules a plurality of checking tasks to check data of a service database based on a checking rule so as to find problem data, for example, a field B of a table a in the service database a is checked for a null value, a field C is checked for a unique constraint, a field D of the table B is checked for value validity, and the like. Multiple checking tasks may be executed in parallel, and when multiple checking tasks are executed simultaneously and in parallel for the same service database, great pressure may be applied to the service database or the computing node.
the invention adds a task monitor which provides a task stopping interface, when a user sends a task stopping instruction through the task stopping interface, the task monitor responds to the task stopping instruction, inquires thread information for executing a corresponding task in a task scheduling information base, and sends the task stopping instruction to a computing node where a thread is located so as to enable the computing node to stop the execution of the task.
In a distributed computing application project, for example, when Quartz is used for task scheduling, due to some software and hardware reasons, an execution thread of a certain job task scheduled once or scheduled periodically may fail to stop, and in order to solve the technical problem, the invention adds a task monitor to Quartz.
The task stopping instruction carries a task identifier, the task monitor inquires information of a computing node executing a thread corresponding to a task in a task scheduling information base according to the task identifier, and sends the task stopping instruction to the computing node so as to enable the computing node to stop executing the task. After receiving an instruction for stopping the job task, the computing nodes in the distributed computing cluster find out the thread for executing the corresponding job task according to the thread and the job task identifier in the instruction, and stop the execution of the corresponding thread, so that the thread is released from the thread pool.
And 203, monitoring the execution condition of the periodically scheduled job task through the task monitor, and dynamically adjusting the scheduling strategy of the periodically scheduled job task according to the execution condition of the job task and the resource load condition.
For a large number of job tasks needing periodic scheduling, a large number of threads need to be started for processing, and a server of a computing node executing the job tasks may be down due to resource utilization. The invention provides a technical scheme for dynamically adjusting a scheduling strategy of a periodic job task, aiming at the problems of service downtime and target database downtime caused by the periodically scheduled job task in a distributed application scene.
In an embodiment of the present invention, a task monitor is used to control a flexible scheduling policy, and the scheduling policy of a periodically scheduled job task is dynamically adjusted by obtaining resource load conditions (for example, the number of database connections, the available memory usage ratio, the CPU usage ratio, etc.) of a compute node and service database resource usage conditions (for example, the memory usage ratio, the CPU usage ratio, etc.) of a service node.
For example, in a data quality check application, a periodically scheduled check job task is executed 10 times with a start execution time of 8 am, an end execution time of 13:00 pm, and a scheduling interval of 30 minutes. When the data size in the service database (storing the service data in the production environment of the client, which is the object to be collected and checked by the checking job task) is large, the first time the checking job task is executed is 1 hour, and if the checking job task is executed every half hour by using the previous job task scheduling strategy, then a plurality of threads will be executed in parallel subsequently, which may cause the resource consumption of the computing node or the service database to be too large, and may cause the server to be down in a serious case.
The task monitor can acquire the execution condition of the job task in real time or periodically. For example, the task executor may update the execution result of the job task to the task scheduling information base, and the task monitor may read the execution result from the task scheduling information base. The task monitor can also obtain the execution state of the execution thread of the job task on the computing node, the resource use condition of the computing node and the service database, and the like through the message.
The task monitor can periodically read the execution condition of the job task in the task scheduling information base, acquire the resource use condition of the computing node and the service database, and timely decide whether to adjust the task scheduling strategy according to the completion condition of the periodic job task and the resource use condition. Taking the foregoing data checking job task as an example, assuming that the task monitor obtains the execution condition of the periodically scheduled job task and the resource usage condition of the computing node and the service database every 1 minute, if at point 8 and point 28, the task monitor estimates that it takes 10 minutes to complete the current checking job task according to the completion proportion and the progress information of the current checking job task, if according to the original task scheduling policy, the task scheduler will schedule and execute the checking job task again at point 8 and point 30, and in the case of too heavy resource load of the current computing node or service database, opening a new checking job task according to the original task scheduling policy will further aggravate the resource load of the computing node or service database, possibly resulting in situations such as blocked threads, downtime of the server, and the like. In order to improve the flexibility of task scheduling and the high availability of a service system, the task scheduler may adjust the scheduling policy of the checking job task according to the current available resource condition, for example, in the case of sufficient current available resources, the original task scheduling policy is maintained, and the subsequent checking job tasks are allowed to be executed in parallel. And under the condition that the current available resources are in shortage, dynamically adjusting a current task scheduling strategy, allowing serial execution but not allowing parallel execution of the checking job task, scheduling the task by the task scheduler according to the modified task scheduling strategy, and executing the next checking job task after scheduling after the previous checking job task is executed.
Fig. 3 is a schematic structural diagram of an electronic device for implementing a distributed task scheduling method in a cluster mode, where the device 300 includes: a processor 310, such as a Central Processing Unit (CPU), a communication bus 320, a communication interface 340, and a storage medium 330. Wherein the processor 310 and the storage medium 330 may communicate with each other through a communication bus 320. The storage medium 330 stores therein a computer program which, when executed by the processor 310, implements the functions of one or more steps of the distributed task scheduling method in cluster mode provided by the present invention.
The storage medium may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), for example, at least one disk Memory. In addition, the storage medium may be at least one memory device located remotely from the processor. The Processor may be a general-purpose Processor including a Central Processing Unit (CPU), a Network Processor (NP), etc.; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
It should be recognized that embodiments of the present invention can be realized and implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory memory. The method may be implemented in a computer program using standard programming techniques, including a non-transitory storage medium configured with the computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner. Each program may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose. Further, operations of processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executed on one or more processors, by hardware, or combinations thereof. The computer program includes a plurality of instructions executable by one or more processors.
Further, the method may be implemented in any type of computing platform operatively connected to a suitable interface, including but not limited to a personal computer, mini computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, and the like. Aspects of the invention may be embodied in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optically read and/or write storage medium, RAM, ROM, or the like, such that it may be read by a programmable computer, which when read by the storage medium or device, is operative to configure and operate the computer to perform the procedures described herein. Further, the machine-readable code, or portions thereof, may be transmitted over a wired or wireless network. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media include instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. The invention also includes the computer itself when programmed according to the methods and techniques described herein.
The above description is only an example of the present invention, and is not intended to limit the present invention. Various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A distributed task scheduling method in a cluster mode is applied to task scheduling nodes providing task scheduling functions for distributed computing clusters, and the method comprises the following steps:
monitoring the number of tasks in a task scheduling information base through a thread monitor, and dynamically adjusting configuration parameters of a thread pool according to the number of tasks;
providing a task stopping task interface, responding to a task stopping task instruction through a task monitor, acquiring thread information for executing a task, and stopping a corresponding task;
and monitoring the execution condition of the periodically scheduled job tasks through the task monitor, and dynamically adjusting the scheduling strategy of the periodically scheduled job tasks according to the execution condition of the job tasks and the resource load condition.
2. The method of claim 1, wherein the method for dynamically adjusting the configuration parameters of the thread pool according to the number of tasks comprises:
if the current task number exceeds the task number of the previous statistical period and is smaller than a preset maximum task number threshold value, increasing the thread number of the thread pool; and if the current task number is smaller than the preset proportion of the previous statistical period task number and is larger than a preset minimum task number threshold value, reducing the thread number of the thread pool.
3. The method of claim 1, wherein the step of obtaining thread information for executing the job task in response to the stop job task instruction via the task monitor, the step of stopping the corresponding job task comprises:
and the task monitor inquires thread information for executing the corresponding job task in a task scheduling information base according to the job task identifier carried in the job task stopping instruction, and sends the job task stopping instruction to the computing node where the thread is located so as to enable the computing node to stop executing the job task.
4. The method according to claim 1, wherein the method for monitoring the execution condition of the periodically scheduled job task by the task monitor and dynamically adjusting the scheduling policy of the periodically scheduled job task according to the execution condition of the job task and the resource load condition comprises:
periodically acquiring the execution condition of the periodically scheduled job task and the resource load condition of a computing node and a service database for executing the periodically scheduled job task by a task monitor;
under the condition that the resources of the computing nodes and the service database are sufficient, adjusting the scheduling strategy of the job tasks, and allowing the periodically scheduled job tasks to be executed in parallel;
and under the condition that the resources of the computing node and the service database are insufficient, adjusting the scheduling strategy of the job task to enable the job task scheduled periodically to be executed in series.
5. The method of claim 1,
the execution condition of the periodically scheduled job task comprises whether the job task is completed or not and the estimated completion time of the unfinished job task;
under the condition that the former periodically scheduled job task is not finished, the task monitor dynamically adjusts the scheduling strategy of the subsequent periodically scheduled job task according to the resource conditions of the computing node and the service database;
and the task scheduler performs task scheduling according to the modified task scheduling strategy.
6. A distributed task scheduling apparatus in a cluster mode, the apparatus being applied to a task scheduling node providing a task scheduling function for a distributed computing cluster, the apparatus comprising:
the thread monitor is used for monitoring the number of tasks in the task scheduling information base and dynamically adjusting the configuration parameters of the thread pool according to the number of the tasks;
the task monitor is used for providing a task stopping interface, responding to a task stopping instruction, acquiring thread information for executing a task and stopping a corresponding task; and the scheduling policy module is also used for monitoring the execution condition of the periodically scheduled job tasks and dynamically adjusting the scheduling policy of the periodically scheduled job tasks according to the execution condition of the job tasks and the resource load condition.
7. The apparatus of claim 6,
when the current task number exceeds the task number of the previous statistical period and is smaller than a preset maximum task number threshold value, the task monitor increases the thread number of the thread pool; and when the current task number is smaller than the preset proportion of the previous statistical period task number and is larger than a preset minimum task number threshold value, reducing the thread number of the thread pool.
8. The apparatus of claim 6, wherein the task monitor comprises:
the task stopping module is used for inquiring thread information for executing the corresponding job task in the task scheduling information base according to the job task identifier carried in the job task stopping instruction, and sending the job task stopping instruction to the computing node where the thread is located so as to enable the computing node to stop the execution of the job task;
the state resource acquisition module is used for periodically acquiring the execution condition of the periodically scheduled job task and the resource load condition of a computing node and a service database which execute the periodically scheduled job task;
the scheduling strategy adjusting module is used for adjusting the scheduling strategy of the job task under the condition that the resources of the computing node and the service database are sufficient, and allowing the periodically scheduled job task to be executed in parallel; and under the condition that the resources of the computing node and the service database are insufficient, adjusting the scheduling strategy of the job task to enable the job task scheduled periodically to be executed in series.
9. An electronic device is characterized by comprising a processor, a communication interface, a storage medium and a communication bus, wherein the processor, the communication interface and the storage medium are communicated with each other through the communication bus;
a storage medium for storing a computer program;
a processor for performing the method of any one of claims 1 to 5 when executing a computer program stored on a storage medium.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210549648.9A CN115033375A (en) | 2022-05-20 | 2022-05-20 | Distributed task scheduling method, device, equipment and storage medium in cluster mode |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210549648.9A CN115033375A (en) | 2022-05-20 | 2022-05-20 | Distributed task scheduling method, device, equipment and storage medium in cluster mode |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115033375A true CN115033375A (en) | 2022-09-09 |
Family
ID=83121044
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210549648.9A Pending CN115033375A (en) | 2022-05-20 | 2022-05-20 | Distributed task scheduling method, device, equipment and storage medium in cluster mode |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115033375A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116302451A (en) * | 2023-05-18 | 2023-06-23 | 广州豪特节能环保科技股份有限公司 | Offline energy-saving scheduling method and system for cloud computing data center |
| CN116737345A (en) * | 2023-08-11 | 2023-09-12 | 之江实验室 | Distributed task processing system, distributed task processing method, distributed task processing device, storage medium and storage device |
-
2022
- 2022-05-20 CN CN202210549648.9A patent/CN115033375A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116302451A (en) * | 2023-05-18 | 2023-06-23 | 广州豪特节能环保科技股份有限公司 | Offline energy-saving scheduling method and system for cloud computing data center |
| CN116302451B (en) * | 2023-05-18 | 2023-08-08 | 广州豪特节能环保科技股份有限公司 | Offline energy-saving scheduling method and system for cloud computing data center |
| CN116737345A (en) * | 2023-08-11 | 2023-09-12 | 之江实验室 | Distributed task processing system, distributed task processing method, distributed task processing device, storage medium and storage device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111625331B (en) | Task scheduling method, device, platform, server and storage medium | |
| CN115033375A (en) | Distributed task scheduling method, device, equipment and storage medium in cluster mode | |
| Buttazzo et al. | Limited preemptive scheduling for real-time systems. a survey | |
| US7440973B2 (en) | Systems, methods and software for automating database tasks | |
| US8392572B2 (en) | Method for scheduling cloud-computing resource and system applying the same | |
| US8943353B2 (en) | Assigning nodes to jobs based on reliability factors | |
| WO2021159638A1 (en) | Method, apparatus and device for scheduling cluster queue resources, and storage medium | |
| CN109343939B (en) | Distributed cluster and parallel computing task scheduling method | |
| CN104580396A (en) | Task scheduling method, node and system | |
| CN107241380B (en) | Method and apparatus for time-based adjusted load balancing | |
| Chard et al. | Cost-aware cloud provisioning | |
| JP2008527514A5 (en) | ||
| CN106557369A (en) | A kind of management method and system of multithreading | |
| CN113467933B (en) | Distributed file system thread pool optimization method, system, terminal and storage medium | |
| CN102768637A (en) | Method and device for controlling test execution | |
| CN112181621A (en) | Task scheduling system, method, equipment and storage medium | |
| CN112685160A (en) | Scheduling method and device of timing task, terminal equipment and computer storage medium | |
| CN110196773B (en) | Multi-time-scale security check system and method for unified scheduling computing resources | |
| Buttazzo et al. | Efficient EDF implementation for small embedded systems | |
| CN114461385A (en) | Thread pool scheduling method, device and equipment and readable storage medium | |
| CN111767125B (en) | Task execution method, device, electronic equipment and storage medium | |
| CN103164338B (en) | The analogy method of concurrent processing system and device | |
| CN116719623A (en) | Job scheduling method, job result processing method and device | |
| Zabolotnyi et al. | Profiling-based task scheduling for factory-worker applications in infrastructure-as-a-service clouds | |
| CN115712572A (en) | Task testing method and device, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |