CN111078386A - Control method and control device of distributed scheduling system - Google Patents
Control method and control device of distributed scheduling system Download PDFInfo
- Publication number
- CN111078386A CN111078386A CN201911391951.5A CN201911391951A CN111078386A CN 111078386 A CN111078386 A CN 111078386A CN 201911391951 A CN201911391951 A CN 201911391951A CN 111078386 A CN111078386 A CN 111078386A
- Authority
- CN
- China
- Prior art keywords
- target
- threads
- target task
- processing
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 76
- 230000000875 corresponding effect Effects 0.000 description 81
- 230000001276 controlling effect Effects 0.000 description 17
- 238000010586 diagram Methods 0.000 description 5
- 230000002596 correlated effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multi Processors (AREA)
Abstract
The application discloses a control method and a control device of a distributed scheduling system, which are applied to a server of the distributed scheduling system, wherein the control method comprises the following steps: acquiring the number of unprocessed messages of the target task according to a preset time interval; determining the number of target threads corresponding to the target task according to the number of unprocessed messages, wherein the number of unprocessed messages is in positive correlation with the number of target threads corresponding to the target task; and when the number of the threads for processing the target task at present is different from the target thread number corresponding to the target task, adjusting the threads for processing the target task to enable the number of the threads for processing the target task to be the target thread number. Based on the control method disclosed by the application, the problem of message queue backlog can be avoided during the service peak period.
Description
Technical Field
The present application belongs to the field of data processing technologies, and in particular, to a control method and a control device for a distributed scheduling system.
Background
The distributed scheduling system is a system for managing tasks and scheduling tasks across machine rooms.
Fig. 1 shows an architecture of a distributed scheduling system, comprising: the system comprises a server and a plurality of terminals, wherein the server is used as a main control node of the distributed scheduling system, and the terminals are execution nodes of the distributed scheduling system. The server receives a task processing request sent by the client, allocates resources for the task to be processed according to a preset strategy, namely allocates one or more terminals to process the task, and the terminals report the processing result to the server.
However, the applicant has found that the following problems exist in the operation process of the distributed scheduling system: during the peak period of the service, the problem of message queue backlog is easy to occur.
Disclosure of Invention
In view of the above, an object of the present application is to provide a control method and a control device for a distributed scheduling system, so as to solve the problem that message queue backlog is very likely to occur in a service peak period of the distributed scheduling system.
In order to achieve the above purpose, the present application provides the following technical solutions:
in one aspect, the present application provides a control method for a distributed scheduling system, which is applied to a server of the distributed scheduling system, and the control method includes:
acquiring the number of unprocessed messages of the target task according to a preset time interval;
determining a target thread number corresponding to the target task according to the number of the unprocessed messages, wherein the number of the unprocessed messages and the target thread number corresponding to the target task are in positive correlation;
and under the condition that the number of threads for processing the target task currently is different from the target number of threads corresponding to the target task, adjusting the threads for processing the target task to enable the number of the threads for processing the target task to be the target number of threads.
Optionally, the determining the number of target threads corresponding to the target task according to the number of the unprocessed messages includes:
comparing the number of unprocessed messages with a first threshold and a second threshold, wherein the first threshold and the second threshold are positive numbers, and the first threshold is greater than or equal to the second threshold;
if the number of the unprocessed messages is larger than the first threshold value, taking the sum of the number of the threads currently processing the target task and a first numerical value as the target thread number corresponding to the target task;
and if the number of the unprocessed messages is smaller than the second threshold value, taking the difference value between the number of the threads currently processing the target task and the first numerical value as the target thread number corresponding to the target task.
Optionally, the determining the number of target threads corresponding to the target task according to the number of the unprocessed messages includes:
comparing the number of unprocessed messages with a first threshold and a second threshold, wherein the first threshold and the second threshold are positive numbers, and the first threshold is greater than or equal to the second threshold;
if the number of the unprocessed messages is larger than the first threshold value, calculating a sum of the number of threads currently processing the target task and a first numerical value, if the sum is smaller than or equal to a preset maximum number of threads, taking the sum as the number of the target threads corresponding to the target task, and if the sum is larger than the maximum number of threads, taking the maximum number of threads as the number of the target threads corresponding to the target task;
if the number of the unprocessed messages is smaller than the second threshold value, calculating a difference value between the number of threads currently processing the task and a first numerical value, if the difference value is larger than or equal to a preset minimum number of threads, taking the difference value as a target number of threads corresponding to the target task, and if the difference value is smaller than the minimum number of threads, taking the minimum number of threads as a target number of threads corresponding to the target task.
Optionally, the adjusting the threads of the target task so that the number of threads processing the target task is the number of the target threads includes:
executing one or more times of first operation under the condition that the number of target threads corresponding to the target task is larger than the number of threads for processing the target task at present until the number of threads for processing the target task reaches the target number of threads;
wherein the first operation comprises: searching a first target node in a plurality of execution nodes for processing the target task, wherein the first target node is one execution node with the least number of threads for processing the target task, which is started from the plurality of execution nodes; and controlling the first target node to start a new thread to process the target task.
Optionally, the adjusting the threads of the target task so that the number of threads processing the target task is the number of the target threads, further includes:
executing one or more times of second operation under the condition that the number of target threads corresponding to the target task is smaller than the number of threads for processing the target task at present until the number of threads for processing the target task reaches the target number of threads;
wherein the second operation comprises: searching a second target node in a plurality of execution nodes for processing the target task, wherein the second target node is one execution node with the largest number of threads for processing the target task, which is started from the plurality of execution nodes; and controlling the second target node to close a thread for processing the target task.
In another aspect, the present application provides a control device for a distributed scheduling system, which is applied to a server of the distributed scheduling system, the control device including:
the message quantity acquiring unit is used for acquiring the quantity of unprocessed messages of the target task according to a preset time interval;
a target thread number determining unit, configured to determine a target thread number corresponding to the target task according to the number of the unprocessed messages, where the number of the unprocessed messages and the target thread number corresponding to the target task are in a positive correlation;
and the thread processing unit is used for adjusting the threads for processing the target task under the condition that the number of the threads for processing the target task at present is different from the target thread number corresponding to the target task, so that the number of the threads for processing the target task is the target thread number.
Optionally, the target thread number determining unit includes:
a numerical value comparison subunit, configured to compare the number of unprocessed messages with a first threshold and a second threshold, where the first threshold and the second threshold are positive numbers, and the first threshold is greater than or equal to the second threshold;
a first processing subunit, configured to, when the number of the unprocessed messages is greater than the first threshold, take a sum of a first numerical value and the number of threads currently processing the target task as a target thread number corresponding to the target task;
and the second processing subunit is configured to, when the number of the unprocessed messages is smaller than the second threshold, take a difference between the number of threads currently processing the target task and the first value as a target thread number corresponding to the target task.
Optionally, the target thread number determining unit includes:
a numerical value comparison subunit, configured to compare the number of unprocessed messages with a first threshold and a second threshold, where the first threshold and the second threshold are positive numbers, and the first threshold is greater than or equal to the second threshold;
a third processing subunit, configured to, when the number of unprocessed messages is greater than the first threshold, calculate a sum of a first number and the number of threads currently processing the target task, if the sum is less than or equal to a preset maximum number of threads, use the sum as the target number of threads corresponding to the target task, and if the sum is greater than the maximum number of threads, use the maximum number of threads as the target number of threads corresponding to the target task;
and the fourth processing subunit is configured to, when the number of the unprocessed messages is smaller than the second threshold, calculate a difference between the number of threads currently processing the task and a first value, if the difference is greater than or equal to a preset minimum number of threads, take the difference as a target number of threads corresponding to the target task, and if the difference is smaller than the minimum number of threads, take the minimum number of threads as the target number of threads corresponding to the target task.
Optionally, the thread processing unit includes:
the first thread adjusting subunit is configured to, when the number of target threads corresponding to the target task is greater than the number of threads currently processing the target task, perform one or more first operations until the number of threads processing the target task reaches the target number of threads; wherein the first operation comprises: searching a first target node in a plurality of execution nodes for processing the target task, wherein the first target node is one execution node with the least number of threads for processing the target task, which is started from the plurality of execution nodes; and controlling the first target node to start a new thread to process the target task.
Optionally, the thread processing unit further includes:
a second thread adjusting subunit, configured to, when a number of target threads corresponding to the target task is smaller than a number of threads currently processing the target task, perform one or more second operations until the number of threads processing the target task reaches the target number of threads; wherein the second operation comprises: searching a second target node in a plurality of execution nodes for processing the target task, wherein the second target node is one execution node with the largest number of threads for processing the target task, which is started from the plurality of execution nodes; and controlling the second target node to close a thread for processing the target task.
Therefore, the beneficial effects of the application are as follows:
the control method of the distributed scheduling system disclosed by the application comprises the steps that a server acquires the number of unprocessed messages of a target task according to a preset time interval, determining the number of target threads corresponding to the target task according to the number of unprocessed messages of the target task, then adjusting the threads for processing the target task by the server to ensure that the number of the threads for processing the target task is consistent with the number of the target threads, wherein, the quantity of the unprocessed messages of the target task is positively correlated with the quantity of the target threads corresponding to the target task, therefore, when the target task has more unprocessed messages due to the occurrence of a traffic peak, the server automatically allocates more threads to process the target task, the problem of message queue backlog can be avoided, when fewer messages are not processed by the target task, the server automatically reduces the threads for processing the target task, thereby freeing up more resources in the execution node for processing other tasks.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic structural diagram of a conventional distributed scheduling system;
FIG. 2 is a flow chart of a control method of a distributed scheduling system as disclosed herein;
FIG. 3 is a flow chart of another method of controlling a distributed scheduling system as disclosed herein;
FIG. 4 is a flow chart of another method of controlling a distributed scheduling system as disclosed herein;
FIG. 5 is a flow chart of another method of controlling a distributed scheduling system as disclosed herein;
fig. 6 is a schematic structural diagram of a control device of a distributed scheduling system disclosed in the present application;
fig. 7 is a schematic structural diagram of a control system of a distributed scheduling system disclosed in the present application.
Detailed Description
The application discloses a control method and a control device of a distributed scheduling system, which can avoid the problem that the distributed scheduling system has message queue backlog in the service peak period.
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 2, fig. 2 is a flowchart of a control method of a distributed scheduling system, which is applied to a server of the distributed scheduling system, and specifically includes:
step S1: and acquiring the number of the messages which are not processed by the target task according to a preset time interval.
And pre-configuring a task to be monitored through a configuration center, wherein the task is a target task. It should be noted that the target task may be one or more.
The server acquires the number of the messages which are not processed by the target task according to the configuration of the configuration center and according to a preset time interval, wherein the messages which are not processed can also be called unconsumed messages. Wherein, the message mainly comprises Redis, ActiveMQ and RocktetMQ.
In practice, the preset time interval may be a fixed time interval, for example, the preset time interval is set to 3 seconds. In addition, the preset time interval may also be dynamically changed, for example, when the traffic volume is large, the preset time interval is set to a small value, and when the traffic volume is small, the preset time interval is set to a large value.
Step S2: and determining the target thread number corresponding to the target task according to the number of the unprocessed messages.
Step S3: and under the condition that the number of threads for processing the target task at present is different from the target number of threads corresponding to the target task, adjusting the threads for processing the target task to enable the number of the threads for processing the target task to be the target number of threads.
The server obtains the number of unprocessed messages of the target task, and determines the target thread number corresponding to the target task according to the number of the unprocessed messages, wherein the number of the unprocessed messages of the target task is in positive correlation with the target thread number corresponding to the target task. That is, when the number of the unprocessed messages of the target task is larger, the target thread number corresponding to the target task is a larger value, and when the number of the unprocessed messages of the target task is smaller, the target thread number corresponding to the target task is a smaller value. And the server adjusts the threads for processing the target task, so that the number of the threads for processing the target task is consistent with the number of the target threads.
The control method of the distributed scheduling system disclosed by the application comprises the steps that a server acquires the number of unprocessed messages of a target task according to a preset time interval, determining the number of target threads corresponding to the target task according to the number of unprocessed messages of the target task, then adjusting the threads for processing the target task by the server to ensure that the number of the threads for processing the target task is consistent with the number of the target threads, wherein, the quantity of the unprocessed messages of the target task is positively correlated with the quantity of the target threads corresponding to the target task, therefore, when the target task has more unprocessed messages due to the occurrence of a traffic peak, the server automatically allocates more threads to process the target task, the problem of message queue backlog can be avoided, when fewer messages are not processed by the target task, the server automatically reduces the threads for processing the target task, thereby freeing up more resources in the execution node for processing other tasks.
In implementation, the target thread number corresponding to the target task is determined according to the number of unprocessed messages of the target task, and various schemes can be adopted for implementation. The following description is made with reference to fig. 3 and 4, respectively.
Referring to fig. 3, fig. 3 is a flowchart of another control method for a distributed scheduling system, which is applied to a server of the distributed scheduling system, and specifically includes:
step S1: and acquiring the number of the messages which are not processed by the target task according to a preset time interval.
Step S201: the number of unprocessed messages is compared to a first threshold and a second threshold.
The first threshold and the second threshold are both positive numbers, and the first threshold is greater than or equal to the second threshold.
Step S202: and if the number of the unprocessed messages is larger than the first threshold value, taking the sum of the number of the threads of the current processing target task and the first numerical value as the target thread number corresponding to the target task.
Step S203: and if the number of the unprocessed messages is smaller than a second threshold value, taking the difference value between the number of the threads of the current processing target task and the first numerical value as the target thread number corresponding to the target task.
Step S3: and under the condition that the number of threads for processing the target task at present is different from the target number of threads corresponding to the target task, adjusting the threads for processing the target task to enable the number of the threads for processing the target task to be the target number of threads.
The server obtains the number of the messages which are not processed by the target task, compares the number of the messages which are not processed with the first threshold value and the second threshold value, and determines the number of the target threads corresponding to the target task by adopting different strategies according to the comparison result.
In the case that the number of unprocessed messages of the target task is greater than the first threshold, the server takes the sum of the number of threads currently processing the target task and the first value as the target thread number corresponding to the target task, and accordingly, in step S3, the server adds the threads processing the target task so that the number of threads processing the target task coincides with the target thread number of the target task.
In the case that the number of unprocessed messages of the target task is smaller than the second threshold, the server determines the difference between the number of threads currently processing the target task and the first value as the target thread number corresponding to the target task, and accordingly, in step S3, the server reduces the number of threads processing the target task so that the number of threads processing the target task matches the target thread number of the target task.
In addition, when the number of unprocessed messages is between the second threshold and the first threshold (including two endpoint values of the first threshold and the second threshold), the server takes the number of threads currently processing the target task as the number of target threads corresponding to the target task, and in this case, the threads processing the target task may not be adjusted.
In the control method shown in fig. 3 of the present application, when the number of unprocessed messages of the target task is greater than the first threshold, the server uses the sum of the number of threads currently processing the target task and the first numerical value as the number of target threads corresponding to the target task, so as to increase the number of threads processing the target task, and can avoid the problem of message queue backlog. When the number of the messages which are not processed by the target task is smaller than the second threshold value, the server takes the difference value between the number of the threads which are currently used for processing the target task and the first numerical value as the number of the target threads corresponding to the target task, so that the number of the threads which are used for processing the target task is reduced, and more resources in the execution node are released to be used for processing other tasks.
Referring to fig. 4, fig. 4 is a flowchart of another control method of a distributed scheduling system, which is applied to a server of the distributed scheduling system, and specifically includes:
step S1: and acquiring the number of the messages which are not processed by the target task according to a preset time interval.
Step S211: the number of unprocessed messages is compared to a first threshold and a second threshold.
The first threshold and the second threshold are both positive numbers, and the first threshold is greater than or equal to the second threshold.
Step S212: if the number of unprocessed messages is larger than a first threshold value, the sum of the number of threads of the current processing target task and a first value is calculated.
Step S213: and if the sum is less than or equal to the preset maximum process number, taking the sum as the target process number corresponding to the target task, and if the sum is greater than the maximum process number, taking the maximum process number as the target process number corresponding to the target task.
Step S214: if the number of unprocessed messages is less than a second threshold, a difference between the number of threads of the current processing task and the first value is calculated.
Step S215: and if the difference is greater than or equal to the preset minimum process number, taking the difference as the target process number corresponding to the target task, and if the difference is less than the minimum process number, taking the minimum process number as the target process number corresponding to the target task.
The server obtains the number of the messages which are not processed by the target task, compares the number of the messages which are not processed with the first threshold value and the second threshold value, and determines the number of the target threads corresponding to the target task by adopting different strategies according to the comparison result.
And under the condition that the number of the messages which are not processed by the target task is greater than a first threshold value, the server calculates the sum of the number of the threads which process the target task currently and a first numerical value, compares the sum with a preset maximum number of threads, if the sum is less than or equal to the maximum number of threads, the sum is used as the target number of threads corresponding to the target task, and if the sum is greater than the maximum number of threads, the maximum number of threads is used as the target number of threads corresponding to the target task.
And under the condition that the number of the messages which are not processed by the target task is smaller than a second threshold value, the server calculates the difference value between the number of the threads which process the target task at present and a first numerical value, compares the difference value with a preset minimum number of threads, if the difference value is larger than or equal to the minimum number of threads, the difference value is used as the number of the target threads corresponding to the target task, and if the difference value is smaller than the minimum number of threads, the minimum number of threads is used as the number of the target threads corresponding to the target task.
Step S3: and under the condition that the number of threads for processing the target task at present is different from the target number of threads corresponding to the target task, adjusting the threads for processing the target task to enable the number of the threads for processing the target task to be the target number of threads.
Compared with the control method shown in fig. 3, the target thread number corresponding to the target task, which is determined by the server according to the number of unprocessed messages of the target task, does not exceed the maximum thread number and the minimum thread number configured in advance for the target task, so that excessive resource occupation of a certain task can be avoided, and stable operation of a distributed scheduling system is facilitated.
In the control method shown in fig. 2 and 3 described above in the present application, it is preferable to set the first value to 1. In this case, the server is to gradually add or close the threads for processing the target task, so that the stability of the distributed scheduling system can be well protected.
In a distributed scheduling system, a task is usually processed by a plurality of execution nodes. In the process of step S3, when a new thread needs to be added to process the target task, the server may select any one or more execution nodes to start a new thread for processing the target task, and when a thread for processing the target task needs to be decreased, select any one or more execution nodes to stop the thread for processing the target task.
Preferably, the server adjusts the threads of the processing target task based on the principle of equal distribution during the process of executing step S3, so as to balance the load of each execution node in the distributed scheduling system.
Referring to fig. 5, fig. 5 is a flowchart of a control method of another distributed scheduling system disclosed in the present application. The control method is applied to a server of a distributed scheduling system, and specifically comprises the following steps:
step S1: and acquiring the number of the messages which are not processed by the target task according to a preset time interval.
Step S2: and determining the target thread number corresponding to the target task according to the number of the unprocessed messages.
Step S301: and under the condition that the number of target threads corresponding to the target task is larger than the number of threads for processing the target task currently, executing one or more times of first operation until the number of threads for processing the target task reaches the target thread number.
Wherein the first operation comprises: searching a first target node in a plurality of execution nodes for processing a target task, wherein the first target node is one execution node with the least number of threads for processing the target task, which is started from the plurality of execution nodes; and controlling the first target node to start a new thread to process the target task.
That is, after determining the target thread number corresponding to the target task, if the number of threads currently processing the target task is smaller than the target thread number corresponding to the target task, searching for an execution node with the least number of threads for starting processing the target task from among the plurality of execution nodes for processing the target task, controlling the execution node to start a new thread for processing the target task, and repeating the first operation until the number of threads for processing the target task increases to the target thread number corresponding to the target task.
Optionally, the first target node further satisfies: after the first target node starts a new thread for the target task, the number of all threads started by the first target node does not exceed a preset thread upper limit value.
Optionally, the first operation may further include: and searching a third target node in a plurality of execution nodes which do not start threads aiming at the target task, and controlling the third target node to start a new thread to process the target task.
That is, in addition to searching the first target node among the multiple executing nodes currently processing the target task, and controlling the first target node to start a new thread to process the target task, a new executing node may be added to process the target task.
Step S302: and under the condition that the number of the target threads corresponding to the target task is less than the number of the threads for processing the target task currently, executing one or more times of second operation until the number of the threads for processing the target task reaches the target thread number.
Wherein the second operation comprises: searching a second target node in a plurality of execution nodes for processing the target task, wherein the second target node is one execution node which is started to process the target task and has the maximum number of threads; and controlling the second target node to close a thread for processing the target task.
That is, after determining the target thread number corresponding to the target task, if the number of threads currently processing the target task is greater than the target thread number corresponding to the target task, searching for an execution node with the largest number of threads for processing the target task from among the plurality of execution nodes for processing the target task, controlling the execution node to close one thread for processing the target task, and repeating the second operation until the number of threads for processing the target task is reduced to the target thread number corresponding to the target task.
In the control method shown in fig. 5 of the present application, after determining a target thread number corresponding to a target task according to a number of unprocessed messages of the target task, if a number of threads currently processing the target task is smaller than the target thread number corresponding to the target task, then searching for an execution node with a minimum number of threads for starting processing the target task from among a plurality of execution nodes for processing the target task, controlling the execution node to start a new thread for processing the target task, repeating the above operations until the number of threads for processing the target task is increased to the target thread number corresponding to the target task, if the number of threads currently processing the target task is larger than the target thread number corresponding to the target task, then searching for an execution node with a maximum number of threads for starting processing the target task from among the plurality of execution nodes for processing the target task, and controlling the execution node to close a thread for processing the target task, and repeating the operation until the number of the threads for processing the target task is reduced to the target thread number corresponding to the target task. It can be seen that, based on the control method shown in fig. 5 of the present application, the server adjusts the threads for processing the target task according to the principle of average allocation, which makes the load of each execution node in the distributed scheduling system more balanced.
The application discloses a control method of the distributed scheduling system, and correspondingly, the application also discloses a control device of the distributed scheduling system, wherein the control device is arranged in a server of the distributed scheduling system. The above description about the control method and the following description about the control device may be referred to each other.
Referring to fig. 6, fig. 6 is a schematic structural diagram of a control device of a distributed scheduling system disclosed in the present application, which specifically includes:
a message quantity obtaining unit 100, configured to obtain the quantity of unprocessed messages of the target task according to a preset time interval;
a target thread number determining unit 200, configured to determine a target thread number corresponding to the target task according to the number of unprocessed messages, where the number of unprocessed messages and the target thread number corresponding to the target task are in a positive correlation;
the thread processing unit 300 is configured to, when the number of threads currently processing the target task is different from the target number of threads corresponding to the target task, adjust the threads processing the target task such that the number of threads processing the target task is the target number of threads.
Based on the control device disclosed by the application, when the number of unprocessed messages of the target task is large due to a service peak, the server automatically allocates more threads to process the target task, so that the problem of overstocked message queues can be avoided, and when the number of unprocessed messages of the target task is small, the server automatically reduces the threads for processing the target task, so that more resources in the execution node are released for processing other tasks.
As an example, the target thread number determination unit 200 includes:
the numerical value comparison subunit is used for comparing the number of the unprocessed messages with a first threshold value and a second threshold value, wherein the first threshold value and the second threshold value are positive numbers, and the first threshold value is greater than or equal to the second threshold value;
the first processing subunit is used for taking the sum of the number of threads of the current processing target task and the first numerical value as the target thread number corresponding to the target task under the condition that the number of unprocessed messages is greater than a first threshold value;
and the second processing subunit is used for taking the difference value between the number of threads of the current processing target task and the first numerical value as the target thread number corresponding to the target task under the condition that the number of the unprocessed messages is smaller than the second threshold value.
As another example, the target thread number determination unit 200 includes:
the numerical value comparison subunit is used for comparing the number of the unprocessed messages with a first threshold value and a second threshold value, wherein the first threshold value and the second threshold value are positive numbers, and the first threshold value is greater than or equal to the second threshold value;
the third processing subunit is used for calculating the sum of the number of threads of the current processing target task and a first numerical value under the condition that the number of unprocessed messages is greater than a first threshold value, taking the sum as the target thread number corresponding to the target task if the sum is less than or equal to the preset maximum thread number, and taking the maximum thread number as the target thread number corresponding to the target task if the sum is greater than the maximum thread number;
and the fourth processing subunit is used for calculating the difference value between the number of the threads of the current processing task and the first numerical value under the condition that the number of the unprocessed messages is smaller than the second threshold value, taking the difference value as the target thread number corresponding to the target task if the difference value is larger than or equal to the preset minimum thread number, and taking the minimum thread number as the target thread number corresponding to the target task if the difference value is smaller than the minimum thread number.
In a distributed scheduling system, a task is usually processed by a plurality of execution nodes. When a new thread needs to be added to process a target task, the thread processing unit 300 may select any one or more execution nodes to start a new thread for processing the target task, and when a thread for processing the target task needs to be reduced, the thread processing unit 300 may select any one or more execution nodes to stop the thread for processing the target task.
In a preferred embodiment, the thread processing unit 300 adjusts the threads of the processing target task based on the principle of average distribution to balance the load of each execution node in the distributed scheduling system.
As one example, thread processing unit 300 includes a first thread adjustment subunit and a second thread adjustment subunit. Wherein:
the first thread adjusting subunit is used for executing one or more times of first operation under the condition that the number of target threads corresponding to the target task is greater than the number of threads for processing the target task at present until the number of threads for processing the target task reaches the target number of threads; wherein the first operation comprises: searching a first target node in a plurality of execution nodes for processing a target task, wherein the first target node is one execution node with the least number of threads for processing the target task, which is started from the plurality of execution nodes; and controlling the first target node to start a new thread to process the target task.
The second thread adjusting subunit is used for executing one or more times of second operation under the condition that the number of the target threads corresponding to the target task is smaller than the number of the threads for processing the target task at present until the number of the threads for processing the target task reaches the target number of the threads; wherein the second operation comprises: searching a second target node in a plurality of execution nodes for processing the target task, wherein the second target node is one execution node which is started to process the target task and has the maximum number of threads; and controlling the second target node to close a thread for processing the target task.
The control method disclosed above in the present application may be implemented by a control system of a distributed scheduling system. Referring to fig. 7, fig. 7 is a schematic structural diagram of a control system of a distributed scheduling system disclosed in the present application. The control system includes a configuration center 10, a server 20, and a manager 30.
The configuration center 10 is configured to: determining a target task needing to be monitored, determining a queue node corresponding to the target task, determining a queue Key of the queue node, and determining a first threshold value and a second threshold value. The queue node is used for storing the messages which are not processed by the target task, and the number of the messages which are not processed by the target task can be read based on the queue Key.
The server 20 is configured to: reading the number of messages which are not processed by the target task according to a preset time interval (such as 3 seconds), and comparing the number of messages which are not processed by the target task with a first threshold value and a second threshold value; if the number of the messages which are not processed by the target task is larger than a first threshold value, adding a first numerical value (such as 1) on the basis of the number of the threads which process the target task at present, if the sum value is smaller than or equal to the preset maximum number of the threads, taking the sum value as the number of the target threads, if the sum value is larger than the preset maximum number of the threads, taking the maximum number of the threads as the number of the target threads, and sending a request to the manager 30, wherein the request carries the number of the target threads; if the number of the messages which are not processed by the target task is smaller than the second threshold, a first numerical value (such as 1) is subtracted on the basis of the number of the threads which process the target task at present, if the difference is larger than or equal to a preset minimum number of threads, the difference is used as the number of the target threads, if the difference is smaller than the preset minimum number of threads, the minimum number of threads is used as the number of the target threads, and a request is sent to the manager 30, wherein the request carries the number of the target threads.
The manager 30 is configured to: receiving a request sent by the server 20, and if the number of target threads is greater than the number of threads currently processing the target task, executing one or more times of first operation until the number of threads processing the target task is the same as the number of target threads; and if the number of the target threads is less than the number of the threads for processing the target task at present, executing one or more times of second operation until the number of the threads for processing the target task is the same as the number of the target threads.
With regard to the first operation and the second operation, reference may be made to the description hereinbefore.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A control method of a distributed scheduling system is applied to a server of the distributed scheduling system, and is characterized by comprising the following steps:
acquiring the number of unprocessed messages of the target task according to a preset time interval;
determining a target thread number corresponding to the target task according to the number of the unprocessed messages, wherein the number of the unprocessed messages and the target thread number corresponding to the target task are in positive correlation;
and under the condition that the number of threads for processing the target task currently is different from the target number of threads corresponding to the target task, adjusting the threads for processing the target task to enable the number of the threads for processing the target task to be the target number of threads.
2. The control method according to claim 1, wherein the determining a target thread number corresponding to the target task according to the number of the unprocessed messages comprises:
comparing the number of unprocessed messages with a first threshold and a second threshold, wherein the first threshold and the second threshold are positive numbers, and the first threshold is greater than or equal to the second threshold;
if the number of the unprocessed messages is larger than the first threshold value, taking the sum of the number of the threads currently processing the target task and a first numerical value as the target thread number corresponding to the target task;
and if the number of the unprocessed messages is smaller than the second threshold value, taking the difference value between the number of the threads currently processing the target task and the first numerical value as the target thread number corresponding to the target task.
3. The control method according to claim 1, wherein the determining a target thread number corresponding to the target task according to the number of the unprocessed messages comprises:
comparing the number of unprocessed messages with a first threshold and a second threshold, wherein the first threshold and the second threshold are positive numbers, and the first threshold is greater than or equal to the second threshold;
if the number of the unprocessed messages is larger than the first threshold value, calculating a sum of the number of threads currently processing the target task and a first numerical value, if the sum is smaller than or equal to a preset maximum number of threads, taking the sum as the number of the target threads corresponding to the target task, and if the sum is larger than the maximum number of threads, taking the maximum number of threads as the number of the target threads corresponding to the target task;
if the number of the unprocessed messages is smaller than the second threshold value, calculating a difference value between the number of threads currently processing the task and a first numerical value, if the difference value is larger than or equal to a preset minimum number of threads, taking the difference value as a target number of threads corresponding to the target task, and if the difference value is smaller than the minimum number of threads, taking the minimum number of threads as a target number of threads corresponding to the target task.
4. The control method according to claim 1, 2 or 3, wherein the adjusting the threads of the target task so that the number of threads processing the target task is the target number of threads comprises:
executing one or more times of first operation under the condition that the number of target threads corresponding to the target task is larger than the number of threads for processing the target task at present until the number of threads for processing the target task reaches the target number of threads;
wherein the first operation comprises: searching a first target node in a plurality of execution nodes for processing the target task, wherein the first target node is one execution node with the least number of threads for processing the target task, which is started from the plurality of execution nodes; and controlling the first target node to start a new thread to process the target task.
5. The control method according to claim 4, wherein the adjusting the threads of the target task so that the number of threads processing the target task is the target number of threads, further comprises:
executing one or more times of second operation under the condition that the number of target threads corresponding to the target task is smaller than the number of threads for processing the target task at present until the number of threads for processing the target task reaches the target number of threads;
wherein the second operation comprises: searching a second target node in a plurality of execution nodes for processing the target task, wherein the second target node is one execution node with the largest number of threads for processing the target task, which is started from the plurality of execution nodes; and controlling the second target node to close a thread for processing the target task.
6. A control apparatus for a distributed scheduling system, the control apparatus being applied to a server of the distributed scheduling system, the control apparatus comprising:
the message quantity acquiring unit is used for acquiring the quantity of unprocessed messages of the target task according to a preset time interval;
a target thread number determining unit, configured to determine a target thread number corresponding to the target task according to the number of the unprocessed messages, where the number of the unprocessed messages and the target thread number corresponding to the target task are in a positive correlation;
and the thread processing unit is used for adjusting the threads for processing the target task under the condition that the number of the threads for processing the target task at present is different from the target thread number corresponding to the target task, so that the number of the threads for processing the target task is the target thread number.
7. The control device according to claim 6, wherein the target thread number determination unit includes:
a numerical value comparison subunit, configured to compare the number of unprocessed messages with a first threshold and a second threshold, where the first threshold and the second threshold are positive numbers, and the first threshold is greater than or equal to the second threshold;
a first processing subunit, configured to, when the number of the unprocessed messages is greater than the first threshold, take a sum of a first numerical value and the number of threads currently processing the target task as a target thread number corresponding to the target task;
and the second processing subunit is configured to, when the number of the unprocessed messages is smaller than the second threshold, take a difference between the number of threads currently processing the target task and the first value as a target thread number corresponding to the target task.
8. The control device according to claim 6, wherein the target thread number determination unit includes:
a numerical value comparison subunit, configured to compare the number of unprocessed messages with a first threshold and a second threshold, where the first threshold and the second threshold are positive numbers, and the first threshold is greater than or equal to the second threshold;
a third processing subunit, configured to, when the number of unprocessed messages is greater than the first threshold, calculate a sum of a first number and the number of threads currently processing the target task, if the sum is less than or equal to a preset maximum number of threads, use the sum as the target number of threads corresponding to the target task, and if the sum is greater than the maximum number of threads, use the maximum number of threads as the target number of threads corresponding to the target task;
and the fourth processing subunit is configured to, when the number of the unprocessed messages is smaller than the second threshold, calculate a difference between the number of threads currently processing the task and a first value, if the difference is greater than or equal to a preset minimum number of threads, take the difference as a target number of threads corresponding to the target task, and if the difference is smaller than the minimum number of threads, take the minimum number of threads as the target number of threads corresponding to the target task.
9. The control apparatus according to claim 6, 7 or 8, wherein the thread processing unit comprises:
the first thread adjusting subunit is configured to, when the number of target threads corresponding to the target task is greater than the number of threads currently processing the target task, perform one or more first operations until the number of threads processing the target task reaches the target number of threads; wherein the first operation comprises: searching a first target node in a plurality of execution nodes for processing the target task, wherein the first target node is one execution node with the least number of threads for processing the target task, which is started from the plurality of execution nodes; and controlling the first target node to start a new thread to process the target task.
10. The control apparatus according to claim 9, wherein the thread processing unit further comprises:
a second thread adjusting subunit, configured to, when a number of target threads corresponding to the target task is smaller than a number of threads currently processing the target task, perform one or more second operations until the number of threads processing the target task reaches the target number of threads; wherein the second operation comprises: searching a second target node in a plurality of execution nodes for processing the target task, wherein the second target node is one execution node with the largest number of threads for processing the target task, which is started from the plurality of execution nodes; and controlling the second target node to close a thread for processing the target task.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911391951.5A CN111078386A (en) | 2019-12-30 | 2019-12-30 | Control method and control device of distributed scheduling system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911391951.5A CN111078386A (en) | 2019-12-30 | 2019-12-30 | Control method and control device of distributed scheduling system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111078386A true CN111078386A (en) | 2020-04-28 |
Family
ID=70319336
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911391951.5A Pending CN111078386A (en) | 2019-12-30 | 2019-12-30 | Control method and control device of distributed scheduling system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111078386A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112463314A (en) * | 2020-11-06 | 2021-03-09 | 贝壳技术有限公司 | Asynchronous queue processing method and device, electronic equipment and storage medium |
| CN113590925A (en) * | 2020-04-30 | 2021-11-02 | 中国移动通信集团北京有限公司 | User determination method, device, equipment and computer storage medium |
| CN113778617A (en) * | 2021-08-10 | 2021-12-10 | 万翼科技有限公司 | Container horizontal expansion method and device, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105306552A (en) * | 2015-09-30 | 2016-02-03 | 中国科学院计算技术研究所 | Consumption equilibrium method and system based on message queues |
| CN105389208A (en) * | 2015-11-10 | 2016-03-09 | 中国建设银行股份有限公司 | Job processing method and apparatus |
| US9354931B1 (en) * | 2015-05-14 | 2016-05-31 | Tmaxsoft, Co., Ltd. | Method, server and computer-readable recording medium for managing size of thread pool dynamically |
| CN108600403A (en) * | 2018-01-24 | 2018-09-28 | 北京奇艺世纪科技有限公司 | A kind of DNS thread management methods, device and server |
| CN109492024A (en) * | 2018-10-26 | 2019-03-19 | 平安科技（深圳）有限公司 | Data processing method, device, computer equipment and storage medium |
-
2019
- 2019-12-30 CN CN201911391951.5A patent/CN111078386A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9354931B1 (en) * | 2015-05-14 | 2016-05-31 | Tmaxsoft, Co., Ltd. | Method, server and computer-readable recording medium for managing size of thread pool dynamically |
| CN105306552A (en) * | 2015-09-30 | 2016-02-03 | 中国科学院计算技术研究所 | Consumption equilibrium method and system based on message queues |
| CN105389208A (en) * | 2015-11-10 | 2016-03-09 | 中国建设银行股份有限公司 | Job processing method and apparatus |
| CN108600403A (en) * | 2018-01-24 | 2018-09-28 | 北京奇艺世纪科技有限公司 | A kind of DNS thread management methods, device and server |
| CN109492024A (en) * | 2018-10-26 | 2019-03-19 | 平安科技（深圳）有限公司 | Data processing method, device, computer equipment and storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113590925A (en) * | 2020-04-30 | 2021-11-02 | 中国移动通信集团北京有限公司 | User determination method, device, equipment and computer storage medium |
| CN112463314A (en) * | 2020-11-06 | 2021-03-09 | 贝壳技术有限公司 | Asynchronous queue processing method and device, electronic equipment and storage medium |
| CN113778617A (en) * | 2021-08-10 | 2021-12-10 | 万翼科技有限公司 | Container horizontal expansion method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111078386A (en) | Control method and control device of distributed scheduling system | |
| CN107800768B (en) | Open platform control method and system | |
| WO2010100859A1 (en) | Distributed system | |
| US9405588B2 (en) | Cloud resource allocation system and method | |
| CN112165436B (en) | Flow control method, device and system | |
| WO2023050901A1 (en) | Load balancing method and apparatus, device, computer storage medium and program | |
| CN108023759B (en) | Self-adaptive resource scheduling method and device | |
| CN107819797B (en) | Access request processing method and device | |
| CN110933178B (en) | Method for adjusting node configuration in cluster system and server | |
| CN110933136A (en) | Service node selection method, device, equipment and readable storage medium | |
| CN108664321B (en) | System resource allocation adjusting method and device | |
| CN111291252B (en) | Method and device for adjusting query rate per second, electronic equipment and storage medium | |
| CN115277577B (en) | Data processing method, apparatus, computer device, and computer readable storage medium | |
| CN110674102A (en) | Method, system, device and storage medium for distributed file system directory quota | |
| CN112416594A (en) | Micro-service distribution method, electronic equipment and computer storage medium | |
| EP3672203A1 (en) | Distribution method for distributed data computing, device, server and storage medium | |
| CN106021026B (en) | Backup method and device | |
| CN105634932B (en) | Message pushing method, device, system and computer readable storage medium | |
| CN112035249A (en) | Application program network proportion adjusting method and device and storage medium | |
| CN111092959B (en) | Request processing method, system and related device for servers in cluster | |
| CN111683133B (en) | Service flow limiting method based on micro-service architecture and related device | |
| CN111813546B (en) | Resource allocation method, system and related device for multi-network connection application | |
| CN111193760B (en) | Information sending method, device and storage medium | |
| CN114466365A (en) | Spectrum resource acquisition method, spectrum resource acquisition device and computer readable storage medium | |
| CN114143263A (en) | Method, device and medium for limiting current of user request |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20231121 Address after: 200335 room 205, floor 2, building 9, No. 365, Linhong Road, Changning District, Shanghai Applicant after: Shanghai iqiyi New Media Technology Co.,Ltd. Address before: 10 / F and 11 / F, iqiyi innovation building, No.2 Beiyi street, Haidian District, Beijing 100080 Applicant before: BEIJING QIYI CENTURY SCIENCE & TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right |