CN117194035A - Method, device, equipment and medium for adjusting concurrency number of batch processing tasks - Google Patents
Method, device, equipment and medium for adjusting concurrency number of batch processing tasks Download PDFInfo
- Publication number
- CN117194035A CN117194035A CN202311218834.5A CN202311218834A CN117194035A CN 117194035 A CN117194035 A CN 117194035A CN 202311218834 A CN202311218834 A CN 202311218834A CN 117194035 A CN117194035 A CN 117194035A
- Authority
- CN
- China
- Prior art keywords
- servers
- utilization rate
- server
- resource
- distributed system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 189
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000004590 computer program Methods 0.000 claims description 13
- 238000004891 communication Methods 0.000 claims description 8
- 238000012935 Averaging Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 230000008859 change Effects 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000026676 system process Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Multi Processors (AREA)
Abstract
The application relates to the technical field of task scheduling, in particular to a method, a device, equipment and a medium for adjusting the concurrency number of batch processing tasks. The method is applied to any one server in a distributed system comprising a plurality of servers, and the plurality of servers are combined to complete batch processing tasks of the distributed system, and the method comprises the following steps: acquiring feedback information of a plurality of servers; the feedback information of each server in the plurality of servers is used for indicating the resource utilization rate of each server; determining an optimal concurrency number of the distributed system in a set time period based on feedback information of a plurality of servers; and in the set time period, distributing the sub-processing tasks with the number corresponding to the optimal concurrency number to a plurality of servers in the plurality of sub-processing tasks included in the batch processing tasks.
Description
Technical Field
The application relates to the technical field of task scheduling, in particular to a method, a device, equipment and a medium for adjusting the concurrency number of batch processing tasks.
Background
Batch processing tasks are large-scale processing tasks for objects such as data and files, and generally require a long processing time and use a large amount of processing resources. In order to improve the processing efficiency of a batch processing task, the batch processing task is generally divided into a plurality of sub-processing tasks to be executed concurrently. In the prior art, a distributed system processes a batch processing task by using a fixed concurrency number, but the resource use condition of the distributed system is dynamically changed. At some point, the use of a fixed number of concurrency may exceed the load capacity of the distributed system, rather increasing the time spent on batch processing tasks; at another time, using a fixed number of concurrency may underutilize the computing resources of the distributed system. Therefore, how to determine the adjustment concurrency number and improve the efficiency of the distributed system in processing batch processing tasks becomes a problem to be solved.
Disclosure of Invention
The embodiment of the application provides a method, a device, equipment and a medium for adjusting the concurrency number of batch processing tasks, which are used for improving the efficiency of the distributed system for processing the batch processing tasks.
In a first aspect, an embodiment of the present application provides a method for adjusting a concurrency number of a batch processing task, where the method is applied to any one server in a distributed system including a plurality of servers, and the plurality of servers combine to complete the batch processing task of the distributed system, and the method includes:
acquiring feedback information of the servers; the feedback information of each server in the plurality of servers is used for indicating the resource utilization rate of each server;
determining an optimal concurrency number of the distributed system in a set time period based on feedback information of the plurality of servers;
and in the set time period, distributing the sub-processing tasks with the number corresponding to the optimal concurrency number to the servers in the sub-processing tasks included in the batch processing task.
In the scheme, any server in the distributed system determines the optimal concurrency number of the distributed system in a set time period according to the feedback information of the plurality of servers, and sub-processing tasks with the number corresponding to the optimal concurrency number are distributed to the plurality of servers, so that the computing resources of the distributed system can be fully utilized within the load capacity range of the distributed system, and the efficiency of the distributed system in processing batch processing tasks is improved.
Optionally, the determining, based on feedback information of the plurality of servers, an optimal concurrency number of the distributed system in a set period of time includes: according to a preset time interval, periodically determining the optimal concurrency number of the distributed system in the current period based on feedback information of the plurality of servers in the previous period; the duration of the preset time interval is related to the processing duration of each sub-processing task in the plurality of sub-processing tasks.
According to the method, the optimal concurrency number can be changed according to the change of the resource use condition of the distributed system, so that the optimal concurrency number is determined periodically according to the feedback information, and the flexibility and the reliability of the scheme are improved.
Optionally, the determining, based on feedback information of the plurality of servers, an optimal concurrency number of the distributed system in a set period of time includes: averaging the utilization rate of the same resource of the plurality of servers to obtain the utilization rate average value of at least one resource of the plurality of servers; determining a representative resource with the maximum average value of the resource utilization rate from the at least one resource; if the average value of the utilization rate of the representative resource is not equal to a threshold value, determining a functional relation between the average value of the utilization rate of the representative resource and the concurrency number of the distributed system according to the average value of the utilization rate of the representative resource in at least one period and the concurrency number; and determining the concurrency number when the average value of the utilization rate of the representative resource is a threshold value based on the functional relation, and taking the concurrency number when the average value of the utilization rate of the representative resource is the threshold value as the optimal concurrency number of the current period.
In this way, from at least one resource related to the plurality of servers, the representative resource with the largest average value of the utilization rate of the resource is determined, if the average value of the utilization rate of the representative resource is not equal to the threshold value, the concurrency number when the average value of the utilization rate of the representative resource is the threshold value is determined according to the functional relationship between the average value of the utilization rate of the representative resource and the concurrency number of the distributed system, and the concurrency number is used as the optimal concurrency number. The average value of the utilization rate of the representative resources is the largest, and the concurrency number of the representative resources is the optimal concurrency number when the average value of the utilization rate of the representative resources is the threshold value, so that the average value of the utilization rates of the rest resources can be ensured to be smaller than the threshold value, the determined optimal concurrency number does not exceed the load capacity of the distributed system, and the computing resources of the distributed system can be fully utilized.
Optionally, the method further comprises: and if the average value of the utilization rate of the representative resources is equal to the threshold value, the concurrency number of the last period of the distributed system is used as the optimal concurrency number of the current period.
By the method, when the average value of the utilization rate of the representative resources is equal to the threshold value, the concurrency number of the last period of the distributed system is the optimal concurrency number of the current period, and the concurrency number is not required to be adjusted in the current period, so that the scheme integrity is improved.
Optionally, the distributing, among the plurality of sub-processing tasks included in the batch processing task, a number of sub-processing tasks corresponding to the optimal concurrency number to the plurality of servers includes: the priority ranking is carried out on the plurality of servers according to the order of the utilization rate of the representative resources of the plurality of servers from big to small; if the optimal concurrency number of the current period is greater than that of the previous period, sequentially increasing the number of sub-processing tasks allocated to the server from the server with the smallest priority according to the order of the priorities from small to large; or if the optimal concurrency number of the current period is smaller than that of the previous period, sequentially reducing the number of sub-processing tasks allocated to the server from the server with the largest priority according to the order of the priorities from the largest priority.
In the mode, the optimal concurrency number of the current period is larger than that of the previous period, and the number of the sub-processing tasks allocated to the server is sequentially increased from the server with the minimum representative resource utilization rate, so that the load of the server with the higher representative resource utilization rate is avoided being exceeded, and the computing capacity of the server with the lower representative resource utilization rate is fully utilized; the optimal concurrency number of the current period is smaller than that of the previous period, and the number of sub-processing tasks allocated to the server is sequentially reduced from the server with the maximum representative resource utilization rate, so that the load of the server with the higher representative resource utilization rate is reduced, and the stability and reliability of the distributed system are improved.
In a second aspect, an embodiment of the present application provides a concurrency number adjustment device for batch processing tasks, where the device is applied to any one server in a distributed system including a plurality of servers, where the plurality of servers combine to complete batch processing tasks of the distributed system, and the device includes a module/unit/technical means for performing the method in the first aspect or any optional implementation manner of the first aspect.
Illustratively, the apparatus may include:
the acquisition module is used for acquiring feedback information of the servers; the feedback information of each server in the plurality of servers is used for indicating the resource utilization rate of each server;
the processing module is used for determining the optimal concurrency number of the distributed system in a set time period based on the feedback information of the servers; and in the set time period, distributing the sub-processing tasks with the number corresponding to the optimal concurrency number to the servers in the sub-processing tasks included in the batch processing task.
Optionally, when determining the optimal concurrency number of the distributed system in the set time period based on the feedback information of the plurality of servers, the processing module is specifically configured to: according to a preset time interval, periodically determining the optimal concurrency number of the distributed system in the current period based on feedback information of the plurality of servers in the previous period; the duration of the preset time interval is related to the processing duration of each sub-processing task in the plurality of sub-processing tasks.
Optionally, when determining the optimal concurrency number of the distributed system in the set time period based on the feedback information of the plurality of servers, the processing module is specifically configured to: averaging the utilization rate of the same resource of the plurality of servers to obtain the utilization rate average value of at least one resource of the plurality of servers; determining a representative resource with the maximum average value of the resource utilization rate from the at least one resource; if the average value of the utilization rate of the representative resource is not equal to a threshold value, determining a functional relation between the average value of the utilization rate of the representative resource and the concurrency number of the distributed system according to the average value of the utilization rate of the representative resource in at least one period and the concurrency number; and determining the concurrency number when the average value of the utilization rate of the representative resource is a threshold value based on the functional relation, and taking the concurrency number when the average value of the utilization rate of the representative resource is the threshold value as the optimal concurrency number of the current period.
Optionally, the processing module is further configured to: and if the average value of the utilization rate of the representative resources is equal to the threshold value, the concurrency number of the last period of the distributed system is used as the optimal concurrency number of the current period.
Optionally, the processing module is specifically configured to, when distributing, among the plurality of sub-processing tasks included in the batch processing task, a number of sub-processing tasks corresponding to the optimal concurrency number to the plurality of servers: the priority ranking is carried out on the plurality of servers according to the order of the utilization rate of the representative resources of the plurality of servers from big to small; if the optimal concurrency number of the current period is greater than that of the previous period, sequentially increasing the number of sub-processing tasks allocated to the server from the server with the smallest priority according to the order of the priorities from small to large; or if the optimal concurrency number of the current period is smaller than that of the previous period, sequentially reducing the number of sub-processing tasks allocated to the server from the server with the largest priority according to the order of the priorities from the largest priority.
In a third aspect, the present application provides an electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor, a communication interface; wherein the memory stores instructions executable by the at least one processor, which, by executing the instructions stored by the memory, cause the electronic device to perform the method as described in the first aspect or any of the alternative implementations of the first aspect via a communication interface.
In a fourth aspect, the present application provides a computer readable storage medium for storing instructions that, when executed, cause a method as described in the first aspect or any of the alternative embodiments of the first aspect to be carried out.
In a fifth aspect, the present application provides a computer program product comprising: computer program code which, when run on a computer, causes the computer to perform the method as described in the first aspect or any of the alternative embodiments of the first aspect.
The technical effects or advantages of one or more technical solutions provided in the second, third, fourth, and fifth aspects of the embodiments of the present application may be correspondingly explained by the technical effects or advantages of the corresponding one or more technical solutions provided in the first aspect, which are not described herein.
Drawings
FIG. 1 is a flow chart of a method for adjusting concurrency of batch processing tasks according to an embodiment of the present application;
FIG. 2 is a flow chart of a concurrent number change according to an embodiment of the present application;
FIG. 3 is a block diagram of a device for adjusting concurrency of batch processing tasks according to an embodiment of the present application;
fig. 4 is a block diagram of an electronic device according to an embodiment of the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application. Embodiments of the application and features of the embodiments may be combined with one another arbitrarily without conflict. Also, while a logical order of illustration is depicted in the flowchart, in some cases the steps shown or described may be performed in a different order than presented.
The terms first and second in the description and claims of the application and in the above-mentioned figures are used for distinguishing between different objects and not for describing a particular sequential order. Furthermore, the term "include" and any variations thereof is intended to cover non-exclusive protection. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus. The term "plurality" in the present application may mean at least two, for example, two, three or more, and embodiments of the present application are not limited.
The term "and/or" in the embodiment of the present application is merely an association relationship describing the association object, and indicates that three relationships may exist, for example, a and/or B may indicate: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.
In the technical scheme of the application, the data is collected, transmitted, used and the like, and all meet the requirements of national relevant laws and regulations.
In order to facilitate understanding of the scheme of the embodiment of the present application, a possible application scenario of the embodiment of the present application is described below.
The distributed system comprises a plurality of servers, and the plurality of servers are combined to complete batch processing tasks of the distributed system. In order to improve the processing efficiency of a batch processing task, the batch processing task is generally divided into a plurality of sub-processing tasks to be executed concurrently. The concurrency number of the distributed system refers to the number of sub-processing tasks processed by the distributed system at the same time.
Because the resource use condition of the distributed system is dynamically changed, at a certain moment, the use of a fixed concurrency number may exceed the load capacity of the distributed system, and instead, the time consumed by batch processing tasks is increased; at another time, using a fixed number of concurrency may underutilize the computing resources of the distributed system. Therefore, how to determine the adjustment concurrency number and improve the efficiency of the distributed system in processing batch processing tasks becomes a problem to be solved.
In view of this, the technical solution of the embodiment of the present application is provided for improving the efficiency of the distributed system in processing batch processing tasks.
In view of the above, the following describes in detail the method for adjusting the concurrency number of batch processing tasks provided by the application with reference to the attached drawings. It should be noted that the method can be applied to a distributed system including a plurality of servers, where the plurality of servers combine to complete batch tasks of the distributed system.
Referring to fig. 1, a flowchart of a method for adjusting concurrency of batch processing tasks is provided in an embodiment of the present application. The method may be performed by any server in a distributed system, the method comprising:
s101: and acquiring feedback information of a plurality of servers.
Any server receives feedback information reported by other servers and acquires feedback information generated by the server. The feedback information of each server is used to indicate the resource usage of each server. Illustratively, the resources of each server include one or more of the following: CPU (Central Processing Unit ), disk I/O (Input/Output), memory.
Optionally, if the server type is a database server, the feedback information of each database server is further used to indicate the average execution time of the SQL statement.
S102: and determining the optimal concurrency number of the distributed system in the set time period based on the feedback information of the plurality of servers.
In a possible implementation manner, according to a preset time interval, the optimal concurrency number of the distributed system in the current period is determined periodically based on feedback information of a plurality of servers in the previous period.
Optionally, the duration of the preset time interval is related to the processing duration of each of the plurality of sub-processing tasks, e.g., the longer the processing duration of each sub-processing task, the longer the preset time interval. Alternatively, the duration of the preset time interval is related to the size of each sub-processing task, for example, the shorter the number of code lines of each sub-processing task, the shorter the preset time interval.
It should be understood that the foregoing is merely exemplary and not limiting, and the preset time interval may be defined according to actual needs, which is not limited by the present application.
According to the method, the optimal concurrency number can be changed according to the change of the resource use condition of the distributed system, so that the optimal concurrency number is determined periodically according to the feedback information, and the flexibility and the reliability of the scheme are improved.
The method for determining the optimal concurrency number is described below.
In one possible implementation manner, the average value of the utilization rate of the same resource of the plurality of servers is obtained, the average value of the utilization rate of at least one resource of the plurality of servers is obtained, and the optimal concurrency number is determined according to the average value of the utilization rate of at least one resource.
Specifically, a representative resource with the largest average value of the resource utilization rate is determined from at least one resource. If the average value of the utilization rate of the representative resource is not equal to the threshold value, determining a function relation between the average value of the utilization rate of the representative resource and the concurrency number of the distributed system according to the average value of the utilization rate of the representative resource in at least one latest period and the concurrency number; based on the functional relation, determining the concurrency number when the average value of the utilization rate of the representative resource is the threshold value, and taking the concurrency number when the average value of the utilization rate of the representative resource is the threshold value as the optimal concurrency number of the current period. And if the average value of the utilization rate of the representative resources is equal to the threshold value, the concurrency number of the last period of the distributed system is used as the optimal concurrency number of the current period.
Alternatively, different representative resources may correspond to different thresholds, for example, 70% for a representative resource being a CPU and 75% for a representative resource being a memory, or different representative resources may correspond to the same threshold, for example, 80% for a representative resource being a CPU or for a representative resource being a memory.
The distributed system comprises three servers, namely a server 1, a server 2 and a server 3, and at least one resource comprises three resources, namely a CPU, a memory and an I/O. The feedback information of the server 1 indicates that the CPU utilization of the server 1 is 70%, the memory utilization is 40%, and the I/O utilization is 80%; the feedback information of the server 2 indicates that the CPU utilization rate of the server 1 is 65%, the memory utilization rate is 50%, and the IO utilization rate is 90%; the feedback information of the server 3 indicates that the CPU usage of the server 1 is 60%, the memory usage is 30%, and the I/O usage is 85%. The average value of the utilization rate of the same resource of the three servers is calculated, and the average value of the utilization rate of the CPU is 65%, the average value of the utilization rate of the memory is 40%, and the average value of the utilization rate of the I/O is 85%. Wherein the average value of the utilization rate of the I/O is the largest, i.e. the I/O is the representative resource. In the last two periods, the concurrency number is 10 when the average value of the resource utilization rate of the I/O is 60%, 12 when the average value of the resource utilization rate of the I/O is 70%, and 1 when the function relationship between the average value of the resource utilization rate and the concurrency number of the distributed system is represented as a one-time function, the average value of the resource utilization rate of the I/O is increased by 5%. Let the threshold value for representing the resource as I/O be 80%, the optimal concurrency number corresponding to the threshold value is 14.
It will be appreciated that the number of cycles of at least one cycle, at least one resource, and the specific type of functional relationship may all be specified according to actual requirements, and the present application is not limited.
In this way, from at least one resource related to the plurality of servers, the representative resource with the largest average value of the utilization rate of the resource is determined, if the average value of the utilization rate of the representative resource is not equal to the threshold value, the concurrency number when the average value of the utilization rate of the representative resource is the threshold value is determined according to the functional relationship between the average value of the utilization rate of the representative resource and the concurrency number of the distributed system, and the concurrency number is used as the optimal concurrency number. The average value of the utilization rate of the representative resources is the largest, and the concurrency number of the representative resources is the optimal concurrency number when the average value of the utilization rate of the representative resources is the threshold value, so that the average value of the utilization rates of the rest resources can be ensured to be smaller than the threshold value, the determined optimal concurrency number does not exceed the load capacity of the distributed system, and the computing resources of the distributed system can be fully utilized.
S103: and in the set time period, distributing the sub-processing tasks with the number corresponding to the optimal concurrency number to a plurality of servers in the plurality of sub-processing tasks included in the batch processing tasks.
Optionally, assigning a sub-processing task to a certain server refers to sending a task instruction to the server, where the task instruction carries an identifier of the sub-processing task, and when the server receives the task instruction and currently has an idle processing capability, executing a corresponding sub-processing task based on the task instruction.
In one possible implementation, the plurality of servers are prioritized in order of greater than lesser than the utilization of the representative resources of the plurality of servers; if the optimal concurrency number of the current period is greater than that of the previous period, sequentially increasing the number of sub-processing tasks allocated to the server from the server with the smallest priority according to the order of the priorities from small to large; or if the optimal concurrency number of the current period is smaller than that of the previous period, sequentially reducing the number of sub-processing tasks allocated to the server from the server with the largest priority according to the order of the priorities from the largest priority.
The distributed system includes three servers, namely a server 4, a server 5 and a server 6, wherein representative resources are CPUs, CPU utilization rates of the server 4, the server 5 and the server 6 are 80%, 60% and 70%, the three servers are prioritized according to the order of the utilization rates of the representative resources from big to small, and the priorities are the server 4, the server 6 and the server 5 from big to small. If the optimal concurrency number of the current period is greater than that of the previous period, the number of the sub-processing tasks allocated to each server is increased according to the order of the server 5, the server 6 and the server 4, so that the server 5 with small representative resource utilization rate preferentially increases the allocated sub-processing tasks, and the computing capacity of the server 5 is fully utilized. If the optimal concurrency number of the current period is smaller than that of the previous period, the number of the sub-processing tasks allocated to each server is reduced according to the order of the server 4, the server 6 and the server 5, so that the server 4 with high representing resource utilization rate preferentially reduces the allocated sub-processing tasks, the load of the server 4 is reduced, and the stability of the distributed system is improved.
Particularly, if the optimal concurrency number of the current period is the same as the optimal concurrency number of the previous period, the manner of allocating sub-processing tasks to the plurality of servers in the current period may be the same as the manner of allocating sub-processing tasks to the plurality of servers in the previous period.
In the mode, the optimal concurrency number of the current period is larger than that of the previous period, and the number of the sub-processing tasks allocated to the server is sequentially increased from the server with the minimum representative resource utilization rate, so that the load of the server with the higher representative resource utilization rate is avoided being exceeded, and the computing capacity of the server with the lower representative resource utilization rate is fully utilized; the optimal concurrency number of the current period is smaller than that of the previous period, and the number of sub-processing tasks allocated to the server is sequentially reduced from the server with the maximum representative resource utilization rate, so that the load of the server with the higher representative resource utilization rate is reduced, and the stability and reliability of the distributed system are improved.
In another possible implementation, the weight of each server is determined according to the processing power of each server; dividing the optimal concurrency number of the current period according to the weight of each server to obtain a plurality of sub concurrency numbers, wherein the plurality of sub concurrency numbers correspond to the plurality of servers one by one, and the number of each sub concurrency number is the same as the number of sub processing tasks allocated to the corresponding server.
In a distributed system, the processing capacity of each server is not necessarily the same, and for example, a server may process at most two sub-processing tasks at the same time, and a server may process at most three sub-processing tasks at the same time. The more sub-processing tasks the server is assigned to, the greater the resource utilization of the server. In the embodiment of the application, the processing capacity of each server is quantified by the number of the processing units included in each server, wherein one processing unit can only process one sub-processing task at the same time, namely, the server which processes at most two sub-processing tasks at the same time comprises two processing units, and the server which processes at most three sub-processing tasks at the same time comprises three processing units.
Optionally, heartbeat information of all processing units included in the plurality of servers is obtained, the heartbeat information of each processing unit includes an identifier of the server where the processing unit is located, and processing capacity of each server is determined according to the heartbeat information of all the processing units.
The distributed system comprises four servers 7, 8, 9 and 10, wherein the number of the processing units included in the four servers is 6, 4, 2 and 8 in sequence, and the weight of the four servers is 3/10, 2/10, 1/10 and 4/10 in sequence by dividing the number of the processing units included in each server by the total number of the processing units included in the plurality of servers. The optimal concurrency number of the current period is 10, and the number of sub-processing tasks distributed to the server 7, the server 8, the server 9 and the server 10 is obtained by multiplying the weight of each server by the optimal concurrency number, and the number of sub-processing tasks distributed to the server 7, the server 8, the server 9 and the server 10 is 3, 2, 1 and 4 in sequence.
By the method, the sub-processing tasks can be distributed to each server according to the processing capacity of each server, the computing capacity of each server can be fully utilized, the possibility that the distributed sub-processing tasks of each server exceed the load of the distributed sub-processing tasks is reduced, and the stability of the distributed system is further improved.
In addition, in the embodiment of the present application, when the distributed system starts to execute the batch processing task, a concurrency number may be estimated in advance based on the carrying capacity of the distributed system, and any server in the distributed system distributes a number of sub-processing tasks corresponding to the estimated concurrency number to a plurality of servers, and after a period of time, the steps S101 to S103 are executed.
Illustratively, referring to FIG. 2, there are X cycles from the start of a batch task to the end of the batch task, where in the first three cycles, the optimal concurrency number for the first cycle is N, and the distributed system performs N sub-processing tasks altogether; the optimal concurrency number of the second period is N+M, and the distributed system executes N+M sub-processing tasks altogether; the third cycle has N-M concurrency, the distributed system executes N-M sub-processing tasks, wherein X, N, M is a positive integer, and N is larger than M.
In the above schemes S101 to S103, any server in the distributed system determines the optimal concurrency number of the distributed system in the set period according to the feedback information of the multiple servers, and allocates the number of sub-processing tasks corresponding to the optimal concurrency number to the multiple servers, so that the computing resources of the distributed system can be fully utilized within the load capacity range of the distributed system, and the efficiency of the distributed system in processing batch processing tasks is improved.
Referring to fig. 3, an embodiment of the present application provides a concurrency number adjustment device 300 for batch processing tasks, where the device is applied to any one server in a distributed system including a plurality of servers, where the plurality of servers combine to complete batch processing tasks of the distributed system, and the device includes modules/units/technical means for executing a method executed by any one server in the above method embodiment.
Illustratively, the apparatus 300 may include:
an obtaining module 301, configured to obtain feedback information of the plurality of servers; the feedback information of each server in the plurality of servers is used for indicating the resource utilization rate of each server;
a processing module 302, configured to determine an optimal concurrency number of the distributed system in a set period of time based on feedback information of the plurality of servers; and in the set time period, distributing the sub-processing tasks with the number corresponding to the optimal concurrency number to the servers in the sub-processing tasks included in the batch processing task.
Optionally, when determining the optimal concurrency number of the distributed system in the set time period based on the feedback information of the plurality of servers, the processing module is specifically configured to: according to a preset time interval, periodically determining the optimal concurrency number of the distributed system in the current period based on feedback information of the plurality of servers in the previous period; the duration of the preset time interval is related to the processing duration of each sub-processing task in the plurality of sub-processing tasks.
Optionally, when determining the optimal concurrency number of the distributed system in the set time period based on the feedback information of the plurality of servers, the processing module is specifically configured to: averaging the utilization rate of the same resource of the plurality of servers to obtain the utilization rate average value of at least one resource of the plurality of servers; determining a representative resource with the maximum average value of the resource utilization rate from the at least one resource; if the average value of the utilization rate of the representative resource is not equal to a threshold value, determining a functional relation between the average value of the utilization rate of the representative resource and the concurrency number of the distributed system according to the average value of the utilization rate of the representative resource in at least one period and the concurrency number; and determining the concurrency number when the average value of the utilization rate of the representative resource is a threshold value based on the functional relation, and taking the concurrency number when the average value of the utilization rate of the representative resource is the threshold value as the optimal concurrency number of the current period.
Optionally, the processing module is further configured to: and if the average value of the utilization rate of the representative resources is equal to the threshold value, the concurrency number of the last period of the distributed system is used as the optimal concurrency number of the current period.
Optionally, the processing module is specifically configured to, when distributing, among the plurality of sub-processing tasks included in the batch processing task, a number of sub-processing tasks corresponding to the optimal concurrency number to the plurality of servers: the priority ranking is carried out on the plurality of servers according to the order of the utilization rate of the representative resources of the plurality of servers from big to small; if the optimal concurrency number of the current period is greater than that of the previous period, sequentially increasing the number of sub-processing tasks allocated to the server from the server with the smallest priority according to the order of the priorities from small to large; or if the optimal concurrency number of the current period is smaller than that of the previous period, sequentially reducing the number of sub-processing tasks allocated to the server from the server with the largest priority according to the order of the priorities from the largest priority.
It should be understood that all relevant contents of each step involved in the above method embodiments may be cited to the functional descriptions of the corresponding functional modules, and are not described herein.
As one possible product form of the above apparatus, referring to fig. 4, an embodiment of the present application further provides an electronic device 400, including:
at least one processor 401; and a communication interface 403 communicatively coupled to the at least one processor 401; the at least one processor 401, by executing instructions stored in the memory 402, causes the electronic device 400 to perform the method in the embodiment shown in fig. 1 via the communication interface 403.
Optionally, the memory 402 is located outside the electronic device 400.
Optionally, the electronic device 400 includes the memory 402, where the memory 402 is connected to the at least one processor 401, and the memory 402 stores instructions executable by the at least one processor 401. Fig. 4 shows, with a dashed line, that the memory 402 is optional for the electronic device 400.
The processor 401 and the memory 402 may be coupled through an interface circuit, or may be integrated together, which is not limited herein.
The specific connection medium between the processor 401, the memory 402, and the communication interface 403 is not limited in the embodiment of the present application. In the embodiment of the present application, the processor 401, the memory 402 and the communication interface 403 are connected by a bus 404 in fig. 4, where the bus is indicated by a thick line in fig. 4, and the connection manner between other components is only schematically illustrated, but not limited thereto. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one thick line is shown in fig. 4, but not only one bus or one type of bus.
It should be understood that the processors mentioned in the embodiments of the present application may be implemented by hardware or may be implemented by software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, or the like. When implemented in software, the processor may be a general purpose processor, implemented by reading software code stored in a memory.
By way of example, the processor may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
It should be understood that the memory referred to in embodiments of the present application may be volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable EPROM (EEPROM), or a flash Memory. The volatile memory may be random access memory (Random Access Memory, RAM) which acts as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data rate Synchronous DRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), and Direct RAM (DR RAM).
It should be noted that when the processor is a general purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, the memory (storage module) may be integrated into the processor.
It should be noted that the memory described herein is intended to comprise, without being limited to, these and any other suitable types of memory.
As another possible product form, the embodiment of the application also provides a computer-readable storage medium for storing instructions that, when executed, cause a computer to perform the method of the embodiment shown in fig. 1.
As another possible product form, the embodiment of the application also provides a computer program product comprising instructions stored therein, which when run on a computer, cause the computer to perform the method of the embodiment shown in fig. 1.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (13)
1. A method for adjusting the concurrency number of batch processing tasks, wherein the method is applied to any one server in a distributed system including a plurality of servers, and the plurality of servers combine to complete the batch processing tasks of the distributed system, the method comprising:
acquiring feedback information of the servers; the feedback information of each server in the plurality of servers is used for indicating the resource utilization rate of each server;
Determining an optimal concurrency number of the distributed system in a set time period based on feedback information of the plurality of servers;
and in the set time period, distributing the sub-processing tasks with the number corresponding to the optimal concurrency number to the servers in the sub-processing tasks included in the batch processing task.
2. The method of claim 1, wherein the determining an optimal number of concurrency of the distributed system over a set period of time based on feedback information of the plurality of servers comprises:
according to a preset time interval, periodically determining the optimal concurrency number of the distributed system in the current period based on feedback information of the plurality of servers in the previous period; the duration of the preset time interval is related to the processing duration of each sub-processing task in the plurality of sub-processing tasks.
3. The method of claim 2, wherein the determining an optimal number of concurrency of the distributed system over a set period of time based on feedback information of the plurality of servers comprises:
averaging the utilization rate of the same resource of the plurality of servers to obtain the utilization rate average value of at least one resource of the plurality of servers;
Determining a representative resource with the maximum average value of the resource utilization rate from the at least one resource;
if the average value of the utilization rate of the representative resource is not equal to a threshold value, determining a functional relation between the average value of the utilization rate of the representative resource and the concurrency number of the distributed system according to the average value of the utilization rate of the representative resource in at least one period and the concurrency number;
and determining the concurrency number when the average value of the utilization rate of the representative resource is a threshold value based on the functional relation, and taking the concurrency number when the average value of the utilization rate of the representative resource is the threshold value as the optimal concurrency number of the current period.
4. A method as claimed in claim 3, wherein the method further comprises:
and if the average value of the utilization rate of the representative resources is equal to the threshold value, the concurrency number of the last period of the distributed system is used as the optimal concurrency number of the current period.
5. The method of claim 3, wherein the assigning a number of sub-processing tasks corresponding to the optimal number of concurrency among the plurality of sub-processing tasks included in the batch processing task to the plurality of servers comprises:
the priority ranking is carried out on the plurality of servers according to the order of the utilization rate of the representative resources of the plurality of servers from big to small;
If the optimal concurrency number of the current period is greater than that of the previous period, sequentially increasing the number of sub-processing tasks allocated to the server from the server with the smallest priority according to the order of the priorities from small to large; or,
and if the optimal concurrency number of the current period is smaller than that of the previous period, sequentially reducing the number of sub-processing tasks allocated to the server from the server with the largest priority according to the order of the priorities from the largest priority.
6. A concurrency number adjusting apparatus for batch processing tasks, the apparatus being applied to any one of servers in a distributed system including a plurality of servers that combine to complete batch processing tasks of the distributed system, the apparatus comprising:
the acquisition module is used for acquiring feedback information of the servers; the feedback information of each server in the plurality of servers is used for indicating the resource utilization rate of each server;
the processing module is used for determining the optimal concurrency number of the distributed system in a set time period based on the feedback information of the servers; and in the set time period, distributing the sub-processing tasks with the number corresponding to the optimal concurrency number to the servers in the sub-processing tasks included in the batch processing task.
7. The apparatus of claim 6, wherein the processing module, when determining an optimal number of concurrency of the distributed system over a set time period based on feedback information of the plurality of servers, is specifically configured to:
according to a preset time interval, periodically determining the optimal concurrency number of the distributed system in the current period based on feedback information of the plurality of servers in the previous period; the duration of the preset time interval is related to the processing duration of each sub-processing task in the plurality of sub-processing tasks.
8. The apparatus of claim 7, wherein the processing module, when determining an optimal number of concurrency of the distributed system over a set time period based on feedback information of the plurality of servers, is specifically configured to:
averaging the utilization rate of the same resource of the plurality of servers to obtain the utilization rate average value of at least one resource of the plurality of servers;
determining a representative resource with the maximum average value of the resource utilization rate from the at least one resource;
if the average value of the utilization rate of the representative resource is not equal to a threshold value, determining a functional relation between the average value of the utilization rate of the representative resource and the concurrency number of the distributed system according to the average value of the utilization rate of the representative resource in at least one period and the concurrency number;
And determining the concurrency number when the average value of the utilization rate of the representative resource is a threshold value based on the functional relation, and taking the concurrency number when the average value of the utilization rate of the representative resource is the threshold value as the optimal concurrency number of the current period.
9. The apparatus of claim 8, wherein the processing module is further to:
and if the average value of the utilization rate of the representative resources is equal to the threshold value, the concurrency number of the last period of the distributed system is used as the optimal concurrency number of the current period.
10. The apparatus of claim 8, wherein the processing module, when allocating, to the plurality of servers, a number of sub-processing tasks corresponding to the optimal number of concurrency, among a plurality of sub-processing tasks included in the batch processing task, is specifically configured to:
the priority ranking is carried out on the plurality of servers according to the order of the utilization rate of the representative resources of the plurality of servers from big to small;
if the optimal concurrency number of the current period is greater than that of the previous period, sequentially increasing the number of sub-processing tasks allocated to the server from the server with the smallest priority according to the order of the priorities from small to large; or,
And if the optimal concurrency number of the current period is smaller than that of the previous period, sequentially reducing the number of sub-processing tasks allocated to the server from the server with the largest priority according to the order of the priorities from the largest priority.
11. An electronic device, comprising:
at least one processor; and a memory communicatively coupled to the at least one processor, a communication interface;
wherein the memory stores instructions executable by the at least one processor, which, by executing the instructions stored by the memory, cause the electronic device to perform the method of any one of claims 1-5 via the communication interface.
12. A computer readable storage medium storing computer instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1 to 5.
13. A computer program product, the computer program product comprising: computer program code which, when run on a computer, causes the computer to perform the method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311218834.5A CN117194035A (en) | 2023-09-20 | 2023-09-20 | Method, device, equipment and medium for adjusting concurrency number of batch processing tasks |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311218834.5A CN117194035A (en) | 2023-09-20 | 2023-09-20 | Method, device, equipment and medium for adjusting concurrency number of batch processing tasks |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117194035A true CN117194035A (en) | 2023-12-08 |
Family
ID=88988548
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311218834.5A Pending CN117194035A (en) | 2023-09-20 | 2023-09-20 | Method, device, equipment and medium for adjusting concurrency number of batch processing tasks |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117194035A (en) |
-
2023
- 2023-09-20 CN CN202311218834.5A patent/CN117194035A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107291546B (en) | Resource scheduling method and device | |
| US20170255496A1 (en) | Method for scheduling data flow task and apparatus | |
| CN107688492B (en) | Resource control method and device and cluster resource management system | |
| EP2701074A1 (en) | Method, device, and system for performing scheduling in multi-processor core system | |
| JP2014123365A (en) | DEVICE AND METHOD FOR OPTIMIZATION OF DATA PROCESSING IN MapReduce FRAMEWORK | |
| CN112214319B (en) | Task scheduling method for sensing computing resources | |
| CN112860387A (en) | Distributed task scheduling method and device, computer equipment and storage medium | |
| JP2006244479A (en) | System and method for scheduling executable program | |
| CN107145388A (en) | Method for scheduling task and system under a kind of multitask environment | |
| CN110413393B (en) | Cluster resource management method and device, computer cluster and readable storage medium | |
| CN112817722A (en) | Time-sharing scheduling method based on priority, terminal and storage medium | |
| CN113032102A (en) | Resource rescheduling method, device, equipment and medium | |
| CN114816709A (en) | Task scheduling method, device, server and readable storage medium | |
| CN109189581B (en) | Job scheduling method and device | |
| CN106775975B (en) | Process scheduling method and device | |
| CN113051063B (en) | Task scheduling method and device for distributed tasks and electronic equipment | |
| CN117971491A (en) | In-process resource control method, device, equipment and storage medium | |
| CN103823712A (en) | Data flow processing method and device for multi-CPU virtual machine system | |
| CN110175078B (en) | Service processing method and device | |
| CN112000485A (en) | Task allocation method and device, electronic equipment and computer readable storage medium | |
| CN109445863B (en) | Data processing method, device, equipment and medium based on FPGA | |
| CN108279977B (en) | Data processing method and device based on RTOS (real time operating System) and controller | |
| CN117194035A (en) | Method, device, equipment and medium for adjusting concurrency number of batch processing tasks | |
| CN111143063A (en) | Task resource reservation method and device | |
| CN112486638A (en) | Method, apparatus, device and storage medium for executing processing task |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |