CN111176840A - Distributed task allocation optimization method and device, storage medium and electronic device - Google Patents

Distributed task allocation optimization method and device, storage medium and electronic device Download PDF

Info

Publication number
CN111176840A
CN111176840A CN201911330857.9A CN201911330857A CN111176840A CN 111176840 A CN111176840 A CN 111176840A CN 201911330857 A CN201911330857 A CN 201911330857A CN 111176840 A CN111176840 A CN 111176840A
Authority
CN
China
Prior art keywords
node
task
distributed
determining
distributed system
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201911330857.9A
Other languages
Chinese (zh)
Inventor
尹德帅
徐志方
刘超
沈顺厚
王守峰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qingdao Haier Technology Co Ltd
Original Assignee
Qingdao Haier Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qingdao Haier Technology Co Ltd filed Critical Qingdao Haier Technology Co Ltd
Priority to CN201911330857.9A priority Critical patent/CN111176840A/en
Publication of CN111176840A publication Critical patent/CN111176840A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061Partitioning or combining of resources
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083Techniques for rebalancing the load in a distributed system

Abstract

The invention provides a distributed task allocation optimization method and a distributed task allocation optimization device, wherein the method comprises the following steps: determining a second node set in a first node set formed by all nodes of the distributed system according to the state information of each node in the distributed system, wherein each node in the second node set is a node allowing execution of a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set; determining a degree of association parameter between the current task to be distributed and each node in the second node set, wherein the degree of association parameter is used for indicating the degree of association between the current task to be distributed and the task received by each node in the second node set; and distributing the current task to be distributed to the target node with the highest association degree in the second node set for execution.

Description

Distributed task allocation optimization method and device, storage medium and electronic device
Technical Field
The invention relates to the field of computers, in particular to a distributed task allocation optimization method and device, a storage medium and an electronic device.
Background
In the related technology, when a scheduling system of a distributed system performs node scheduling, the resource load conditions of all nodes are mainly calculated through a load balancing algorithm, then the nodes with low load utilization rate are preferentially added into a scheduling queue, and tasks are randomly distributed to the nodes added into the queue.
In the node allocation mode in the related art, for nodes added into the scheduling queue, the problem of unequal high and low task execution efficiency still exists due to different load utilization rates, and if the random allocation mode is simply adopted, the task execution effect is good or bad, and the optimization of the allocation optimization of the distributed tasks cannot be realized.
Aiming at the problems that in the related art, when a scheduling system of a distributed system performs node scheduling, due to the difference of load utilization rates of nodes, distribution optimization of distributed tasks is still unreasonable and the like, an effective technical scheme is not provided.
Disclosure of Invention
The embodiment of the invention provides a distributed task allocation optimization method and a distributed task allocation optimization device, which are used for at least solving the problems that in the related art, when a scheduling system of a distributed system performs node scheduling, due to the difference of load utilization rates of nodes, the distributed task allocation optimization is still unreasonable and the like.
According to an embodiment of the present invention, there is provided a distributed task allocation optimization method, including: determining a second node set in a first node set formed by all nodes of the distributed system according to state information of each node in the distributed system, wherein each node in the second node set is a node which is allowed to execute a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set; determining a degree of association parameter between the current task to be distributed and each node in the second node set; the association degree parameter is used for indicating the association degree between the current task to be distributed and the task received by each node in the second node set; and optimizing the distribution of the current distributed tasks to be distributed to the target node with the highest association degree in the second node set for execution.
In this embodiment of the present invention, the state information of each node in the distributed system includes: the number of uncompleted tasks, the average load parameter within a first predetermined time period, and the estimated completion time of the uncompleted tasks; determining a second node set in a first node set formed by all nodes of the distributed system according to the state information of each node in the distributed system, wherein the determining comprises the following steps: acquiring state information of each node in the distributed system; determining a performance evaluation value of each node in the distributed system according to the state information; determining the second set of nodes in the first set of nodes in the distributed system according to the performance evaluation value.
In an embodiment of the present invention, the determining, according to the performance evaluation value, the second node set in the first node set in the distributed system includes: comparing the performance evaluation value of each node in the distributed system with a preset threshold value; and determining a set formed by nodes of which the performance evaluation values are smaller than the preset threshold value as the second node set.
In an embodiment of the present invention, the determining a performance evaluation value of each node in the distributed system includes: normalizing the number of the uncompleted tasks to obtain the normalized number of the uncompleted tasks; determining a first product of a first weight and the number of uncompleted tasks after the normalization, determining a second product of a second weight and the average load parameter, and determining a third product of a third weight and the estimated completion time; wherein the first weight is greater than the second weight, which is greater than the third weight; determining a sum of the first product, the second product, and the third product as the performance evaluation value.
In this embodiment of the present invention, the determining a parameter of a degree of association between a current task to be allocated and each node in the second node set includes: acquiring the maximum task number of each node in the second node set within second preset time, determining the maximum task number as a first task number, acquiring the average value of the task numbers of each node in the second node set within the second preset time, determining the average value as a second task number, acquiring the task number included in a task set to be allocated in the distributed system, and determining the task number included in the task set to be allocated as a third task number; the task set to be allocated comprises the current task to be allocated; determining a ratio of the third number of tasks to the second number of tasks; determining a covariance value between the ratio and the first task number, and determining the covariance value as the association degree parameter.
In this embodiment of the present invention, allocating the task to be currently allocated to the target node with the highest degree of association in the second node set, and performing the task includes: arranging each node in the second node set according to a first preset sequence according to the magnitude of the association degree parameter to obtain a target node set arranged in the first preset sequence; determining a node corresponding to the maximum value of the association degree parameter in the target node set as the target node; and distributing the current task to be distributed to the target node for execution.
In the embodiment of the present invention, each task to be allocated in the task sets to be allocated in the distributed system is arranged in a task pool in the distributed system according to a second preset sequence, the current task to be allocated is any one task to be allocated in the task sets to be allocated, and the task pool is used for storing the task sets to be allocated in the distributed system.
According to another embodiment of the present invention, there is also provided a distributed task allocation optimizing apparatus including: a first determining module, configured to determine a second node set in a first node set composed of all nodes in the distributed system according to state information of each node in the distributed system, where each node in the second node set is a node that is allowed to execute a task to be allocated in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set; a second determining module, configured to determine a degree of association parameter between a current task to be allocated and each node in the second node set, where the degree of association parameter is used to indicate a degree of association between the current task to be allocated and a task that has been received by each node in the second node set; and the distribution module is used for distributing the current task to be distributed to the target node with the highest correlation degree in the second node set for execution.
There is also provided, in accordance with another embodiment of the present invention, a computer-readable storage medium including a stored program, wherein the program when executed performs the method for optimizing distribution of distributed tasks as set forth in any one of the above.
There is also provided, in accordance with another embodiment of the present invention, an electronic apparatus, the storage medium including a stored program, wherein the program executes, when running, the method for optimizing distribution of distributed tasks according to any one of the above.
According to the invention, a second node set is determined in a first node set formed by all nodes of the distributed system according to the state information of each node in the distributed system, wherein each node in the second node set is a node which is allowed to execute a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set; determining a degree of association parameter between the current task to be distributed and each node in the second node set; the association degree parameter is used for indicating the association degree between the current task to be distributed and the task received by each node in the second node set; and optimizing the distribution of the current distributed tasks to be distributed to the target node with the highest association degree in the second node set for execution. By adopting the technical scheme, the problems that in the related art, when the dispatching system of the distributed system carries out node dispatching, the distributed tasks are still unreasonable in distribution optimization due to different load utilization rates of the nodes and the like are solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
fig. 1 is a hardware structure block diagram of a distributed task allocation optimization method according to an embodiment of the present invention;
FIG. 2 is a flow chart of an alternative method for optimizing the distribution of distributed tasks according to an embodiment of the present invention;
FIG. 3 is a flow diagram of an alternative method for optimizing the distribution of distributed tasks according to an embodiment of the invention;
FIG. 4 is a schematic diagram of an alternative distributed system according to embodiments of the present invention;
FIG. 5 is a block diagram of an alternative method for optimizing the distribution of distributed tasks, according to an embodiment of the present invention;
fig. 6 is a block diagram of another alternative first determination module according to an embodiment of the invention.
Detailed Description
The invention will be described in detail hereinafter with reference to the accompanying drawings in conjunction with embodiments. It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
The method provided by the embodiment of the application can be applied to a distributed system for execution. Fig. 1 is a block diagram of a hardware configuration of a distributed task allocation optimization apparatus according to an embodiment of the present invention. As shown in fig. 1, the distributed task allocation optimization apparatus 10 may include one or more processors 102 (only one is shown in fig. 1) (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), and a memory 104 for storing data, and optionally, the intelligent door lock may further include a transmission device 106 for communication function and an input/output device 108. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration, and does not limit the structure of the above-described distributed task allocation optimizing device. For example, the distributed task allocation optimization apparatus 10 may also include more or fewer components than shown in FIG. 1, or have a different configuration with equivalent functionality to that shown in FIG. 1 or with more functionality than that shown in FIG. 1.
The memory 104 may be used to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to the distributed task allocation optimization method in the embodiment of the present invention, and the processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, so as to implement the method described above. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the mobile terminal 10 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used for receiving or transmitting data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal 10. In one example, the transmission device 106 includes a Network adapter (NIC), which can be connected to other Network devices through a base station so as to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
In this embodiment, a method for optimizing distribution of distributed tasks running on the intelligent door lock is provided, and fig. 2 is a flowchart of an optional method for optimizing distribution of distributed tasks according to an embodiment of the present invention, as shown in fig. 2, the method includes the following steps:
step S202, according to the state information of each node in the distributed system, determining a second node set in a first node set formed by all nodes in the distributed system, wherein each node in the second node set is a node allowing execution of a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set;
step S204, determining a degree of association parameter between the current task to be distributed and each node in the second node set, wherein the degree of association parameter is used for indicating the degree of association between the current task to be distributed and the task received by each node in the second node set;
step S206, the task to be currently allocated is allocated to the target node with the highest association degree in the second node set for execution.
According to the invention, a second node set is determined in a first node set formed by all nodes of the distributed system according to the state information of each node in the distributed system, wherein each node in the second node set is a node which is allowed to execute a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set; determining a degree of association parameter between the current task to be distributed and each node in the second node set; the association degree parameter is used for indicating the association degree between the current task to be distributed and the task received by each node in the second node set; and optimizing the distribution of the current distributed tasks to be distributed to the target node with the highest association degree in the second node set for execution. By adopting the technical scheme, the problems that in the related art, when the dispatching system of the distributed system carries out node dispatching, the distributed tasks are still unreasonable in distribution optimization due to different load utilization rates of the nodes and the like are solved.
In step S202, the state information of each node in the distributed system includes: the number of uncompleted tasks, the average load parameter within a first predetermined time period, and the estimated completion time of the uncompleted tasks; determining a second node set in a first node set formed by all nodes of the distributed system according to the state information of each node in the distributed system, wherein the determining comprises the following steps: acquiring state information of each node in the distributed system; determining a performance evaluation value of each node in the distributed system according to the state information; determining the second set of nodes in the first set of nodes in the distributed system according to the performance evaluation value.
Optionally, the determining, according to the performance evaluation value, the second node set in the first node set in the distributed system includes: comparing the performance evaluation value of each node in the distributed system with a preset threshold value; and determining a set formed by nodes of which the performance evaluation values are smaller than the preset threshold value as the second node set.
There are various implementations of determining the performance evaluation value, and in an alternative embodiment, the determining the performance evaluation value of each node in the distributed system includes: normalizing the number of the uncompleted tasks to obtain the normalized number of the uncompleted tasks; determining a first product of a first weight and the number of uncompleted tasks after the normalization, determining a second product of a second weight and the average load parameter, and determining a third product of a third weight and the estimated completion time; wherein the first weight is greater than the second weight, which is greater than the third weight; determining a sum of the first product, the second product, and the third product as the performance evaluation value.
In optional embodiments, the determining a parameter of a degree of association between the current task to be distributed and each node in the second node set includes: acquiring the maximum task number of each node in the second node set within second preset time, determining the maximum task number as a first task number, acquiring the average value of the task numbers of each node in the second node set within the second preset time, determining the average value as a second task number, acquiring the task number included in a task set to be allocated in the distributed system, and determining the task number included in the task set to be allocated as a third task number; the task set to be allocated comprises the current task to be allocated; determining a ratio of the third number of tasks to the second number of tasks; determining a covariance value between the ratio and the first task number, and determining the covariance value as the association degree parameter.
In this embodiment of the present invention, allocating the current task to be allocated to the target node with the highest association degree in the second node set for execution includes: arranging each node in the second node set according to a first preset sequence according to the magnitude of the association degree parameter to obtain a target node set arranged in the first preset sequence; determining a node corresponding to the maximum value of the association degree parameter in the target node set as the target node; and distributing the current task to be distributed to the target node for execution.
Optionally, each task to be allocated in the task set to be allocated in the distributed system is arranged in a task pool in the distributed system according to a second preset sequence, the current task to be allocated is any one task to be allocated in the task set to be allocated, and the task pool is used for storing the task set to be allocated in the distributed system.
The following describes a flow of a task allocation method with reference to an alternative example, as shown in fig. 3, the method includes the following steps:
step S302, acquiring real-time task execution data of each node in the system;
when a new task in the system task pool is distributed, considering the load balance condition of the nodes to be distributed, the task execution conditions of all the nodes are analyzed, the task execution conditions of all the nodes are comprehensively considered, and the task is dynamically distributed according to the actual conditions of all the nodes. The phenomenon that when a node has too many tasks which are not finished at present, the node load is too heavy is avoided.
As shown in fig. 4, an average load parameter L of each node in a first node set composed of all nodes in the distributed system may be obtained, where the average load parameter L is an average number of processes in a first preset time period, and the distributed system can preferentially allocate tasks to nodes with low average load; the time parameter index of each node in the distributed system can be obtained, for example, the estimated completion time sum of the uncompleted task of each node is obtained, the estimated completion time sum of the uncompleted task can accurately reflect the capability of the node to execute the new task, if the time for the node to process the uncompleted task is long, the node is not allocated with the new task, and the parameter index can be determined based on the uncompleted task data, the CPU usage data and the content usage data of the node through comprehensive calculation; the number M of uncompleted tasks of each node in all nodes in the distributed system can be obtained.
Step S304, calculating the comprehensive performance index of the node based on the number M of unfinished tasks of the node, the average load parameter L of the node and the estimated finish time sum T of unfinished tasks of the node;
in an alternative embodiment, the overall performance index of the node (corresponding to the performance assessment value described above) is represented by Q, and the threshold Q for the schedulable task is determined at the same timei(corresponding to the above-mentioned preset threshold), only if the Q value is less than a defined threshold QiAs shown in FIG. 4, the Q value is made smaller than a defined threshold QiAnd adding the corresponding node into a scheduling queue (corresponding to the second node set), wherein the node comprehensive performance index exceeds a threshold value, the average load of the identified node is higher, the number of uncompleted tasks is larger, and the estimated completion time sum of the uncompleted tasks of the node is larger.
By the method and the device, the calculation parameters of the sum of the estimated completion time of the uncompleted tasks of the nodes are increased, and the actual task execution capacity of the nodes is comprehensively considered. The nodes in the distributed system have different CPU frequencies, memory sizes and the like, the processing capacities of different nodes for the same task are different, the actual task execution capacity of the nodes is comprehensively considered, the dynamic task allocation can be carried out according to the actual condition of each node, the phenomenon that when the number of tasks which are not completed by one node at present is too large, the node load is too heavy is avoided, and the task allocation reasonability in the distributed system is improved.
Optionally, the following is a performance evaluation value calculation process, which is specifically described as follows:
Q=M′k1+Lk2+Tk3(1)
wherein, M' is log (M), which is a result obtained after normalization processing is performed on the number M of uncompleted tasks;
m is the number of unfinished tasks of the node;
t is the sum of estimated completion time of uncompleted tasks;
l is an average load parameter of each node in the distributed system in a first preset time period, and only the nodes with low average load rate are assigned with tasks in limited consideration;
k1、k2、k3weight values set according to the degree of importance of the index, which should satisfy k1>k2>k3. Wherein k is1Corresponding to the first weight, k1Corresponding to the second weight, k1Corresponding to the third weight described above.
Step S306, based on the comprehensive performance indexes of all the nodes calculated in step S304, the set schedulable task threshold value Q is passed throughiComparing, determining nodes capable of distributing tasks, and adding the screened nodes into a scheduling queue (corresponding to the second node set);
in an alternative embodiment, the task threshold Q may be schedulediCan be set according to the computing capability of the computing equipment of the nodes in the system when the task threshold value Q can be schedulediToo high a definition may result in some idle nodesScheduling cannot be obtained; when schedulable task threshold QiWhen the definition is too low, the nodes receive too many tasks, and the load among the nodes is unbalanced. The node number and the task number of the system need to be comprehensively determined according to the ratio of the node number and the task number of the system.
Step S308, carrying out secondary optimization on the nodes added into the scheduling queue, calculating the association degree index value between the nodes in the queue and the tasks to be distributed, grouping according to the tasks to be distributed, determining a node set corresponding to each task to be distributed, and determining the optimal execution node (corresponding to the target node) of the task to be distributed based on the determined node set;
by executing step S306, the nodes that can perform task allocation are added to the scheduling queue, but the nodes included in the scheduling queue are not necessarily the optimal nodes for executing the tasks to be allocated.
In an optional embodiment, the process of determining the optimal execution node specifically includes the following steps:
step 1, determining a historical load parameter X (corresponding to the first task quantity) of each node in a scheduling queue;
the historical load parameter can be the maximum value of the task allocation quantity of the node in a historical period of time;
X=max(Xi) (2)
step 2, calculating the ratio Y of the task to be distributed to the historical distribution task quantity average value of the node;
calculating the average value of the historical distribution task quantity of the nodes (corresponding to the second task quantity);
calculating the ratio of the task quantity of the tasks to be distributed to the historical distribution task quantity average value of the nodes (corresponding to the ratio of the third task quantity to the second task quantity);
step 3, calculating a covariance value between the historical load parameter X and the ratio Y (corresponding to a covariance value between the ratio and the first task quantity);
the covariance is defined as:
COV(X,Y)=E[(X-E(X))(Y-E(Y))](3)
the equivalent calculation formula is: COV (X, Y) ═ E (X-Y) -E (X) E (Y).
The covariance value between each task to be allocated and the node in the scheduling queue can be obtained through the formula (3), and the covariance value can reflect the correlation degree between the historical allocation task amount of the node and the task amount of the task to be allocated.
Step 4, grouping the covariance values obtained by calculation by taking the tasks to be distributed as units; obtaining a set of covariance values, for example, Vc ═ { C1, C2, C3, … … }, where C1, C2, and C3 are covariance values between each node in the scheduling queue and the task to be allocated;
then, in each combination, respectively selecting nodes corresponding to S values with the covariance values at the front to form node combinations Vj; for example, Vj ═ N1, N2, N3, … …, Ns }.
Assuming that S is 10 and the number of tasks to be distributed is 10, the node combination formed by each task to be distributed is 1x10 matrix size; the 10 tasks to be assigned may be formed as a matrix of 10 x 10.
5. Selecting the node with the smallest sequencing j in the combination Vj as the optimal execution node of the task to be distributed;
in this way, each task to be distributed uniquely determines an optimal node to which the corresponding task to be distributed is distributed.
Step S310, the scheduling module allocates tasks to the optimal nodes determined in the step S308 and deletes the allocated tasks from the task pool;
when the scheduling module allocates a certain task to be allocated to the determined optimal node from the task pool for execution, the task pool can automatically delete the task, and the task is prevented from being repeatedly allocated.
According to the embodiment, by introducing the time parameter index, the estimated completion time sum of the uncompleted tasks of the node can be calculated, the capability of the node for executing the new task can be accurately reflected, if the time for the node to process the uncompleted tasks is long, the new task is not distributed to the node, the association degree parameter of the node and the task to be distributed can be determined through covariance value calculation, the association degree of the node and the task to be distributed is improved, and the reasonability of the task to be distributed in a distributed system is improved.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
The present embodiment further provides a distributed task allocation optimization apparatus, which is applied to a distributed system, and the apparatus is used to implement the foregoing embodiments and preferred embodiments, and the description of the apparatus is omitted for brevity. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 5 is a block diagram of an alternative apparatus for optimizing distribution of distributed tasks according to an embodiment of the present invention, as shown in fig. 5, the apparatus includes:
a first determining module 502, configured to determine, according to state information of each node in the distributed system, a second node set from the first node set composed of all nodes in the distributed system, where each node in the second node set is a node that is allowed to execute a task to be allocated in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set;
a second determining module 504, configured to determine a degree of association parameter between a current task to be allocated and each node in the second node set, where the degree of association parameter is used to indicate a degree of association between the current task to be allocated and a task that has been received by each node in the second node set;
and an allocating module 506, configured to allocate the current task to be allocated to a target node with the highest association degree in the second node set for execution.
According to the invention, a second node set is determined in a first node set formed by all nodes of the distributed system according to the state information of each node in the distributed system, wherein each node in the second node set is a node which is allowed to execute a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set; determining a degree of association parameter between the current task to be distributed and each node in the second node set; the association degree parameter is used for indicating the association degree between the current task to be distributed and the task received by each node in the second node set; and optimizing the distribution of the current distributed tasks to be distributed to the target node with the highest association degree in the second node set for execution. By adopting the technical scheme, the problems that in the related art, when the dispatching system of the distributed system carries out node dispatching, the distributed tasks are still unreasonable in distribution optimization due to different load utilization rates of the nodes and the like are solved.
In the embodiment of the present invention, as shown in fig. 6, the state information of each node in the distributed system includes: the number of uncompleted tasks, the average load parameter within a first predetermined time period, and the estimated completion time of the uncompleted tasks; the first determining module 502 includes: an obtaining unit 5021, configured to obtain state information of each node in the distributed system; a determining unit 5022, configured to determine a performance evaluation value of each node in the distributed system according to the state information; the determining unit is further configured to determine the second node set from the first node set in the distributed system according to the performance evaluation value.
In this embodiment of the present invention, the determining unit 5022 is further configured to compare the performance evaluation value of each node in the distributed system with a preset threshold; and determining a set formed by nodes of which the performance evaluation values are smaller than the preset threshold value as the second node set.
In this embodiment of the present invention, the determining unit 5022 is further configured to normalize the number of the uncompleted tasks, so as to obtain the number of the uncompleted tasks after normalization; determining a first product of a first weight and the number of uncompleted tasks after the normalization, determining a second product of a second weight and the average load parameter, and determining a third product of a third weight and the estimated completion time; wherein the first weight is greater than the second weight, which is greater than the third weight; determining a sum of the first product, the second product, and the third product as the performance evaluation value.
In this embodiment of the present invention, the second determining module 504 is further configured to obtain a maximum task number of tasks allocated to each node in the second node set within a second predetermined time, determine the maximum task number as a first task number, obtain a mean value of the task numbers of tasks allocated to each node in the second node set within the second predetermined time, determine the mean value as a second task number, obtain a task number included in a task set to be allocated in the distributed system, and determine the task number included in the task set to be allocated as a third task number; the task set to be allocated comprises the current task to be allocated; determining a ratio of the third number of tasks to the second number of tasks; determining a covariance value between the ratio and the first task number, and determining the covariance value as the association degree parameter.
In this embodiment of the present invention, the allocating module 506 is further configured to arrange each node in the second node set according to a first preset order according to the magnitude of the association degree parameter, so as to obtain a target node set arranged in the first preset order; determining a node corresponding to the maximum value of the association degree parameter in the target node set as the target node; and distributing the current task to be distributed to the target node for execution.
In the embodiment of the invention, each task to be allocated in the task sets to be allocated in the distributed system is arranged in the task pool in the distributed system according to a second preset sequence, the current task to be allocated is any one task to be allocated in the task sets to be allocated, and the task pool is used for storing the task sets to be allocated in the distributed system.
Embodiments of the present invention also provide a computer-readable storage medium including a stored program, wherein the program performs any one of the methods described above when executed.
Alternatively, in the present embodiment, the storage medium may be configured to store program codes for performing the following steps:
s1, according to the state information of each node in the distributed system, determining a second node set in a first node set formed by all nodes in the distributed system, wherein each node in the second node set is a node which is allowed to execute a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set;
s2, determining a degree of association parameter between the current task to be distributed and each node in the second node set, wherein the degree of association parameter is used for indicating the degree of association between the current task to be distributed and the task received by each node in the second node set;
and S3, distributing the current task to be distributed to the target node with the highest correlation degree in the second node set for execution.
Optionally, in this embodiment, the storage medium may include, but is not limited to: various media capable of storing program codes, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
Embodiments of the present invention also provide an electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the steps of any of the above method embodiments.
Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:
s1, according to the state information of each node in the distributed system, determining a second node set in a first node set formed by all nodes in the distributed system, wherein each node in the second node set is a node which is allowed to execute a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set;
s2, determining a degree of association parameter between the current task to be distributed and each node in the second node set, wherein the degree of association parameter is used for indicating the degree of association between the current task to be distributed and the task received by each node in the second node set;
and S3, distributing the current task to be distributed to the target node with the highest correlation degree in the second node set for execution.
Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation manners, and this embodiment is not described herein again.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A distributed task allocation optimization method is applied to a distributed system and is characterized by comprising the following steps:
determining a second node set in a first node set formed by all nodes of the distributed system according to state information of each node in the distributed system, wherein each node in the second node set is a node which is allowed to execute a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set;
determining an association degree parameter of the current task to be distributed and each node in the second node set, wherein the association degree parameter is used for indicating the association degree of the current task to be distributed and the task received by each node in the second node set;
and distributing the current task to be distributed to the target node with the highest association degree in the second node set for execution.
2. The method of claim 1, wherein the state information for each node in the distributed system comprises: the number of uncompleted tasks, the average load parameter within a first predetermined time period, and the estimated completion time of the uncompleted tasks;
determining a second node set in a first node set formed by all nodes of the distributed system according to the state information of each node in the distributed system, wherein the determining comprises the following steps:
acquiring state information of each node in the distributed system;
determining a performance evaluation value of each node in the distributed system according to the state information;
determining the second set of nodes in the first set of nodes in the distributed system according to the performance evaluation value.
3. The method of claim 2, wherein determining the second set of nodes in the first set of nodes in the distributed system based on the performance assessment value comprises:
comparing the performance evaluation value of each node in the distributed system with a preset threshold value;
and determining a set formed by nodes of which the performance evaluation values are smaller than the preset threshold value as the second node set.
4. The method of claim 2, wherein determining the performance assessment value for each node in the distributed system comprises:
normalizing the number of the uncompleted tasks to obtain the normalized number of the uncompleted tasks;
determining a first product of a first weight and the number of uncompleted tasks after the normalization, determining a second product of a second weight and the average load parameter, and determining a third product of a third weight and the estimated completion time; wherein the first weight is greater than the second weight, which is greater than the third weight;
determining a sum of the first product, the second product, and the third product as the performance evaluation value.
5. The method of claim 1, wherein determining a degree of association parameter between the current task to be distributed and each node in the second set of nodes comprises:
acquiring the maximum task number of each node in the second node set within second preset time, determining the maximum task number as a first task number, acquiring the average value of the task numbers of each node in the second node set within the second preset time, determining the average value as a second task number, acquiring the task number included in a task set to be allocated in the distributed system, and determining the task number included in the task set to be allocated as a third task number; the task set to be allocated comprises the current task to be allocated;
determining a ratio of the third number of tasks to the second number of tasks;
determining a covariance value between the ratio and the first task number, and determining the covariance value as the association degree parameter.
6. The method according to claim 5, wherein the allocating the task to be currently allocated to the target node with the highest association degree in the second node set comprises:
arranging each node in the second node set according to a first preset sequence according to the magnitude of the association degree parameter to obtain a target node set arranged in the first preset sequence;
determining a node corresponding to the maximum value of the association degree parameter in the target node set as the target node;
and distributing the current task to be distributed to the target node for execution.
7. The method according to any one of claims 1 to 6,
and arranging each task to be distributed in the task set to be distributed in the distributed system in a task pool in the distributed system according to a second preset sequence, wherein the current task to be distributed is any one task to be distributed in the task set to be distributed, and the task pool is used for storing the task set to be distributed in the distributed system.
8. A distributed task allocation optimization device is applied to a distributed system and is characterized by comprising:
a first determining module, configured to determine a second node set in a first node set composed of all nodes in the distributed system according to state information of each node in the distributed system, where each node in the second node set is a node that is allowed to execute a task to be allocated in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set;
a second determining module, configured to determine a degree of association parameter between a current task to be allocated and each node in the second node set, where the degree of association parameter is used to indicate a degree of association between the current task to be allocated and a task that has been received by each node in the second node set;
and the distribution module is used for distributing the current task to be distributed to the target node with the highest correlation degree in the second node set for execution.
9. A computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to carry out the method of any one of claims 1 to 7 when executed.
10. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and wherein the processor is arranged to execute the computer program to perform the method of any of claims 1 to 7.
CN201911330857.9A 2019-12-20 2019-12-20 Distributed task allocation optimization method and device, storage medium and electronic device Pending CN111176840A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911330857.9A CN111176840A (en) 2019-12-20 2019-12-20 Distributed task allocation optimization method and device, storage medium and electronic device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911330857.9A CN111176840A (en) 2019-12-20 2019-12-20 Distributed task allocation optimization method and device, storage medium and electronic device

Publications (1)

Publication Number Publication Date
CN111176840A true CN111176840A (en) 2020-05-19

Family

ID=70655616

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911330857.9A Pending CN111176840A (en) 2019-12-20 2019-12-20 Distributed task allocation optimization method and device, storage medium and electronic device

Country Status (1)

Country Link
CN (1) CN111176840A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111737751A (en) * 2020-07-17 2020-10-02 支付宝(杭州)信息技术有限公司 Method and device for realizing distributed data processing of privacy protection
CN112162846A (en) * 2020-11-27 2021-01-01 腾讯科技(深圳)有限公司 Transaction processing method, device and computer readable storage medium

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111737751A (en) * 2020-07-17 2020-10-02 支付宝(杭州)信息技术有限公司 Method and device for realizing distributed data processing of privacy protection
CN112162846A (en) * 2020-11-27 2021-01-01 腾讯科技(深圳)有限公司 Transaction processing method, device and computer readable storage medium
CN112162846B (en) * 2020-11-27 2021-04-09 腾讯科技(深圳)有限公司 Transaction processing method, device and computer readable storage medium

Similar Documents

Publication Publication Date Title
CN105718479B (en) Execution strategy generation method and device under cross-IDC big data processing architecture
CN105900064A (en) Method and apparatus for scheduling data flow task
CN103699445A (en) Task scheduling method, device and system
CN111176840A (en) Distributed task allocation optimization method and device, storage medium and electronic device
CN105718317A (en) Task scheduling method and task scheduling device
CN103401947A (en) Method and device for allocating tasks to multiple servers
CN107800756A (en) A kind of load-balancing method and load equalizer
CN111427679A (en) Computing task scheduling method, system and device facing edge computing
CN106851747B (en) Dynamic resource allocation method and device in mobile communication system
CN109981744B (en) Data distribution method and device, storage medium and electronic equipment
CN110502321A (en) A kind of resource regulating method and system
US20170339069A1 (en) Allocating Cloud Computing Resources In A Cloud Computing Environment
CN103699443A (en) Task distributing method and scanner
CN110333937A (en) Task distribution method, device, computer equipment and storage medium
CN111428885B (en) User indexing method in federated learning and federated learning device
CN111813523A (en) Duration pre-estimation model generation method, system resource scheduling method, device, electronic equipment and storage medium
Mekala et al. A DRL-Based Service Offloading Approach Using DAG for Edge Computational Orchestration
CN108428051B (en) MapReduce job scheduling method and device facing big data platform and based on maximized benefits
CN108200185B (en) Method and device for realizing load balance
CN113553160A (en) Task scheduling method and system for edge computing node of artificial intelligence Internet of things
CN110795203A (en) Resource scheduling method, device and system and computing equipment
CN112491742B (en) Method and device for scheduling edge computing terminal resources
CN108805397A (en) Electronic device, the method and storage medium for distributing task
Farooq et al. Adaptive and resilient revenue maximizing dynamic resource allocation and pricing for cloud-enabled IoT systems
CN112559147A (en) Dynamic matching algorithm, system and equipment based on GPU resource occupation characteristics

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination