CN107918561B - Task allocation method in industrial distributed data acquisition system - Google Patents

Task allocation method in industrial distributed data acquisition system Download PDF

Info

Publication number
CN107918561B
CN107918561B CN201711142542.2A CN201711142542A CN107918561B CN 107918561 B CN107918561 B CN 107918561B CN 201711142542 A CN201711142542 A CN 201711142542A CN 107918561 B CN107918561 B CN 107918561B
Authority
CN
China
Prior art keywords
acquisition
task
node
tasks
initial
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201711142542.2A
Other languages
Chinese (zh)
Other versions
CN107918561A (en
Inventor
徐泉
冉振莉
张志强
王良勇
柴天佑
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Northeastern University China
Original Assignee
Northeastern University China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Northeastern University China filed Critical Northeastern University China
Priority to CN201711142542.2A priority Critical patent/CN107918561B/en
Publication of CN107918561A publication Critical patent/CN107918561A/en
Application granted granted Critical
Publication of CN107918561B publication Critical patent/CN107918561B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/5044—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering hardware capabilities
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083—Techniques for rebalancing the load in a distributed system

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention provides a task allocation method in an industrial distributed data acquisition system, which comprehensively considers multiple factors such as acquisition node resource utilization rate, acquisition efficiency, load balance, acquisition reliability and heterogeneous acquisition nodes, establishes an initial acquisition node number optimization model and an initial task allocation optimization model based on simulation, optimizes the number of working acquisition nodes according to the actual configuration and the use condition of each acquisition node, improves the acquisition resource utilization rate, realizes the initial allocation of tasks including 1:1 redundant tasks and non-redundant tasks under different conditions of the resource configuration and the resource use condition of each acquisition node in the industrial distributed data acquisition system, and meets the requirements of industrial distributed data acquisition on acquisition instantaneity, reliability, effective resource utilization and the like under the industrial big data environment.

Description

Task allocation method in industrial distributed data acquisition system
Technical Field
The invention belongs to the technical field of data acquisition, and particularly relates to a task allocation method in an industrial distributed data acquisition system.
Background
With the arrival of industrial big data environment, data sources are increasingly diversified and data scale is increased in the industrial process, and in the face of industrial large-scale high-frequency data acquisition and some new application requirements, more and more enterprises begin to consider adopting a distributed system to carry out industrial data acquisition in order to ensure the time sequence, real-time performance and reliability of data acquisition. In the design process of the distributed data acquisition system, the task allocation strategy is a very critical link, and the resource utilization efficiency and the data acquisition efficiency of the acquisition system are directly influenced.
In a distributed environment, because each acquisition node can be flexibly added or withdrawn, the data acquired by each acquisition node does not correspond to a fixed production link any more, but has certain flexibility, and because of the self reason of each acquisition node, the resource allocation condition and the use condition are probably different at the initial task allocation moment, so that some problems are caused under the condition, how to select the acquisition nodes which initially work to complete the given acquisition task by adopting the minimum nodes can save other expenses such as system resources and electric quantity to the maximum extent, how to effectively allocate the data acquisition tasks of each production link to each heterogeneous acquisition node for parallel acquisition, and ensure that the load of each acquisition node in the acquisition process is balanced as much as possible, thereby improving the resource utilization efficiency and the acquisition efficiency to the maximum extent is a problem to be solved, if the tasks are distributed unreasonably, load imbalance among the nodes is caused, the efficiency of data acquisition is directly influenced, and the time sequence of the acquired data is further influenced, so that subsequent real-time data monitoring and correlation analysis cannot be realized; in addition, because the data reliability requirement of an industrially important production link is strict, sometimes the importance degrees of different data of the same production link are different, the collection task of important data usually needs to be backed up, the backup task only collects and does not store the important data, and only the main task fails to collect the important data, so that how to reasonably distribute the common task and the important task in a distributed environment meets the load balance, and meanwhile, the main task and the redundant task of the important task are not distributed in the same collection node, which is another problem to be considered in the industrial data collection.
At present, patents in the aspect of an industrial distributed data acquisition system mainly include "CN 105527948A (a large-scale distributed data acquisition system and method based on an industrial process)", "cn201610522950. x (a large-scale distributed data acquisition system and method)", "CN 201610736266.1 (a large-scale distributed intelligent data acquisition system and method based on an industrial cloud)", and "201610622589.8 (a task scheduling mechanism and system based on a consistent hash algorithm)". Patent CN105527948A adopts a client selecting module to averagely distribute the collection tasks of multiple field control stations to multiple collection clients according to the number of field control stations, but when the number of collection tasks in each field control station is different, the distribution method cannot realize load balancing. The patent CN201610522950.X and the patent CN201610736266.1 both adopt a cyclic sharing mode to sequentially and cyclically share the collection tasks set by the user to each collection node, and distribute the first backup task and the second backup task to the next two adjacent number collection nodes in a cyclic manner; the patent 201610622589.8 discloses that the hash values of the available execution units and the hash values of the tasks are distributed on the same consistent hash ring, and the search direction is selected according to the number n of execution times of the tasks, and the tasks are distributed to the n corresponding execution units closest to the hash values of the tasks on the consistent hash ring according to the search direction, so as to achieve random distribution of the tasks. Although the above patent can realize effective allocation of tasks, it is only for the situation that the resource usage of each acquisition node is the same at the initial allocation time, and for heterogeneous acquisition nodes with different resource allocation or usage, the above allocation scheme is not applicable, and the allocation method of the prior patent is only for the situation that all tasks are important tasks, all need to be backed up, or all tasks are not backed up, and in the actual industrial production process, in order to improve the resource utilization rate and simultaneously meet the acquisition reliability of the important tasks in the acquisition tasks, the acquisition tasks corresponding to part of the important data in the acquisition tasks are often backed up, other tasks are not backed up, the task allocation method of the prior patent cannot realize load balancing of each acquisition node under such a situation, and the prior patent does not provide a specific optimization method of the number of the initial acquisition nodes in work, easily causing the waste of acquisition resources.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a task allocation method in an industrial distributed data acquisition system, so as to achieve the purposes of improving the resource utilization rate, the acquisition efficiency and the acquisition reliability of the industrial data acquisition system.
The technical scheme adopted by the invention is as follows:
a task allocation method in an industrial distributed data acquisition system comprises the following steps:
step 1, establishing a corresponding relation between acquisition resources of each acquisition node and an acquisition task;
step 2, establishing a corresponding relation between the acquisition time of each acquisition node and the resource utilization rate corresponding to the acquisition of different numbers of acquisition tasks;
step 3, optimizing the number of acquisition nodes in the initial work, and selecting and starting the acquisition nodes in the initial work;
and 4, establishing a task allocation optimization model and performing initial allocation of the acquisition tasks.
The step 1 comprises the following steps:
1.1, distributing acquisition tasks to each acquisition node independently, changing the number of the acquisition tasks, and measuring acquisition resource data consumed on the ith acquisition node when a large number of acquisition tasks with different numbers are acquired;
step 1.2, collecting task data T on the ith collecting node obtained in step 1.1iAnd corresponding resource consumption data
Figure BDA0001471732400000021
Fitting to obtain
Figure BDA0001471732400000022
And TiFunctional relation of
Figure BDA0001471732400000023
The step 2 comprises the following steps:
step 2.1, distributing acquisition tasks to each acquisition node independently, changing the number of the acquisition tasks, and measuring acquisition resource utilization rate data corresponding to a large number of acquisition tasks with different numbers acquired on each node and acquisition time data required for executing all the acquisition tasks distributed on the node once;
step 2.2, collecting T on each node obtained in step 2.1iAcquisition resource utilization rate u corresponding to each acquisition taskiAnd the collection timeiFitting to obtain the acquisition time on the ith acquisition nodeiAnd collect T on the nodeiAcquisition resource utilization rate u corresponding to each acquisition taskiTime of function relationi＝g(ui)。
The step 3 comprises the following steps:
3.1, establishing an optimization model of the number of acquisition nodes of initial work according to the simulation results of the step 1 and the step 2, and considering acquisition period constraint and acquisition resource utilization rate constraint according to different resource allocation of the acquisition nodes and different resource use conditions of the acquisition nodes at the initial task allocation moment to minimize the number n of the acquisition nodes of the initial work; the optimization model is as follows:
min n
Figure BDA0001471732400000031
Figure BDA0001471732400000032
timei≤Time i＝1,2...n (1-3)
timei＝g(ui)i＝1,2...n (1-4)
Figure BDA0001471732400000033
wherein formula (1-1) represents the resource utilization u of the ith collection nodeiCannot exceed the resource utilization threshold, u, set by the useri0Indicating the resource utilization rate, R, of the ith node before the task is allocatediRepresents the total amount of available resources, u, configured on the ith collection nodehRepresents the upper limit threshold value of the node resource utilization rate, u, set by the userwRepresenting the threshold bandwidth of the node resource utilization rate set by a user; expression (1-3) represents the acquisition time of the ith acquisition node for executing all the allocated acquisition tasks onceiThe acquisition period Time set by a user cannot be exceeded; the expression (1-5) represents the number T of the collection tasks distributed to all the collection nodesiThe sum of the total number of the tasks to be distributed is M; the formulas (1-2) and (1-4) are respectively simulation results of the step 1 and the step 2;
and 3.2, solving the optimization model in the step 3.1 to obtain the number n of the collection nodes in initial work, starting the corresponding nodes, and taking the rest nodes as the collection nodes of the redundant hot standby.
The step 4 comprises the following steps:
step 4.1, numbering the initial acquisition tasks according to the priority sequence;
4.2, extracting redundant tasks according to the important tasks, wherein the number of each redundant task is the same as that of the initial task;
4.3, establishing an initial task allocation optimization model according to the solving result of the step 3:
Figure BDA0001471732400000034
Figure BDA0001471732400000035
Figure BDA0001471732400000041
timei≤Time i＝1,2...n (1-9)
timei＝g(ui)i＝1,2...n (1-10)
Figure BDA0001471732400000042
Figure BDA0001471732400000043
Figure BDA0001471732400000044
Figure BDA0001471732400000045
Figure BDA0001471732400000046
Figure BDA0001471732400000047
Figure BDA0001471732400000048
wherein (1-6) represents that the sum of load differences delta U between all the acquisition nodes is minimized, so that all the acquisition nodes are in load balance as much as possible; the formula (1-7) represents the resource utilization rate u of the ith collection nodeiThe resource utilization rate threshold set by the user cannot be exceeded; the formula (1-9) represents the acquisition time of the acquisition task which is distributed once and is executed on the ith acquisition nodeiThe acquisition period Time set by a user cannot be exceeded; the expression (1-11) represents the number T of the collection tasks distributed to all the collection nodesiThe sum of the total number of the tasks to be distributed is M; the formulas (1-8) and (1-10) are respectively simulation results of the step 1 and the step 2; the total number of the initial tasks is p represented by the formulas (1-12), wherein taskkRepresenting an initial task numbered k; the expressions (1-13) indicate that the number of important tasks in the initial task is q, namely the total number of redundant tasks is q, wherein,
Figure BDA0001471732400000049
indicating a redundant task with the number l; the formula (1-14) represents
Figure BDA00014717324000000410
When the initial task with the number l has a redundant task,
Figure BDA00014717324000000411
when there is no redundant task for the initial task numbered i,
Figure BDA00014717324000000412
Trrepresenting a set of all redundant task sequence numbers; the sum of the number of the initial tasks and the number of the redundant tasks distributed to the ith acquisition node is T as shown in the formula (1-15)iWherein l ≠ k initial tasks with same constraint number and redundant tasks thereof cannot be allocated to the same acquisition nodek,iRepresentation is assigned toThe ith acquisition node is numbered k,
Figure BDA00014717324000000413
representing a redundant task with the number l allocated to the ith acquisition node; the formula (1-16) represents taskk,iWhen the initial task with the number of k is distributed to the ith acquisition node, task is carried outk,iWhen the initial task with the number k is not distributed to the ith acquisition node, task 1k,i＝0，TaskiRepresenting a collection task sequence number set distributed to the ith collection node; formula (1-17) represents
Figure BDA00014717324000000414
When the redundant task with the number l is distributed to the ith collection node,
Figure BDA00014717324000000415
when the redundant task with the number l is not distributed to the ith collection node,
Figure BDA00014717324000000416
step 4.4, solving the optimization model in the step 4.3, and solving a Task sequence number set Task distributed to the ith acquisition nodei。
And 4.5, distributing the acquisition tasks to each acquisition node according to the solution result of the step 4.4.
The invention has the advantages that:
compared with the prior art, the invention has the following advantages: the invention provides a task allocation method in an industrial distributed data acquisition system, which comprehensively considers multiple factors such as acquisition node resource utilization rate, acquisition efficiency, load balance, acquisition reliability and heterogeneous acquisition nodes, establishes an initial acquisition node number optimization model and an initial task allocation optimization model based on simulation, optimizes the number of working acquisition nodes according to the actual configuration and the use condition of each acquisition node, improves the acquisition resource utilization rate, realizes the initial allocation of tasks including 1:1 redundant tasks and non-redundant tasks under different conditions of the resource configuration and the resource use condition of each acquisition node in the industrial distributed data acquisition system, and meets the requirements of industrial distributed data acquisition on acquisition instantaneity, reliability, effective resource utilization and the like under the industrial big data environment.
Drawings
Fig. 1 is a flowchart of a task allocation method in an industrial distributed data acquisition system according to an embodiment of the present invention.
Detailed Description
An embodiment of the present invention is further described below with reference to fig. 1.
A task allocation method in an industrial distributed data acquisition system comprises the following steps:
in the embodiment of the invention, 4 acquisition nodes are arranged, and the numbers are 1,2,3 and 4. And setting 36 initial acquisition tasks which respectively correspond to the acquisition of data on the 36 fans F1-F36. The data on the 12 fans of F11-F22 are important collection tasks and need to be backed up. Each acquisition task contains 404 acquisition data items, wherein 31 Single floor type data items, 52 Double floor type data items, 319 Boolean type data items and 2 unidentified Integer type data items.
Step 1, establishing a corresponding relation between acquisition resources of each acquisition node and acquisition tasks
1.1, distributing acquisition tasks to each acquisition node independently, changing the number of the acquisition tasks, and measuring acquisition resource data consumed on the ith acquisition node when a large number of acquisition tasks with different numbers are acquired;
in the embodiment of the invention, the collected resource data refers to memory resources.
In the embodiment of the present invention, acquisition tasks are respectively allocated to acquisition nodes No. 1,2,3, and 4, memory resources consumed when acquiring different numbers of acquisition tasks are measured, and data obtained by measurement are respectively shown in table 1, table 2, table 3, and table 4:
table 1, No. 1 collection node consumes memory resources and collects task data
Number of collection tasks 1 2 3 4 5 6 7 8 9 10 11 12
Consume memory resources (G) 0.19 0.37 0.55 0.76 0.97 1.15 1.33 1.53 1.72 1.91 2.11 2.29
Table 2, No. 2 collection node consumes memory resource and collects task data
Figure BDA0001471732400000051
Figure BDA0001471732400000061
Table 3, 3 acquisition node consumes memory resources and acquires task data
Number of collection tasks 1 2 3 4 5 6 7 8 9 10 11 12
Consume memory resources (G) 0.18 0.38 0.59 0.77 0.98 1.17 1.34 1.52 1.72 1.88 2.09 2.27
Table 4, No. 4 collection node consumes memory resource and collects task data
Number of collection tasks 1 2 3 4 5 6 7 8 9 10 11 12
Consume memory resources (G) 0.19 0.37 0.56 0.76 0.97 1.15 1.34 1.54 1.73 1.91 2.10 2.27
Number of collection tasks 13 14 15 16 17 18 19 20 21 22 23 24
Consume memory resources (G) 2.46 2.65 2.85 3.04 3.23 3.43 3.61 3.81 3.99 4.18 4.37 4.57
Number of collection tasks 25 26 27 28 29 30 31 32 33 34 35 36
Consume memory resources (G) 4.76 4.96 5.14 5.32 5.52 5.70 5.89 6.10 6.28 6.46 6.66 6.87
Number of collection tasks 37 38 39 40
Consume memory resources (G) 7.05 7.24 7.42 7.61
Step 1.2, collecting task data T on the ith collecting node obtained in step 1.1iAnd corresponding resource consumption data
Figure BDA0001471732400000062
Fitting to obtain
Figure BDA0001471732400000063
And TiFunctional relation of
Figure BDA0001471732400000064
In the embodiment of the invention, No. 1 acquisition node obtained by fitting
Figure BDA0001471732400000065
And T1Has a functional relation of
Figure BDA0001471732400000066
No. 2 acquisition node
Figure BDA0001471732400000067
And T2Has a functional relation of
Figure BDA0001471732400000068
No. 3 acquisition node
Figure BDA0001471732400000069
And T3Has a functional relation of
Figure BDA00014717324000000610
No. 3 acquisition node
Figure BDA00014717324000000611
And T4Has a functional relation of
Figure BDA00014717324000000612
Step 2, establishing a corresponding relation between the acquisition time of each acquisition node and the resource utilization rate corresponding to the acquisition of different numbers of acquisition tasks
Step 2.1, distributing acquisition tasks to each acquisition node independently, changing the number of the acquisition tasks, and measuring acquisition resource utilization rate data corresponding to a large number of acquisition tasks with different numbers acquired on each node and acquisition time data required for executing all the acquisition tasks distributed on the node once;
in the embodiment of the present invention, acquisition tasks are separately allocated to acquisition nodes No. 1,2,3, and 4, and the memory usage rates and acquisition time data of the acquisition nodes corresponding to different numbers of the acquisition tasks are measured, and the measured data are shown in tables 5, 6, 7, and 8:
TABLE 5 No. 1 Collection node collects task number, memory usage and collection time data
Number of collection tasks 1 2 3 4 5 6 7 8 9 10 11 12
Memory usage (%) 14.75 19.25 23.75 29 34.25 38.75 43.25 48.25 53 57.25 62.75 67.25
Acquisition time (ms) 305 330 357 390 427 469 508 559 607 661 728 788
TABLE 6, No. 2 acquisition node number of task, memory utilization rate and acquisition time data
Figure BDA0001471732400000071
TABLE 7, No. 3 acquisition node number of task, memory utilization rate and acquisition time data
Number of collection tasks 1 2 3 4 5 6 7 8 9 10 11 12
Memory usage (%) 17.5 22.5 27.75 32.25 37.5 42.25 46.5 51 56 60 65.25 69.75
Acquisition time (ms) 319 346 380 415 454 498 539 589 647 694 763 824
Table 84 number collection node collects task number, memory utilization rate and collection time data
Figure BDA0001471732400000072
Figure BDA0001471732400000081
Step 2.2, collecting T on each node obtained in step 2.1iAcquisition resource utilization rate u corresponding to each acquisition taskiAnd the collection timeiFitting to obtain the acquisition time on the ith acquisition nodeiAnd collect T on the nodeiAcquisition resource utilization rate u corresponding to each acquisition taskiTime of function relationi＝g(ui)
In the embodiment of the invention, the data obtained in the step 2.1 is fitted by adopting a least square method, and the No. 1 acquisition node time obtained by fitting1And u1Has a functional relation of time1＝0.0869u1 2+2.0390u1+257.6073, No. 2 collecting node time2And u2Has a functional relation of time2＝0.1370u2 2-0.2669u2+161.6201, No. 3 acquisition node time3And u3Has a functional relation of time3＝0.0880u3 2+2.0146u3+256.5218, No. 4 acquisition node time4And u4Has a functional relation of time4＝0.3159u4 2-10.3928u4+190.3659
Step 3, optimizing the number of the collection nodes of the initial work, selecting and starting the collection nodes of the initial work
And 3.1, establishing an optimization model of the number of the acquisition nodes of the initial work according to the simulation results of the step 1 and the step 2, and considering acquisition period constraint and acquisition resource utilization rate constraint according to different resource allocation of the acquisition nodes and different resource use conditions of the acquisition nodes at the initial task allocation moment to minimize the number n of the acquisition nodes of the initial work. The optimization model is as follows:
min n
Figure BDA0001471732400000082
Figure BDA0001471732400000083
timei≤Time i＝1,2...n (2-3)
timei＝g(ui)i＝1,2...n (2-4)
Figure BDA0001471732400000091
wherein formula (1-1) represents the resource utilization u of the ith collection nodeiCannot exceed the resource utilization threshold, u, set by the useri0Indicating the resource utilization rate, R, of the ith node before the task is allocatediRepresents the total amount of available resources, u, configured on the ith collection nodehRepresents the upper limit threshold value of the node resource utilization rate, u, set by the userwRepresenting the threshold bandwidth of the node resource utilization rate set by a user; expression (1-3) represents the acquisition time of the ith acquisition node for executing all the allocated acquisition tasks onceiThe acquisition period Time set by a user cannot be exceeded; the expression (1-5) represents the number T of the collection tasks distributed to all the collection nodesiThe sum of the total number of the tasks to be distributed is M; the formulas (1-2) and (1-4) are simulation results of the step 1 and the step 2, respectively.
In the examples of the present invention, R1＝4G，R2＝8G，R3＝4G，R4＝16G，uh＝80％，uw＝5％，u10＝10％，u20＝15％，u30＝13％，u40＝27％，Time＝1000ms，M＝48
The optimization model is as follows:
min n
Figure BDA0001471732400000092
Figure BDA0001471732400000093
Figure BDA0001471732400000094
Figure BDA0001471732400000095
Figure BDA0001471732400000096
Figure BDA0001471732400000097
Figure BDA0001471732400000098
Figure BDA0001471732400000099
time1≤1000 (2-14)
time2≤1000 (2-15)
time3≤1000 (2-16)
time4≤1000 (2-17)
time1＝0.0869u1 2+2.0390u1+257.6073 (2-18)
time2＝0.1370u2 2-0.2669u2+161.6201 (2-19)
time3＝0.0880u3 2+2.0146u3+256.5218 (2-20)
time4＝0.3159u4 2-10.3928u4+190.3659 (2-21)
Figure BDA0001471732400000101
and 3.2, solving the optimization model in the step 3.1 to obtain the number n of the collection nodes in initial work, starting the corresponding nodes, and taking the rest nodes as the collection nodes of the redundant hot standby.
In the embodiment of the invention, the number n of the acquired acquisition nodes which initially work is 2, the acquisition nodes are respectively a No. 3 acquisition node and a No. 4 acquisition node, and the No. 1 acquisition node and the No. 2 acquisition node are used as redundant hot standby acquisition nodes.
Step 4, establishing a task allocation optimization model, and performing initial allocation of the acquisition tasks;
step 4.1, numbering the initial acquisition tasks according to the priority sequence;
in the embodiment of the invention, the priority of 36 initial acquisition tasks is set as 1, and the 36 acquisition tasks are numbered as 1,2, 3.. 36 respectively.
And 4.2, extracting redundant tasks according to the important tasks, wherein the number of each redundant task is the same as that of the initial task.
In the embodiment of the invention, the collection tasks F11-F22 are important collection tasks and need to be subjected to redundant backup. The redundant tasks are respectively numbered 11*,12*,13*,14*,15*,16*,17*,18*,19*,20*,21*,22*。
4.3, establishing an initial task allocation optimization model according to the solving result of the step 3:
Figure BDA0001471732400000102
Figure BDA0001471732400000103
Figure BDA0001471732400000104
timei≤Time i＝1,2...n (2-25)
timei＝g(ui)i＝1,2...n (2-26)
Figure BDA0001471732400000111
Figure BDA0001471732400000112
Figure BDA0001471732400000113
Figure BDA0001471732400000114
Figure BDA0001471732400000115
Figure BDA0001471732400000116
Figure BDA0001471732400000117
wherein (2-22) represents that the sum of load differences delta U between all the acquisition nodes is minimized, so that all the acquisition nodes are in load balance as much as possible; the equation (2-23) represents the resource utilization rate u of the ith collection nodeiThe resource utilization rate threshold set by the user cannot be exceeded; the expression (2-25) represents the acquisition time of the acquisition node executing all the allocated acquisition tasks onceiThe acquisition period Time set by a user cannot be exceeded; the expression (2-27) represents the number T of acquisition tasks distributed to all acquisition nodesiThe sum of the total number of the tasks to be distributed is M; equations (2-24) and (2-26) are the simulation results of step 1 and step 2, respectively; the expression (2-28) represents that the total number of the initial tasks is p, wherein taskkRepresenting an initial task numbered k; the expression (2-29) indicates that the number of important tasks in the initial task is q, namely the total number of redundant tasks is q, wherein,
Figure BDA0001471732400000118
indicating a redundant task with the number l; the formula (2-30) represents
Figure BDA0001471732400000119
When the initial task with the number l has a redundant task,
Figure BDA00014717324000001110
when there is no redundant task for the initial task numbered i,
Figure BDA00014717324000001111
Trrepresenting a set of all redundant task sequence numbers; the sum of the number of the initial tasks and the number of the redundant tasks distributed to the ith acquisition node is T as shown in the formula (2-31)iWherein l ≠ k initial tasks with same constraint number and redundant tasks thereof cannot be allocated to the same acquisition nodek,iIndicating the initial task assigned to the ith acquisition node with number k,
Figure BDA00014717324000001112
representation is assigned toThe ith collection node is numbered as a redundant task with a number l; the formula (2-32) represents taskk,iWhen the initial task with the number of k is distributed to the ith acquisition node, task is carried outk,iWhen the initial task with the number k is not distributed to the ith acquisition node, task 1k,i＝0，TaskiRepresenting a collection task sequence number set distributed to the ith collection node; formula (2-33) represents
Figure BDA0001471732400000121
When the redundant task with the number l is distributed to the ith collection node,
Figure BDA0001471732400000122
when the redundant task with the number l is not distributed to the ith collection node,
Figure BDA0001471732400000123
in the embodiment of the invention, n is 2, i is 3,4, R3＝4G，R4＝16G，uh＝80％，uw＝5％，
u30＝13％，u40＝27％，Time＝1000ms，M＝48，p＝36，q＝12，
Tr＝{11*,12*,13*,14*,15*,16*,17*,18*,19*,20*,21*,22*}，
Figure BDA0001471732400000124
Figure BDA0001471732400000125
Figure BDA0001471732400000126
Figure BDA0001471732400000127
Figure BDA0001471732400000128
time3≤1000 (2-39)
time4≤1000 (2-40)
time3＝0.0880u3 2+2.0146u3+256.5218 (2-41)
time4＝0.3159u4 2-10.3928u4+190.3659 (2-42)
T3+T4＝48 (2-43)
Figure BDA0001471732400000129
Figure BDA00014717324000001210
Figure BDA00014717324000001211
Figure BDA00014717324000001212
Figure BDA0001471732400000131
Figure BDA0001471732400000132
Figure BDA0001471732400000133
Figure BDA0001471732400000134
Figure BDA0001471732400000135
Step 4.4, solving the model to obtain a group of optimal feasible solutions, namely the Task sequence number set Task distributed to the ith acquisition nodei。
In the embodiment of the invention, the model is solved by adopting a genetic algorithm, and the obtained task sequence number set distributed to the No. 3 acquisition node is {11*,12*,13*,14*,15*,16*,17*,18*,19*,20*,21*,22*And the task sequence number set distributed to the No. 4 acquisition node is {1-36 }.
And 4.5, distributing the acquisition tasks to each acquisition node according to the solution result of the step 4.4.
In the embodiment of the invention, the number 1-36 initial acquisition tasks are all distributed to the number 4 acquisition nodes, and the number 11-22 acquisition tasks are simultaneously distributed to the number 3 acquisition nodes for redundant backup.

Claims (3)

1. A task allocation method in an industrial distributed data acquisition system is characterized by comprising the following steps:
step 1, establishing a corresponding relation between acquisition resources of each acquisition node and an acquisition task;
step 2, establishing a corresponding relation between the acquisition time of each acquisition node and the resource utilization rate corresponding to the acquisition of different numbers of acquisition tasks;
step 3, optimizing the number of acquisition nodes in the initial work, and selecting and starting the acquisition nodes in the initial work;
step 4, establishing a task allocation optimization model, and performing initial allocation of the acquisition tasks;
the step 3 comprises the following steps:
3.1, establishing an optimization model of the number of acquisition nodes of initial work according to the simulation results of the step 1 and the step 2, and considering acquisition period constraint and acquisition resource utilization rate constraint according to different resource allocation of the acquisition nodes and different resource use conditions of the acquisition nodes at the initial task allocation moment to minimize the number n of the acquisition nodes of the initial work; the optimization model is as follows:
min n
Figure FDA0002941271540000011
Figure FDA0002941271540000012
timei≤Time i＝1,2...n (1-3)
timei＝g(ui) i＝1,2...n (1-4)
Figure FDA0002941271540000013
wherein formula (1-1) represents the resource utilization u of the ith collection nodeiCannot exceed the resource utilization threshold, u, set by the useri0Indicating the resource utilization rate, R, of the ith node before the task is allocatediRepresents the total amount of available resources, u, configured on the ith collection nodehRepresents the upper limit threshold value of the node resource utilization rate, u, set by the userwRepresenting the threshold bandwidth of the node resource utilization rate set by a user; expression (1-3) represents the acquisition time of the ith acquisition node for executing all the allocated acquisition tasks onceiThe acquisition period Time set by a user cannot be exceeded; the expression (1-5) represents the number T of the collection tasks distributed to all the collection nodesiThe sum of the total number of the tasks to be distributed is M; the formulas (1-2) and (1-4) are respectively simulation results of the step 1 and the step 2;
step 3.2, solving the optimization model in the step 3.1 to obtain the number n of the collection nodes in initial work, starting the corresponding nodes, and taking the rest nodes as the collection nodes of the redundant hot standby;
the step 4 comprises the following steps:
step 4.1, numbering the initial acquisition tasks according to the priority sequence;
4.2, extracting redundant tasks according to the important tasks, wherein the number of each redundant task is the same as that of the initial task;
4.3, establishing an initial task allocation optimization model according to the solving result of the step 3:
Figure FDA0002941271540000021
Figure FDA0002941271540000022
Figure FDA0002941271540000023
timei≤Time i＝1,2...n (1-9)
timei＝g(ui) i＝1,2...n (1-10)
Figure FDA0002941271540000024
Figure FDA0002941271540000025
Figure FDA0002941271540000026
Figure FDA0002941271540000027
Figure FDA0002941271540000028
Figure FDA0002941271540000029
Figure FDA00029412715400000210
wherein (1-6) represents that the sum of load differences delta U between all the acquisition nodes is minimized, so that all the acquisition nodes are in load balance as much as possible; the formula (1-7) represents the resource utilization rate u of the ith collection nodeiThe resource utilization rate threshold set by the user cannot be exceeded; the formula (1-9) represents the acquisition time of the acquisition task which is distributed once and is executed on the ith acquisition nodeiThe acquisition period Time set by a user cannot be exceeded; the expression (1-11) represents the number T of the collection tasks distributed to all the collection nodesiThe sum of the total number of the tasks to be distributed is M; the formulas (1-8) and (1-10) are respectively simulation results of the step 1 and the step 2; the total number of the initial tasks is p represented by the formulas (1-12), wherein taskkRepresenting an initial task numbered k; the expressions (1-13) indicate that the number of important tasks in the initial task is q, namely the total number of redundant tasks is q, wherein,
Figure FDA00029412715400000211
indicating a redundant task with the number l; the formula (1-14) represents
Figure FDA00029412715400000212
When the initial task with the number l has a redundant taskWhen the temperature of the water is higher than the set temperature,
Figure FDA00029412715400000213
when there is no redundant task for the initial task numbered i,
Figure FDA00029412715400000214
Trrepresenting a set of all redundant task sequence numbers; the sum of the number of the initial tasks and the number of the redundant tasks distributed to the ith acquisition node is T as shown in the formula (1-15)iWherein l ≠ k initial tasks with same constraint number and redundant tasks thereof cannot be allocated to the same acquisition nodek,iIndicating the initial task assigned to the ith acquisition node with number k,
Figure FDA00029412715400000215
representing a redundant task with the number l allocated to the ith acquisition node; the formula (1-16) represents taskk,iWhen the initial task with the number of k is distributed to the ith acquisition node, task is carried outk,iWhen the initial task with the number k is not distributed to the ith acquisition node, task 1k,i＝0，TaskiRepresenting a collection task sequence number set distributed to the ith collection node; formula (1-17) represents
Figure FDA0002941271540000031
When the redundant task with the number l is distributed to the ith collection node,
Figure FDA0002941271540000032
when the redundant task with the number l is not distributed to the ith collection node,
Figure FDA0002941271540000033
step 4.4, solving the optimization model in the step 4.3, and solving a Task sequence number set Task distributed to the ith acquisition nodei；
And 4.5, distributing the acquisition tasks to each acquisition node according to the solution result of the step 4.4.
2. The method of claim 1, wherein the task assignment method in the industrial distributed data acquisition system,
the step 1 comprises the following steps:
1.1, distributing acquisition tasks to each acquisition node independently, changing the number of the acquisition tasks, and measuring acquisition resource data consumed on the ith acquisition node when a large number of acquisition tasks with different numbers are acquired;
step 1.2, collecting task data T on the ith collecting node obtained in step 1.1iAnd corresponding resource consumption data
Figure FDA0002941271540000034
Fitting to obtain
Figure FDA0002941271540000036
And TiFunctional relation of
Figure FDA0002941271540000035
3. The method of claim 1, wherein the task assignment method in the industrial distributed data acquisition system,
the step 2 comprises the following steps:
step 2.1, distributing acquisition tasks to each acquisition node independently, changing the number of the acquisition tasks, and measuring acquisition resource utilization rate data corresponding to a large number of acquisition tasks with different numbers acquired on each node and acquisition time data required for executing all the acquisition tasks distributed on the node once;
step 2.2, collecting T on each node obtained in step 2.1iAcquisition resource utilization rate u corresponding to each acquisition taskiAnd the collection timeiFitting to obtain the acquisition time on the ith acquisition nodeiAnd collect T on the nodeiAcquisition resource utilization rate u corresponding to each acquisition taskiTime of function relationi＝g(ui)。
CN201711142542.2A 2017-11-17 2017-11-17 Task allocation method in industrial distributed data acquisition system Active CN107918561B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201711142542.2A CN107918561B (en) 2017-11-17 2017-11-17 Task allocation method in industrial distributed data acquisition system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201711142542.2A CN107918561B (en) 2017-11-17 2017-11-17 Task allocation method in industrial distributed data acquisition system

Publications (2)

Publication Number Publication Date
CN107918561A CN107918561A (en) 2018-04-17
CN107918561B true CN107918561B (en) 2021-03-26

Family

ID=61896582

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201711142542.2A Active CN107918561B (en) 2017-11-17 2017-11-17 Task allocation method in industrial distributed data acquisition system

Country Status (1)

Country Link
CN (1) CN107918561B (en)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108536539B (en) * 2018-04-26 2021-12-17 东北大学 Task scheduling method in industrial distributed data acquisition system
CN109101333A (en) * 2018-06-27 2018-12-28 北京蜂盒科技有限公司 Image characteristic extracting method, device, storage medium and electronic equipment
CN109120688A (en) * 2018-08-10 2019-01-01 北京天安智慧信息技术有限公司 Distributed acquisition method for industrial real-time data
CN111400045B (en) * 2020-03-16 2023-09-05 杭州海康威视系统技术有限公司 Load balancing method and device
CN113268376B (en) * 2021-04-16 2023-01-31 济南轨道交通集团有限公司 Data center object storage method and system based on genetic algorithm
CN114124958A (en) * 2021-11-26 2022-03-01 智昌科技集团股份有限公司 Operation method of distributed data acquisition system

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101706742A (en) * 2009-11-20 2010-05-12 北京航空航天大学 Method for dispatching I/O of asymmetry virtual machine based on multi-core dynamic partitioning
CN101923493A (en) * 2010-09-21 2010-12-22 深圳市华力特电气股份有限公司 Task scheduling method and task scheduling equipment
CN102799945A (en) * 2012-06-21 2012-11-28 浙江工商大学 Service process execution cost minimization-oriented integrated resource quantity configuration and task distribution optimization method
CN104111983A (en) * 2014-06-30 2014-10-22 中国科学院信息工程研究所 Open-type multi-source data collection system and method
CN105527948A (en) * 2015-12-11 2016-04-27 东北大学 Large scale distributed data acquisition system and method based on industrial process

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8566837B2 (en) * 2010-07-16 2013-10-22 International Business Machines Corportion Dynamic run time allocation of distributed jobs with application specific metrics

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101706742A (en) * 2009-11-20 2010-05-12 北京航空航天大学 Method for dispatching I/O of asymmetry virtual machine based on multi-core dynamic partitioning
CN101923493A (en) * 2010-09-21 2010-12-22 深圳市华力特电气股份有限公司 Task scheduling method and task scheduling equipment
CN102799945A (en) * 2012-06-21 2012-11-28 浙江工商大学 Service process execution cost minimization-oriented integrated resource quantity configuration and task distribution optimization method
CN104111983A (en) * 2014-06-30 2014-10-22 中国科学院信息工程研究所 Open-type multi-source data collection system and method
CN105527948A (en) * 2015-12-11 2016-04-27 东北大学 Large scale distributed data acquisition system and method based on industrial process

Also Published As

Publication number Publication date
CN107918561A (en) 2018-04-17

Similar Documents

Publication Publication Date Title
CN107918561B (en) Task allocation method in industrial distributed data acquisition system
CN108536539B (en) Task scheduling method in industrial distributed data acquisition system
Li et al. An improved max-min task-scheduling algorithm for elastic cloud
CN102063336B (en) Distributed computing multiple application function asynchronous concurrent scheduling method
CN105656973B (en) Method for scheduling task and system in a kind of distributed node group
CN105653204B (en) A kind of distributed figure calculation method based on disk
CN103679388B (en) Production scheduling method and production scheduling system
CN102508714A (en) Green-computer-based virtual machine scheduling method for cloud computing
CN108572873A (en) A kind of load-balancing method and device solving the problems, such as Spark data skews
CN103077083A (en) Method and system for processing tasks
CN104536803A (en) Virtual machine scheduling method based on combination optimization
CN107704069A (en) A kind of Spark energy-saving scheduling methods perceived based on energy consumption
CN104808770A (en) Data center energy consumption management method and system based on dynamic frequency modulation
CN105303452A (en) Power-grid regulation and control real-time data distributed processing method and system
CN103957280A (en) Connection allocation and scheduling method of sensor network in Internet of things
CN103064745A (en) Method and system for distributing and processing tasks
CN101551661A (en) Control method for multi-robot system
CN105577729A (en) Load balancing method and device
CN105550825B (en) Flexible factory job scheduling method based on MapReduce parallelization in cloud computing environment
CN107070965B (en) Multi-workflow resource supply method under virtualized container resource
CN104965762A (en) Scheduling system oriented to hybrid tasks
CN103176850A (en) Electric system network cluster task allocation method based on load balancing
CN103077086A (en) Method and system for controlling task distribution
Natarajan Parallel queue scheduling in dynamic cloud environment using backfilling algorithm
CN111242446A (en) Workshop scheduling method based on GA-TS hybrid algorithm

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant