CN111045808A - Distributed network task scheduling method and device - Google Patents
Distributed network task scheduling method and device Download PDFInfo
- Publication number
- CN111045808A CN111045808A CN201911304562.4A CN201911304562A CN111045808A CN 111045808 A CN111045808 A CN 111045808A CN 201911304562 A CN201911304562 A CN 201911304562A CN 111045808 A CN111045808 A CN 111045808A
- Authority
- CN
- China
- Prior art keywords
- cluster
- preset
- central management
- management node
- clusters
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application discloses a distributed network task scheduling method and a device, wherein the method comprises the following steps: the central management node performs heartbeat information detection on each cluster according to a received preset task to obtain cluster heartbeat information, wherein the preset task comprises a preset priority, and the cluster comprises at least one common node; the central management node preselecting the clusters according to the preset tasks and the cluster heartbeat information to obtain preselection clusters, wherein the preselection takes the mean value and the variance which meet the primary preset indexes of each cluster as screening conditions; the central management node carries out selection on the preselected cluster through the cluster priority obtained according to the two-stage preset index of the preselected cluster to obtain a target cluster; and the central management node slices the preset tasks and distributes the preset tasks to the target clusters. The method and the device solve the technical problems that the prior distributed system does not comprehensively consider according to the specific conditions of the tasks and the states of the nodes, so that the tasks are piled up, the nodes are unbalanced and resources are wasted.
Description
Technical Field
The present application relates to the field of network resource scheduling technologies, and in particular, to a distributed network task scheduling method and apparatus.
Background
In recent years, in the fields of internet and electronic commerce, the management of network resources is more complex, the requirements on the concurrency capability and the response speed of services are higher and higher, the traditional single-machine server mode cannot bear huge access load for a long time, and the access of users is limited because the access resources and the self resources of the machine are always in a tight state.
In recent years, stand-alone storage gradually shifts to distributed storage, compared with a traditional stand-alone storage mode, a distributed system based on object storage brings advantages in space and access speed, storage problems caused by large data volume can be effectively solved through sufficient overall space, and processing pressure caused by high concurrency can be effectively limited by controlling access speed. The existing distributed task scheduling mode only simply considers the storage capacity, the computing power, the network influence and other factors of the nodes, and does not make a harmonious scheme between tasks and node allocation according to different conditions, so that the tasks are bundled and the nodes are unbalanced, and the node downtime and the resource waste are easily caused.
Disclosure of Invention
The application provides a distributed network task scheduling method and device, which are used for solving the technical problems of task bunching, node unbalance and resource waste caused by the fact that the conventional distributed system does not comprehensively consider according to the specific conditions of tasks and the states of nodes.
In view of this, a first aspect of the present application provides a distributed network task scheduling method, including:
the method comprises the steps that a central management node carries out heartbeat information detection on each cluster according to a received preset task to obtain cluster heartbeat information, wherein the preset task comprises a preset priority, and the cluster comprises at least one common node;
the central management node preselecting the clusters according to the preset tasks and the cluster heartbeat information to obtain preselected clusters, wherein the preselection takes the mean value and the variance of a primary preset index meeting each cluster as screening conditions, and the primary preset index comprises cluster idle computing resources and cluster idle storage resources;
the central management node carries out selection on the preselected cluster according to the cluster priority obtained according to the two-stage preset index of the preselected cluster to obtain a target cluster;
and the central management node slices the preset tasks and distributes the preset tasks to the target clusters.
Preferably, the preselecting the cluster by the central management node according to the preset task and the cluster heartbeat information to obtain a preselection cluster, including:
the central management node discards clusters with the number of the off-line or off-line common nodes exceeding the total number of the nodes based on the cluster heartbeat information;
the central management node selects the first-level preset index according to the preset priority;
the central management node calculates the mean value and the variance of the primary preset indexes of each cluster;
the central management node takes the cluster with the mean value difference within a first threshold range and the variance within a second threshold range as the pre-selected cluster.
Preferably, the secondary preset indexes include: cluster computing capacity, cluster minimum idle storage resources, and cluster common node number.
Preferably, the central management node selects the preselected cluster according to the cluster priority obtained according to the secondary preset index of the preselected cluster to obtain a target cluster, and the method includes:
the central management node sets the preset weight of the secondary preset index according to the preset priority;
the central management node calculates the cluster priority according to the preset weight and the secondary preset index;
and the central management node performs descending sorting on the preselected clusters according to the cluster priorities to obtain the target clusters.
Preferably, the slicing and distributing the preset tasks to the target clusters by the central management node includes:
and the central management node distributes the sliced preset tasks to the target cluster based on the cluster priority.
Preferably, the secondary preset index further includes: amount of idle bandwidth resources and latency.
A second aspect of the present application provides a distributed network task scheduling apparatus, including:
the system comprises a detection module, a central management node and a control module, wherein the detection module is used for detecting heartbeat information of each cluster by the central management node according to a received preset task to obtain the heartbeat information of the cluster, the preset task comprises a preset priority, and the cluster comprises at least one common node;
the preselection module is used for preselecting the clusters by the central management node according to the preset tasks and the cluster heartbeat information to obtain preselection clusters, the preselection takes the mean value and the variance of first-stage preset indexes meeting all the clusters as screening conditions, and the first-stage preset indexes comprise cluster idle computing resources and cluster idle storage resources;
the fine selection module is used for the central management node to perform fine selection on the preselected cluster according to the cluster priority obtained according to the secondary preset index of the preselected cluster to obtain a target cluster;
and the scheduling module is used for slicing the preset tasks by the central management node and distributing the sliced preset tasks to the target clusters.
Preferably, the preselection module comprises:
a preselection submodule, configured to discard, by the central management node, clusters in which the number of the common nodes that are offline or offline exceeds the total number of nodes based on the cluster heartbeat information;
the central management node selects the first-level preset index according to the preset priority;
the central management node calculates the mean value and the variance of the primary preset indexes of each cluster;
the central management node takes the cluster with the mean value difference within a first threshold range and the variance within a second threshold range as the pre-selected cluster.
Preferably, the culling module comprises:
the concentration submodule is used for setting the preset weight of the secondary preset index by the central management node according to the preset priority;
the central management node calculates the cluster priority according to the preset weight and the secondary preset index;
and the central management node performs descending sorting on the preselected clusters according to the cluster priorities to obtain the target clusters.
Preferably, the scheduling module includes:
and the scheduling submodule is used for distributing the sliced preset tasks to the target cluster by the central management node based on the cluster priority.
According to the technical scheme, the embodiment of the application has the following advantages:
the application provides a distributed network task scheduling method, which comprises the following steps: the central management node performs heartbeat information detection on each cluster according to a received preset task to obtain cluster heartbeat information, wherein the preset task comprises a preset priority, and the cluster comprises at least one common node; the central management node preselecting the clusters according to the preset tasks and the cluster heartbeat information to obtain preselection clusters, wherein the preselection takes the mean value and the variance of the primary preset indexes of each cluster as screening conditions, and the primary preset indexes comprise cluster idle computing resources and cluster idle storage resources; the central management node carries out selection on the preselected cluster through the cluster priority obtained according to the two-stage preset index of the preselected cluster to obtain a target cluster; and the central management node slices the preset tasks and distributes the preset tasks to the target clusters.
In the distributed network task scheduling method provided by the application, the received tasks are subjected to priority division through a central management node, so that the actual requirements of the tasks are met, then clusters are selected according to the task requirements, the selection of a first-level preset index can be confirmed according to the preset priorities of the tasks, the average value and the variance of the first-level preset indexes of all the clusters are obtained, the basic level and the stability degree of the corresponding indexes of the clusters can be measured, the screened preselected clusters can meet the task requirements, the preselected clusters are selected again, the indexes which are not considered in the preselection process are comprehensively considered, the task bearing capacity of the obtained target cluster plays the greatest role, and the tasks of the nodes share the target object; the purpose of selection is to screen the existing resource states of clusters, select clusters with stronger computing capacity, larger idle storage or more nodes, so that the clusters have stronger task execution capacity and higher speed, and can avoid the phenomenon that tasks are piled up at a certain node to cause node imbalance; moreover, through task fragment redistribution, the resource burden of each node is shared, and simultaneously, the state of executing tasks at more nodes can be ensured, so that the resource waste is reduced, and the task execution speed can be accelerated. Therefore, the distributed network task scheduling method provided by the application can solve the technical problems of task bunching, node imbalance and resource waste caused by the fact that the conventional distributed system does not comprehensively consider according to the specific conditions of tasks and the states of nodes.
Drawings
Fig. 1 is a schematic flowchart of a distributed network task scheduling method according to a first embodiment of the present disclosure;
fig. 2 is a schematic flowchart of a second embodiment of a distributed network task scheduling method provided in the present application;
fig. 3 is a schematic structural diagram of an embodiment of a distributed network task scheduling apparatus provided in the present application.
Detailed Description
In order to make the technical solutions of the present application better understood, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
For convenience of understanding, please refer to fig. 1, a first embodiment of a distributed network task scheduling method provided in the present application includes:
It should be noted that the central management node is obtained by selecting nodes layer by layer, and is used for task scheduling, lower node management, and the like; the common node is a node with stronger execution capacity; the detection of the heartbeat information is mainly to perform online investigation on common nodes in the existing cluster, if most of the nodes in the cluster are in an offline or offline state, the health state of the cluster is judged to be poor, the cluster is not listed for screening any more, the priority of the preset task refers to that the task has higher requirement on a certain index, such as the size of a storage space or the requirement of computing resources, and when the task has higher requirement on a certain index, the task is set as the priority of the task, namely the index is considered in priority when the cluster or the node is selected.
And 102, the central management node preselects the clusters according to the preset tasks and the cluster heartbeat information to obtain preselection clusters, and the preselection takes the mean value and the variance which meet the primary preset indexes of each cluster as screening conditions.
The first-level preset indexes comprise cluster idle computing resources and cluster idle storage resources.
It should be noted that the cluster heartbeat information filters out the clusters with a large number of nodes offline, the preset tasks include preset priorities, and the clusters are selected by adopting a first-level preset index selected according to the preset priorities, so that the clusters and the nodes which meet the task requirements better can be obtained, and the task allocation is more targeted. The average value of the first-level preset indexes of the clusters is obtained to solve the average resource amount of the clusters, and the clusters with the average value difference exceeding a threshold value are removed; the variance of the primary preset index is calculated to eliminate unstable clusters.
And 103, the central management node carries out selection on the preselected cluster according to the cluster priority obtained according to the second-level preset index of the preselected cluster to obtain a target cluster.
It should be noted that the secondary preset index is an index that is not considered in the preselection operation, for example, the primary preset index in the preselection is the minimum idle storage resource, and the secondary preset index is the cluster computing capacity, the number of common nodes of the cluster, and the like; if the pre-selected primary preset index is the cluster computing capacity, the secondary preset index is the minimum idle storage resource, the number of common nodes of the cluster and the like. If a plurality of secondary preset indexes exist, the proportion of each index needs to be calculated through a weight value, the priority can be converted according to the score of each index, the total priority of each cluster can be obtained by accumulating the priorities of different indexes, and a target cluster can be selected according to the total priority.
And step 104, slicing the preset tasks by the central management node, and distributing the preset tasks to the target clusters.
It should be noted that the purpose of slicing is to disperse the resource occupation amount, and also to ensure that each node can be utilized to the maximum extent, thereby reducing the waste of resources; the assignment of the slicing tasks to the clusters is also performed according to the priorities of the clusters.
In the distributed network task scheduling method provided by this embodiment, a central management node performs priority division on a received task, so as to meet actual requirements of the task, then a cluster is selected according to the task requirements, the selection of a first-level preset index can be confirmed according to the preset priority of the task, the average value and variance of the first-level preset indexes of all the clusters can be obtained to measure the basic level and stability of corresponding indexes of the cluster, so that the screened preselected clusters can meet the task requirements, and the re-selection of the preselected clusters is to comprehensively consider the indexes which are not considered in the preselection, so that the task carrying capacity of the obtained target cluster plays the greatest role, and the task of the node shares a target; the purpose of selection is to screen the existing resource states of clusters, select clusters with stronger computing capacity, larger idle storage or more nodes, so that the clusters have stronger task execution capacity and higher speed, and can avoid the phenomenon that tasks are piled up at a certain node to cause node imbalance; moreover, through task fragment redistribution, the resource burden of each node is shared, and simultaneously, the state of executing tasks at more nodes can be ensured, so that the resource waste is reduced, and the task execution speed can be accelerated. Therefore, the distributed network task scheduling method provided by the embodiment can solve the technical problems of task bunching, node imbalance and resource waste caused by the fact that the conventional distributed system does not comprehensively consider according to the specific situations of tasks and the states of nodes.
For convenience of understanding, please refer to fig. 2, an embodiment two of a distributed network task scheduling method is provided in the embodiment of the present application, including:
It should be noted that the clusters can be classified, the lowest cluster is composed of common nodes, secondary management nodes obtained by election exist in the clusters, the upper layer of the clusters can be a larger management cluster, and the management is performed layer by layer until reaching a central management node, so that a cluster tree network is formed; the cluster can also be directly managed by the central management node, and the specific implementation of the scheme is not influenced.
It should be noted that the central management node specifically includes a receiver, a monitor, and a scheduler; the receiver can analyze the received tasks, perform priority division according to the service types and the requirements of the tasks, and issue the tasks with high priority to the monitor, the monitor is used for detecting the heartbeat information of the cluster nodes and can also record the resource new energy conditions of the cluster nodes, and finally the non-scheduler is sent by combining the tasks, the scheduler can only allocate one task at a time, and the scheduler is responsible for scheduling the tasks according to the screening conditions of the clusters.
It should be noted that, the heartbeat information of the cluster reflects that the node in the cluster is online, and the node that is offline or disconnected cannot execute the task, and when the number of the offline nodes of a cluster is half, it indicates that the health condition of the cluster is poor, and the cluster is not considered.
And 203, the central management node selects a first-level preset index according to the preset priority.
And step 204, the central management node calculates the mean value and the variance of the primary preset indexes of each cluster.
It should be noted that the first-stage preset index includes a cluster idle calculation resource and a cluster idle storage resource, and one of the two indexes can be selected as the preset index to perform subsequent calculation; the first-level preset indexes are selected for meeting the service requirements of tasks, and the average value of the first-level preset indexes of the cluster is calculated for obtaining the average resource amount and the variance of the cluster, so that a stable cluster is selected. The total amount and the occupation amount of each node resource in the cluster can be obtained through the monitor, and the idle average resource can be obtained by using an average value formula, wherein the average value formula is as follows:
wherein M is1，M2，……，MnRespectively the resource amount of each node, N1，N2，……，NnAnd the resource occupation of each node.
Is an average value, and n is the total number of nodes in the cluster.
The average value filters out clusters with larger differences, the obtained clusters use variance to judge the stability of the clusters, a preselected cluster with high and low performance can be obtained,
for variance, the formula is as follows:
and step 206, the central management node sets the preset weight of the secondary preset index according to the preset priority.
And step 207, the central management node calculates the cluster priority according to the preset weight and the second-level preset index.
It should be noted that the secondary preset indexes include cluster computing capacity, minimum idle storage resources of the cluster, the number of common nodes of the cluster, the amount of idle bandwidth resources, and delay time; the selection of the preset weight of the secondary preset index is operated according to the preset priority, because although the secondary preset index is used for screening the clusters, the importance of each index is weighted according to a certain priority by referring to task requirements. Each index can obtain a specific proportion score through the weight, the score is converted into a priority, each cluster can obtain the priority of different indexes, and the priorities are accumulated to obtain the total priority of each cluster, namely the cluster priority.
And step 208, the central management node performs descending sorting on the preselected clusters according to the cluster priorities to obtain target clusters.
It should be noted that the number of the secondary preset indexes is large, so that the final priority of the cluster needs to be determined by the weight, and the priority may be selected according to the need by sorting from high to low.
And step 209, the central management node distributes the sliced preset tasks to the target clusters based on the cluster priorities.
It should be noted that, in order to ensure the execution of the slicing task, the distribution is directly started from the cluster with the higher priority until the distribution of the slicing task is completed; when the slicing tasks are distributed and the clusters all receive the slicing tasks, the slicing tasks which are not distributed need to be queued, and the nodes are waited to execute the tasks and then are distributed; and when a plurality of nodes are idle at the same time, distributing the slicing tasks to the nodes with high priority according to the priority. The specific task allocation process comprises the following steps: selecting the cluster with the highest priority as a first task distribution cluster, wherein the number of common nodes of the clusterIs n1The minimum size of the idle memory resource is V1The total amount of tasks received by the first cluster is Q1＝n1×V1The remaining task amount is Δ Q ═ Q-Q1When delta Q is larger than 0, selecting the next-level cluster, continuing to distribute tasks, wherein the calculation method of the distributed task quantity and the residual task quantity is the same as that of the first distribution operation, and the task quantity of the second cluster is obtained and is Q2＝n2×V2The remaining task amount is Δ Q ═ Q-Q1-Q2(ii) a By analogy with the method, the amount of the guided tasks is 0, and the scheduling of the tasks is finished.
For ease of understanding, please refer to fig. 3, an embodiment of a distributed network task scheduling apparatus is further provided in the present application, including:
the detection module 301 is configured to perform heartbeat information detection on each cluster by the central management node according to a received preset task to obtain cluster heartbeat information, where the preset task includes a preset priority, and the cluster includes at least one common node;
a preselection module 302, configured to preselection the cluster by the central management node according to a preset task and cluster heartbeat information to obtain a preselection cluster, where the preselection takes a mean and a variance of first-level preset indexes of each cluster as a screening condition, where the first-level preset indexes include cluster idle computing resources and cluster idle storage resources;
the selection module 303 is configured to select, by the central management node, the preselected cluster according to the cluster priority obtained according to the second-level preset index of the preselected cluster, so as to obtain a target cluster;
and the scheduling module 304 is used for the central management node to slice the preset tasks and distribute the preset tasks to the target clusters.
Further, the preselection module 302 includes: a preselection submodule 3021, configured to discard, by the central management node, clusters in which the number of offline or offline common nodes exceeds the total number of nodes based on cluster heartbeat information; the central management node selects a first-level preset index according to the preset priority; the central management node calculates the mean value and the variance of the first-stage preset indexes of each cluster; and the central management node takes the cluster with the mean value difference within a first threshold range and the variance within a second threshold range as a pre-selected cluster.
Further, the culling module 303 includes: a fine sub-module 3031, configured to set, by the central management node, a preset weight of the second-level preset index according to the preset priority; the central management node calculates the cluster priority according to the preset weight and the second-level preset index; and the central management node performs descending sorting on the preselected clusters according to the cluster priorities to obtain target clusters.
Further, the scheduling module 304 includes: the scheduling submodule 3041 is configured to, based on the cluster priority, allocate the sliced preset task to the target cluster by the central management node.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for executing all or part of the steps of the method described in the embodiments of the present application through a computer device (which may be a personal computer, a server, or a network device). And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (10)
1. A distributed network task scheduling method is characterized by comprising the following steps:
the method comprises the steps that a central management node carries out heartbeat information detection on each cluster according to a received preset task to obtain cluster heartbeat information, wherein the preset task comprises a preset priority, and the cluster comprises at least one common node;
the central management node preselecting the clusters according to the preset tasks and the cluster heartbeat information to obtain preselected clusters, wherein the preselection takes the mean value and the variance of a primary preset index meeting each cluster as screening conditions, and the primary preset index comprises cluster idle computing resources and cluster idle storage resources;
the central management node carries out selection on the preselected cluster according to the cluster priority obtained according to the two-stage preset index of the preselected cluster to obtain a target cluster;
and the central management node slices the preset tasks and distributes the preset tasks to the target clusters.
2. The distributed network task scheduling method according to claim 1, wherein the preselecting of the cluster by the central management node according to the preset task and the cluster heartbeat information to obtain a preselecting cluster comprises:
the central management node discards clusters with the number of the off-line or off-line common nodes exceeding the total number of the nodes based on the cluster heartbeat information;
the central management node selects the first-level preset index according to the preset priority;
the central management node calculates the mean value and the variance of the primary preset indexes of each cluster;
the central management node takes the cluster with the mean value difference within a first threshold range and the variance within a second threshold range as the pre-selected cluster.
3. The distributed network task scheduling method of claim 1, wherein the secondary preset index comprises: cluster computing capacity, cluster minimum idle storage resources, and cluster common node number.
4. The distributed network task scheduling method of claim 1, wherein the central management node performs fine selection on the preselected cluster according to a cluster priority obtained according to a secondary preset index of the preselected cluster to obtain a target cluster, and the method comprises:
the central management node sets the preset weight of the secondary preset index according to the preset priority;
the central management node calculates the cluster priority according to the preset weight and the secondary preset index;
and the central management node performs descending sorting on the preselected clusters according to the cluster priorities to obtain the target clusters.
5. The distributed network task scheduling method of claim 4, wherein the slicing and distributing the preset tasks to the target clusters by the central management node comprises:
and the central management node distributes the sliced preset tasks to the target cluster based on the cluster priority.
6. The distributed network task scheduling method of claim 1, wherein the secondary preset index further comprises: amount of idle bandwidth resources and latency.
7. A distributed network task scheduler, comprising:
the system comprises a detection module, a central management node and a control module, wherein the detection module is used for detecting heartbeat information of each cluster by the central management node according to a received preset task to obtain the heartbeat information of the cluster, the preset task comprises a preset priority, and the cluster comprises at least one common node;
the preselection module is used for preselecting the clusters by the central management node according to the preset tasks and the cluster heartbeat information to obtain preselection clusters, the preselection takes the mean value and the variance of first-stage preset indexes meeting all the clusters as screening conditions, and the first-stage preset indexes comprise cluster idle computing resources and cluster idle storage resources;
the fine selection module is used for the central management node to perform fine selection on the preselected cluster according to the cluster priority obtained according to the secondary preset index of the preselected cluster to obtain a target cluster;
and the scheduling module is used for slicing the preset tasks by the central management node and distributing the sliced preset tasks to the target clusters.
8. The distributed network task scheduler of claim 7, wherein the preselection module comprises:
a preselection submodule, configured to discard, by the central management node, clusters in which the number of the common nodes that are offline or offline exceeds the total number of nodes based on the cluster heartbeat information;
the central management node selects the first-level preset index according to the preset priority;
the central management node calculates the mean value and the variance of the primary preset indexes of each cluster;
the central management node takes the cluster with the mean value difference within a first threshold range and the variance within a second threshold range as the pre-selected cluster.
9. The distributed network task scheduler of claim 7, wherein the culling module comprises:
the concentration submodule is used for setting the preset weight of the secondary preset index by the central management node according to the preset priority;
the central management node calculates the cluster priority according to the preset weight and the secondary preset index;
and the central management node performs descending sorting on the preselected clusters according to the cluster priorities to obtain the target clusters.
10. The distributed network task scheduler of claim 7, wherein the scheduling module comprises:
and the scheduling submodule is used for distributing the sliced preset tasks to the target cluster by the central management node based on the cluster priority.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911304562.4A CN111045808B (en) | 2019-12-17 | 2019-12-17 | Distributed network task scheduling method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911304562.4A CN111045808B (en) | 2019-12-17 | 2019-12-17 | Distributed network task scheduling method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111045808A true CN111045808A (en) | 2020-04-21 |
| CN111045808B CN111045808B (en) | 2023-04-18 |
Family
ID=70237540
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911304562.4A Active CN111045808B (en) | 2019-12-17 | 2019-12-17 | Distributed network task scheduling method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111045808B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111930489A (en) * | 2020-09-15 | 2020-11-13 | 南京领行科技股份有限公司 | Task scheduling method, device, equipment and storage medium |
| CN112272203A (en) * | 2020-09-18 | 2021-01-26 | 苏州浪潮智能科技有限公司 | Cluster service node selection method, system, terminal and storage medium |
| CN113840014A (en) * | 2021-11-29 | 2021-12-24 | 中国电子科技集团公司第二十八研究所 | Distributed task decomposition method adaptive to high-strength weak connection environment |
| CN115174695A (en) * | 2022-07-18 | 2022-10-11 | 中软航科数据科技(珠海横琴)有限公司 | Scheduling system and method for distributed network resources |
| CN115562879A (en) * | 2022-12-06 | 2023-01-03 | 北京邮电大学 | Computing power sensing method, computing power sensing device, electronic device and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103207814A (en) * | 2012-12-27 | 2013-07-17 | 北京仿真中心 | Decentralized cross cluster resource management and task scheduling system and scheduling method |
| CN105302638A (en) * | 2015-11-04 | 2016-02-03 | 国家计算机网络与信息安全管理中心 | MPP (Massively Parallel Processing) cluster task scheduling method based on system load |
| CN106909451A (en) * | 2017-02-28 | 2017-06-30 | 郑州云海信息技术有限公司 | A kind of distributed task dispatching system and method |
| CN107066319A (en) * | 2017-01-17 | 2017-08-18 | 北京国电通网络技术有限公司 | A kind of multidimensional towards heterogeneous resource dispatches system |
| CN107291545A (en) * | 2017-08-07 | 2017-10-24 | 星环信息科技（上海）有限公司 | The method for scheduling task and equipment of multi-user in computing cluster |
-
2019
- 2019-12-17 CN CN201911304562.4A patent/CN111045808B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103207814A (en) * | 2012-12-27 | 2013-07-17 | 北京仿真中心 | Decentralized cross cluster resource management and task scheduling system and scheduling method |
| CN105302638A (en) * | 2015-11-04 | 2016-02-03 | 国家计算机网络与信息安全管理中心 | MPP (Massively Parallel Processing) cluster task scheduling method based on system load |
| CN107066319A (en) * | 2017-01-17 | 2017-08-18 | 北京国电通网络技术有限公司 | A kind of multidimensional towards heterogeneous resource dispatches system |
| CN106909451A (en) * | 2017-02-28 | 2017-06-30 | 郑州云海信息技术有限公司 | A kind of distributed task dispatching system and method |
| CN107291545A (en) * | 2017-08-07 | 2017-10-24 | 星环信息科技（上海）有限公司 | The method for scheduling task and equipment of multi-user in computing cluster |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111930489A (en) * | 2020-09-15 | 2020-11-13 | 南京领行科技股份有限公司 | Task scheduling method, device, equipment and storage medium |
| CN112272203A (en) * | 2020-09-18 | 2021-01-26 | 苏州浪潮智能科技有限公司 | Cluster service node selection method, system, terminal and storage medium |
| CN112272203B (en) * | 2020-09-18 | 2022-06-14 | 苏州浪潮智能科技有限公司 | Cluster service node selection method, system, terminal and storage medium |
| CN113840014A (en) * | 2021-11-29 | 2021-12-24 | 中国电子科技集团公司第二十八研究所 | Distributed task decomposition method adaptive to high-strength weak connection environment |
| CN115174695A (en) * | 2022-07-18 | 2022-10-11 | 中软航科数据科技(珠海横琴)有限公司 | Scheduling system and method for distributed network resources |
| CN115174695B (en) * | 2022-07-18 | 2024-01-26 | 中软航科数据科技(珠海横琴)有限公司 | Scheduling system and method for distributed network resources |
| CN115562879A (en) * | 2022-12-06 | 2023-01-03 | 北京邮电大学 | Computing power sensing method, computing power sensing device, electronic device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111045808B (en) | 2023-04-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111045808B (en) | Distributed network task scheduling method and device | |
| US8510747B2 (en) | Method and device for implementing load balance of data center resources | |
| CN108683720B (en) | Container cluster service configuration method and device | |
| KR101540631B1 (en) | System, method and recording medium recording the program thereof for dynamic expansion of the virtual cluster | |
| CN110795203B (en) | Resource scheduling method, device, system and computing equipment | |
| CN103391299B (en) | Load-balancing method and SiteServer LBS | |
| CN104102543B (en) | The method and apparatus of adjustment of load in a kind of cloud computing environment | |
| CN106657191B (en) | Load balancing method and related device and system | |
| CN103502944A (en) | Method and device for adjusting memories of virtual machines | |
| CN102045396A (en) | Load balancing method of server document | |
| CN105871983A (en) | Method and device for selecting servers | |
| US20190280945A1 (en) | Method and apparatus for determining primary scheduler from cloud computing system | |
| CN109189578B (en) | Storage server allocation method, device, management server and storage system | |
| CN106201711B (en) | A kind of task processing method and server | |
| CN110099061B (en) | Cloud platform video streaming service selection method and device | |
| CN104899072B (en) | Fine granularity resource scheduling system based on virtual platform and method | |
| CN106603631A (en) | Distributed message system and expansion method thereof | |
| CN116546028A (en) | Service request processing method and device, storage medium and electronic equipment | |
| CN115604269A (en) | Load balancing method and device of server, electronic equipment and storage medium | |
| CN107704320B (en) | A kind of method for allocating tasks and system of distributed system | |
| CN109670691A (en) | Method, equipment and the customer service system distributed for customer service queue management and customer service | |
| CN106210120B (en) | A kind of recommended method and its device of server | |
| CN114584565B (en) | Application protection method and system, electronic equipment and storage medium | |
| CN112003900B (en) | Method and system for realizing high service availability under high-load scene in distributed system | |
| CN113079062A (en) | Resource adjusting method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |