CN117632505A - Heterogeneous calculation power intelligent scheduling system and method - Google Patents
Heterogeneous calculation power intelligent scheduling system and method Download PDFInfo
- Publication number
- CN117632505A CN117632505A CN202311666674.0A CN202311666674A CN117632505A CN 117632505 A CN117632505 A CN 117632505A CN 202311666674 A CN202311666674 A CN 202311666674A CN 117632505 A CN117632505 A CN 117632505A
- Authority
- CN
- China
- Prior art keywords
- cluster
- crd
- scheduling
- policy
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000004364 calculation method Methods 0.000 title claims abstract description 21
- 238000012216 screening Methods 0.000 claims description 15
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 230000006978 adaptation Effects 0.000 claims description 5
- 238000005516 engineering process Methods 0.000 claims description 4
- 230000000052 comparative effect Effects 0.000 claims 1
- 238000010606 normalization Methods 0.000 claims 1
- 230000008569 process Effects 0.000 description 11
- RYAUSSKQMZRMAI-YESZJQIVSA-N (S)-fenpropimorph Chemical compound C([C@@H](C)CC=1C=CC(=CC=1)C(C)(C)C)N1C[C@H](C)O[C@H](C)C1 RYAUSSKQMZRMAI-YESZJQIVSA-N 0.000 description 10
- 230000006870 function Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000005259 measurement Methods 0.000 description 2
- 238000013138 pruning Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Power Sources (AREA)
Abstract
The invention belongs to the technical field of power calculation task scheduling, and discloses a heterogeneous power calculation intelligent scheduling system and method. The heterogeneous computing power intelligent scheduling system comprises a computing power scheduling module, a cloud primary storage ETCD module and a third party expansion strategy module, wherein the Yun Yuansheng storage ETCD module stores heterogeneous computing power resources CRD, a scheduling strategy CRD, computing power tasks CRD and a cluster CRD; the third party expansion strategy module is used for registering the third party expansion strategy to the cloud primary storage ETCD module through an computing task interface provided by the heterogeneous computing resource CRD; yun Yuansheng the ETCD module is used for updating the scheduling policy CRD according to the third-party expansion policy to obtain an updated scheduling policy CRD; and the computing power scheduling module is used for determining target clusters in the cluster CRD according to the updated scheduling strategy CRD. The compatibility and the expansion function of heterogeneous computation scheduling can be realized.
Description
Technical Field
The invention relates to the technical field of power calculation task scheduling, in particular to a heterogeneous power calculation intelligent scheduling system and method.
Background
Along with the deepening of digital transformation in production operation of various industries, various industry terminals generate massive original data, so a great amount of calculation power is needed to process, and a technical architecture with high availability, high throughput and high expansion is often provided by adopting a calculation power cluster mode in order to meet the requirements of second-level response, low delay and service continuity of a service layer. In the annual construction process of the computing power clusters, a large amount of heterogeneous computing power cluster resources exist in the same data center or different data centers, and the computing power resources can be managed and scheduled on a unified platform, so that a computing power user can conveniently call the computing power resources according to the needs and the use habits, and the problem to be solved is solved. Aiming at intelligent scheduling of heterogeneous computing power, a set of intelligent scheduling method is customized according to specific scenes in the current mainstream technical scheme, each scheduling scheme cannot match the heterogeneous computing power scheduling requirements of other scenes, and the existing scheduling scheme cannot provide expandability, so that the existing scheduling scheme lacks compatibility and expandability.
Disclosure of Invention
The invention mainly aims to provide a heterogeneous computing power intelligent scheduling system and a heterogeneous computing power intelligent scheduling method, and aims to solve the technical problem that a scheduling scheme in the prior art lacks compatibility and expansibility.
In order to achieve the above purpose, the invention provides a heterogeneous computing power intelligent scheduling system, which comprises a computing power scheduling module, a cloud primary storage ETCD module and a third party expansion policy module, wherein the Yun Yuansheng storage ETCD module stores heterogeneous computing power resources CRD, scheduling policies CRD, computing power tasks CRD and cluster CRD, and a unified model and an interface library provided by the heterogeneous computing power resources CRD are used for realizing that the scheduling policies CRD are built in the heterogeneous computing power intelligent scheduling system; wherein,
the third party expansion policy module is configured to register a third party expansion policy to the Yun Yuansheng storage ETCD module through a computing task interface provided by the heterogeneous computing resource CRD;
the Yun Yuansheng storage ETCD module is configured to update the scheduling policy CRD according to the third party extension policy, to obtain an updated scheduling policy CRD;
the computing power scheduling module is used for determining a target cluster in the cluster CRD according to the updated scheduling policy CRD and binding computing power tasks to the target cluster.
Optionally, the heterogeneous computing power resource CRD is defined based on a cloud native technology, and includes an interface abstraction layer, where the interface abstraction layer uniformly processes unit adaptation of the computing power resource.
Optionally, the interface abstraction layer includes Deep Copy interface, canonicalize system normalized conversion interface, ADD addition operation interface, SUB subtraction operation interface, CMP comparison size operation interface, and Equal judgment operation interface.
Optionally, the heterogeneous computing power resource CRD uniformly provides a pruning and checking interface through the k8s apiserver.
Optionally, the update scheduling policy CRD includes a built-in policy and a third party expansion policy, where the built-in policy is loaded into the Cache during an operation phase of the computing power scheduling module, and the third party expansion policy is read from the Yun Yuansheng storage ETCD module by means of dynamic loading during the operation phase of the computing power scheduling module.
In addition, in order to achieve the above purpose, the invention also provides a heterogeneous computing power intelligent scheduling method, which comprises the following steps:
registering a third party expansion strategy to a cloud primary storage ETCD module through a computing task interface provided by a heterogeneous computing resource CRD;
updating Yun Yuansheng the scheduling policy CRD stored in the ETCD module according to the third party expansion policy to obtain an updated scheduling policy CRD;
and determining a target cluster in the cluster CRD according to the updated scheduling policy CRD, and binding the computing task to the target cluster.
Optionally, the update scheduling policy CRD includes a built-in policy and a third party expansion policy; wherein the determining a target cluster in the cluster CRD according to the updated scheduling policy CRD and binding a computing task to the target cluster includes:
triggering the power scheduling module to start a scheduling workflow after a power calculation task CRD in the ETCD module is stored according to the Yun Yuansheng to create the power calculation task, wherein the scheduling workflow comprises a pre-selection stage, a preferred stage and a binding stage;
traversing the built-in strategies at the pre-selection stage, and screening out first clusters meeting all built-in strategies from the cluster CRDs;
in the preferred stage, screening a second cluster from the first cluster according to the third party expansion strategy;
and in the binding stage, determining a target cluster in the second cluster, and binding the computing task to the target cluster.
Optionally, the screening the second cluster from the first cluster according to the third party expansion policy includes:
s301, a third party Operator/schedule HTTPS interface is called to execute a target strategy in the third party expansion strategy, and a cluster meeting the target strategy in the first cluster is scored to obtain a scheduling result;
s302, judging whether a new target strategy exists in the third-party expansion strategy;
s303, if yes, repeating the steps S301-S303 by taking the new target strategy as the target strategy in the step S301; and if not, screening a second cluster from the first clusters based on the scheduling result.
Optionally, the screening the second cluster from the first clusters based on the scheduling result includes:
setting a preset threshold value;
determining weight scoring of each cluster in the first cluster according to the scheduling result;
and taking the cluster with the weight scoring larger than the preset threshold value in the first cluster as a second cluster.
Optionally said determining a target cluster in said second cluster and binding said computing task to said target cluster comprises:
determining weight scoring of each cluster in the second cluster according to the scheduling result;
and taking the cluster with the highest weight value in the second cluster as the target cluster, and binding the computing task to the target cluster.
The invention provides a heterogeneous computing power intelligent scheduling system and a heterogeneous computing power intelligent scheduling method, wherein the heterogeneous computing power intelligent scheduling system comprises a computing power scheduling module, a cloud primary storage ETCD module and a third party expansion strategy module, wherein the Yun Yuansheng storage ETCD module stores heterogeneous computing power resources CRD, scheduling strategies CRD, computing power tasks CRD and cluster CRD, and a unified model and an interface library provided by the heterogeneous computing power resources CRD are used for realizing that the scheduling strategies CRD are built in the heterogeneous computing power intelligent scheduling system; the third party expansion policy module is configured to register a third party expansion policy to the Yun Yuansheng storage ETCD module through a computing task interface provided by the heterogeneous computing resource CRD; the Yun Yuansheng storage ETCD module is configured to update the scheduling policy CRD according to the third party extension policy, to obtain an updated scheduling policy CRD; the computing power scheduling module is used for determining a target cluster in the cluster CRD according to the updated scheduling policy CRD and binding computing power tasks to the target cluster. Through the system, the unified framework of heterogeneous computation power intelligent scheduling is used, and the compatibility and the expansion function of the heterogeneous computation power scheduling strategy are realized by combining the operators of the Kubernetes, the CRD and the built-in strategy.
Drawings
FIG. 1 is a block diagram of a first embodiment of a heterogeneous computing power intelligent scheduling system of the present invention;
FIG. 2 is a schematic diagram of heterogeneous computational power scheduling in a first embodiment of the heterogeneous computational power intelligent scheduling system of the present invention;
FIG. 3 is a schematic flow chart of a first embodiment of the heterogeneous power intelligent scheduling method of the present invention;
FIG. 4 is a schematic flow chart of determining a second cluster in the first embodiment of the heterogeneous computing power intelligent scheduling method of the present invention;
fig. 5 is a scheduling flow chart in the first embodiment of the heterogeneous power intelligent scheduling method of the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that, the ETCD represents a distributed key-value storage system, and CRD (Custom Resource Definition) is a mechanism in Kubernetes, which is used to extend the Kubernetes API and allow the user to define custom resources; operator Controller is a custom controller for managing and automating specific types of applications or resources in a Kubernetes cluster.
The invention provides an intelligent heterogeneous computing power dispatching system.
Referring to fig. 1, in the embodiment of the present invention, the heterogeneous computing power intelligent scheduling system includes a computing power scheduling module 10, a cloud primary storage ETCD module 20, and a third party expansion policy module 30, where the Yun Yuansheng storage ETCD module 20 stores a heterogeneous computing power resource CRD, a scheduling policy CRD, a computing power task CRD, and a cluster CRD, and a unified model and an interface library provided by the heterogeneous computing power resource CRD implement that the scheduling policy CRD is built in the heterogeneous computing power intelligent scheduling system; the third party expansion policy module 30 is configured to register a third party expansion policy to the Yun Yuansheng storage ETCD module 20 through a computing task interface provided by the heterogeneous computing resource CRD; the Yun Yuansheng storage ETCD module 20 is configured to update the scheduling policy CRD according to the third party extension policy, to obtain an updated scheduling policy CRD; the computing power scheduling module 10 is configured to determine a target cluster in the cluster CRD according to the updated scheduling policy CRD, and bind a computing power task to the target cluster.
It should be noted that, the scheduling policy includes a built-in policy and a third party expansion policy, and the scheduling policy is stored in the cloud primary storage ETCD module through the CRD data mode; updating the scheduling policy CRD refers to the updated scheduling policy CRD; the calculation task is stored in the cloud primary storage ETCD module through a CRD data mode; after the multi-cluster information is reported, the multi-cluster information is stored in a cloud primary storage ETCD module through a CRD data mode; the calculation force scheduling module refers to calculation force scheduling Operator Controller; the third party expansion policy module refers to a third party expansion policy Operator Controller, the third party expansion policy Operator Controller needs to implement/schedule HTTP interface, and the third party expansion policy module is mainly used for registering the third party expansion policy in the scheduling policy CRD of the cloud primary storage ETCD module;
it can be understood that the heterogeneous computing power intelligent scheduling system can register at any time to cause dynamic load in the running process; the target cluster can be the cluster with highest coincidence degree in the cluster CRD and can be used as the cluster which needs to be operated in the current computing task.
It should be noted that, updating the scheduling policy CRD according to the third party expansion policy is dynamically updated in real time, and may be implemented based on an asynchronous watch mechanism, specifically, when the third party user updates the scheduling policy CRD based on the third party expansion policy module, the method may be implemented by the following two steps: 1. based on a unified model and an interface library provided by heterogeneous computing power resource CRD, enabling a third party to realize extended computing power scheduling Operator Controller, and providing a standard HTTPS computing power task scheduling interface (/ schedule), wherein HTTPS can ensure communication and data security; 2. registering a third party extended policy to the cloud primary store ETCD triggers a policy Cache update of the computational schedule Operator Controller, registering not only the extended policy, but also the access address of the callback to let the computational schedule Operator Controller know the address of the/schedule interface call.
In an embodiment, the heterogeneous computing power resource CRD is defined based on a cloud native technology, and the heterogeneous computing power resource CRD includes an interface abstraction layer, and the interface abstraction layer uniformly processes unit adaptation of computing power resources.
It should be noted that, the interface abstraction layer uniformly processes the unit adaptation of the computational power resources such as binary, decimal, floating point number and the like; the interface abstraction layer contains the known adaptation of all computational power resource units.
In the embodiment, all heterogeneous computing power resources are compatible through support of different systems and units, and the problem of unified heterogeneous computing power model and measurement is solved.
In an embodiment, the interface abstraction layer includes Deep Copy interface, canonicalize system normalized conversion interface, ADD addition operation interface, SUB subtraction operation interface, CMP comparison size operation interface, and Equal equality judgment operation interface.
In this embodiment, the problems of heterogeneous computing power resource conversion, addition, subtraction, comparison and the like can be solved through the Deep Copy interface, the Canonicalize system standardized conversion interface, the ADD addition operation interface, the SUB subtraction operation interface, the CMP comparison size operation interface, and the Equal judgment operation interface in the interface abstraction layer.
In an embodiment, the heterogeneous computing resource CRD provides the pruning and modifying interface uniformly through the k8s apiserver.
In an embodiment, the update scheduling policy CRD includes a built-in policy and a third party expansion policy, where the built-in policy is loaded into the Cache during an operation phase of the computing power scheduling module, and the third party expansion policy is read from the Yun Yuansheng storage ETCD module by dynamically loading during the operation phase of the computing power scheduling module.
In a specific implementation, as shown in a heterogeneous computing power scheduling schematic diagram in fig. 2, it may be determined that the heterogeneous computing power intelligent scheduling system includes computing power scheduling Operator Controller, cloud primary storage ETCD and third party expansion policy Operator Controller, where Yun Yuansheng storage ETCD stores heterogeneous computing power resources CRD, scheduling policy CRD, computing power task CRD and cluster CRD, and an interface abstraction layer unified by the heterogeneous computing power resources CRD includes Deep Copy interface, canonicalize system normalized conversion interface, ADD operation interface, SUB subtraction operation interface, CMP comparison size operation interface and Equal judgment operation interface.
It should be noted that the reference model of the heterogeneous computing resource CRD is a YAML configuration file of a custom resource definition (custom resource definition) object in Kubernetes, which defines a custom resource named "kubrese resource. The API version of the custom resource is "apieextensions.k8s.io/v 1", belonging to the group "resource.kubuearenna.io". Its object type is "KubeResource", the complex form is "KubeResourceList", and has a short name "res"; the custom resource can be accessed in the whole cluster and provides a version named "v 1"; the definition of this version contains specification and state information about the "KubeResource" object.
The heterogeneous computing power intelligent scheduling system comprises a computing power scheduling module, a cloud primary storage ETCD module and a third party expansion strategy module, wherein the Yun Yuansheng storage ETCD module stores heterogeneous computing power resources CRD, scheduling strategies CRD, computing power tasks CRD and cluster CRD, and a unified model and an interface library provided by the heterogeneous computing power resources CRD are used for realizing that the scheduling strategies CRD are built in the heterogeneous computing power intelligent scheduling system; the third party expansion policy module is configured to register a third party expansion policy to the Yun Yuansheng storage ETCD module through a computing task interface provided by the heterogeneous computing resource CRD; the Yun Yuansheng storage ETCD module is configured to update the scheduling policy CRD according to the third party extension policy, to obtain an updated scheduling policy CRD; the computing power scheduling module is used for determining a target cluster in the cluster CRD according to the updated scheduling policy CRD and binding computing power tasks to the target cluster. Through the system, the unified framework of heterogeneous computation power intelligent scheduling is used, and the compatibility and the expansion function of the heterogeneous computation power scheduling strategy are realized by combining the operators of the Kubernetes, the CRD and the built-in strategy.
In addition, as shown in fig. 3, the embodiment of the invention also provides a heterogeneous computing power intelligent scheduling method.
In this embodiment, the heterogeneous computing power intelligent scheduling method includes the following steps:
step S10: and registering the third party expansion strategy to the cloud primary storage ETCD module through a computing power task interface provided by the heterogeneous computing power resource CRD.
It should be noted that, the execution body of the embodiment may be a computing service device with functions of data processing, network communication and program running, such as a mobile phone, a tablet computer, a personal computer, etc., or an electronic device or a heterogeneous computing power intelligent scheduling system capable of implementing the above functions. The present embodiment and the following embodiments will be described below by taking the heterogeneous computing power intelligent scheduling system as an example.
It should be noted that, the computing power task interface provided by the heterogeneous computing power resource CRD refers to a standard HTTPS computing power task scheduling interface (/ schedule);
step S20: and updating Yun Yuansheng the scheduling policy CRD stored in the ETCD module according to the third party expansion policy to obtain an updated scheduling policy CRD.
It can be understood that the heterogeneous computing power intelligent scheduling system can register at any time to cause dynamic load in the running process; the target cluster can be the cluster with highest coincidence degree in the cluster CRD and can be used as the cluster which needs to be operated in the current computing task.
It should be noted that, updating the scheduling policy CRD refers to the updated scheduling policy CRD; the updated scheduling policy CRD comprises a built-in policy and a third party expansion policy; the updating of the scheduling policy CRD according to the third party expansion policy is dynamically updated in real time, and may be implemented based on an asynchronous watch mechanism, specifically, when the third party user updates the scheduling policy CRD based on the third party expansion policy module, the updating may be implemented through the following two steps: 1. based on a unified model and an interface library provided by heterogeneous computing power resource CRD, enabling a third party to realize extended computing power scheduling Operator Controller, and providing a standard HTTPS computing power task scheduling interface (/ schedule), wherein HTTPS can ensure communication and data security; 2. registering a third party extended policy to the cloud primary store ETCD triggers a policy Cache update of the computational schedule Operator Controller, registering not only the extended policy, but also the access address of the callback to let the computational schedule Operator Controller know the address of the/schedule interface call.
Step S30: and determining a target cluster in the cluster CRD according to the updated scheduling policy CRD, binding the computing task to the target cluster, and binding the computing task to the target cluster.
After binding the computing task to the target cluster, the current computing task is scheduled, and the new computing task can be created based on the computing task CRD stored in the cloud primary storage ETCD module and then scheduled.
In an embodiment, the updated scheduling policy CRD includes a built-in policy and a third party expansion policy; wherein the determining a target cluster in the cluster CRD according to the updated scheduling policy CRD and binding a computing task to the target cluster includes:
triggering the power scheduling module to start a scheduling workflow after a power calculation task CRD in the ETCD module is stored according to the Yun Yuansheng to create the power calculation task, wherein the scheduling workflow comprises a pre-selection stage, a preferred stage and a binding stage;
traversing the built-in strategies at the pre-selection stage, and screening out first clusters meeting all built-in strategies from the cluster CRDs;
in the preferred stage, screening a second cluster from the first cluster according to the third party expansion strategy;
and in the binding stage, determining a target cluster in the second cluster, and binding the computing task to the target cluster.
It should be noted that, the Workflow of the scheduling Workflow includes three phases, which are a pre-selected phase, a priority phase and a binding phase in sequence; in the pre-selection stage, all built-in strategies need to be traversed and executed, and aiming at a single calculation task, the built-in strategies are sequentially executed according to the configuration sequence (the execution of the built-in strategies can be understood as judging whether the cluster meets the built-in strategies); the first cluster refers to a cluster meeting all built-in strategies in a cluster CRD; in the preferred stage, a third party expansion policy needs to be executed (it can be understood that whether the cluster meets the third party expansion policy is judged), and the second cluster refers to a cluster meeting the third expansion policy in the first cluster; in the binding stage, the cluster with the highest coincidence degree in the second cluster can be used as a target cluster, and the target cluster is the cluster which needs to be operated in the current calculation task.
It should be noted that, when executing the third party expansion policy, the third party Operator/schedule HTTPS interface needs to be called, and a scheduling result is received; the parameters of the schedule interface are the calculation Task (calculation Task) and the list of previously screened clusters (cluster CRD), and the list of clusters screened this time (i.e. the second cluster) is returned.
In the embodiment, the target cluster is determined from the cluster CRD through three stages of the scheduling workflow, so that the cluster with the highest coincidence degree can be rapidly determined, and the scheduling accuracy can be effectively improved.
In one embodiment, as shown in fig. 4, the screening the second cluster from the first cluster according to the third party expansion policy includes:
step S301: and calling a third party Operator/schedule HTTPS interface to execute a target strategy in the third party expansion strategy, and scoring a cluster meeting the target strategy in the first cluster to obtain a scheduling result.
Step S302: and judging whether a new target strategy exists in the third-party expansion strategy.
Step S303: if yes, repeating steps S301-S303 by taking the new target strategy as the target strategy in the step S301; and if not, screening a second cluster from the first clusters based on the scheduling result.
It is understood that the new target policy may be understood as a next third party expansion policy of the target policies among the third party expansion policies.
It should be noted that, the target policy may be determined according to a registration order of the third party expansion policy, for example, when the first round of clustering is performed, the target policy is a first registered policy in the third party expansion policy; when the scoring of each round of cluster needs to update a scheduling result, for example, when the scoring of the first round of cluster is performed, because the cluster A does not meet the target strategy in the first round of scoring, the cluster B meets the target strategy in the first round of scoring, and therefore, the scheduling result obtained through the first round of scoring is as follows: the cluster A is divided into 0 and the cluster B is divided into 1; when the second round of scoring is performed, because both the cluster A and the cluster B meet the target strategy in the second round of scoring, the scheduling result updated after the second round of scoring is: cluster a is given a score of 1 and cluster B is given a score of 2.
It can be appreciated that the scheduling result obtained when the new target policy does not exist in the third party expanded policy is the final scheduling result.
In an embodiment, the screening the second cluster from the first cluster based on the scheduling result includes:
setting a preset threshold value;
determining weight scoring of each cluster in the first cluster according to the scheduling result;
and taking the cluster with the weight scoring larger than the preset threshold value in the first cluster as a second cluster.
It should be noted that, the scheduling result refers to a final scheduling result, and can be used to determine a weight score of each cluster in the first cluster; the preset threshold may be preset, preferably, the preset threshold may be set to 0, and since the weight score of the cluster is greater than or equal to 0, the cluster may be used as the second cluster when the weight score of the cluster is not 0, and the cluster with the weight score of 0 may be regarded as the cluster that is not satisfied by all the target policies in the third party expansion policy.
In this embodiment, the threshold is set to determine the second cluster according to the scoring condition of each cluster in the first cluster, so that the determination efficiency of the second cluster can be effectively improved.
In an embodiment, the determining a target cluster in the second cluster and binding the computing task to the target cluster includes:
determining weight scoring of each cluster in the second cluster according to the scheduling result;
and taking the cluster with the highest weight value in the second cluster as the target cluster, and binding the computing task to the target cluster.
It should be noted that, the scheduling result refers to a final scheduling result, and can be used to determine a weight score of each cluster in the second cluster; specifically, for example, the weight score of each cluster in the second cluster is determined according to the scheduling result: and the weight of the cluster A is scored as 1, and the weight of the cluster B is scored as 2, so that the cluster B is the target cluster in the second cluster.
After binding the computing task to the target cluster, the current computing task is scheduled, and the new computing task can be created based on the computing task CRD stored in the cloud primary storage ETCD module and then scheduled.
In this embodiment, the cluster with the highest weight score is selected to be used as the cluster to be operated for the current computing task, so that the accuracy of computing power dispatching can be effectively improved.
In a specific implementation, as shown in fig. 5, a heterogeneous computing power resource CRD needs to be defined based on a cloud native technology, a unified interface abstraction layer is realized through the heterogeneous computing power resource CRD, a heterogeneous computing power unified model and a measurement framework can be completed, then a built-in scheduling strategy is realized based on the unified model and the interface library provided by the heterogeneous computing power resource CRD, then a computing power task triggering scheduling Workflow can be created, and scheduling of the computing power task can be completed based on three stages of a scheduling Workflow.
It should be understood that the foregoing is illustrative only and is not limiting, and that in specific applications, those skilled in the art may set the invention as desired, and the invention is not limited thereto.
It should be noted that the above-described working procedure is merely illustrative, and does not limit the scope of the present invention, and in practical application, a person skilled in the art may select part or all of them according to actual needs to achieve the purpose of the embodiment, which is not limited herein.
In addition, technical details which are not described in detail in the present embodiment can refer to the heterogeneous computing power intelligent scheduling method provided in any embodiment of the present invention, and are not described herein.
Furthermore, it should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. Read Only Memory)/RAM, magnetic disk, optical disk) and including several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (10)
1. The heterogeneous computing power intelligent scheduling system is characterized by comprising a computing power scheduling module, a cloud primary storage ETCD module and a third party expansion strategy module, wherein the Yun Yuansheng storage ETCD module stores heterogeneous computing power resources CRD, scheduling strategies CRD, computing power tasks CRD and cluster CRD, and a unified model and an interface library provided by the heterogeneous computing power resources CRD are used for realizing that the scheduling strategies CRD are built in the heterogeneous computing power intelligent scheduling system; wherein,
the third party expansion policy module is configured to register a third party expansion policy to the Yun Yuansheng storage ETCD module through a computing task interface provided by the heterogeneous computing resource CRD;
the Yun Yuansheng storage ETCD module is configured to update the scheduling policy CRD according to the third party extension policy, to obtain an updated scheduling policy CRD;
the computing power scheduling module is used for determining a target cluster in the cluster CRD according to the updated scheduling policy CRD and binding computing power tasks to the target cluster.
2. The system of claim 1, wherein the heterogeneous computing power resource CRD is defined based on a cloud-native technology, the heterogeneous computing power resource CRD including an interface abstraction layer that uniformly handles unit adaptation of computing power resources.
3. The system of claim 2, wherein the interface abstraction layer includes a Deep Copy interface, a Canonicalize system normalization conversion interface, an ADD addition operation interface, a SUB subtraction operation interface, a CMP comparative size operation interface, and an Equal equivalent judgment operation interface.
4. The system of claim 1, wherein the heterogeneous computing resource CRD provides a censored-censored interface through a k8s apiserver.
5. The system of claim 1, wherein the updated scheduling policy CRD includes a built-in policy that is loaded into a Cache during an operational phase of the power scheduling module and a third party extension policy that is read from the Yun Yuansheng store ETCD module by way of dynamic loading during the operational phase of the power scheduling module.
6. The heterogeneous computing power intelligent scheduling method is characterized in that the heterogeneous computing power intelligent scheduling method is applied to the heterogeneous computing power intelligent scheduling system according to any one of claims 1 to 5, and the heterogeneous computing power intelligent scheduling method comprises the following steps:
registering a third party expansion strategy to a cloud primary storage ETCD module through a computing task interface provided by a heterogeneous computing resource CRD;
updating Yun Yuansheng the scheduling policy CRD stored in the ETCD module according to the third party expansion policy to obtain an updated scheduling policy CRD;
and determining a target cluster in the cluster CRD according to the updated scheduling policy CRD, and binding the computing task to the target cluster.
7. The method of claim 6, wherein the updated scheduling policy CRD comprises a built-in policy and a third party extension policy; wherein the determining a target cluster in the cluster CRD according to the updated scheduling policy CRD and binding a computing task to the target cluster includes:
triggering the power scheduling module to start a scheduling workflow after a power calculation task CRD in the ETCD module is stored according to the Yun Yuansheng to create the power calculation task, wherein the scheduling workflow comprises a pre-selection stage, a preferred stage and a binding stage;
traversing the built-in strategies at the pre-selection stage, and screening out first clusters meeting all built-in strategies from the cluster CRDs;
in the preferred stage, screening a second cluster from the first cluster according to the third party expansion strategy;
and in the binding stage, determining a target cluster in the second cluster, and binding the computing task to the target cluster.
8. The method of claim 7, wherein the screening the second cluster from the first cluster according to the third party extension policy comprises:
s301, a third party Operator/schedule HTTPS interface is called to execute a target strategy in the third party expansion strategy, and a cluster meeting the target strategy in the first cluster is scored to obtain a scheduling result;
s302, judging whether a new target strategy exists in the third-party expansion strategy;
s303, if yes, repeating the steps S301-S303 by taking the new target strategy as the target strategy in the step S301; and if not, screening a second cluster from the first clusters based on the scheduling result.
9. The method of claim 8, wherein the screening the second cluster from the first cluster based on the scheduling result comprises:
setting a preset threshold value;
determining weight scoring of each cluster in the first cluster according to the scheduling result;
and taking the cluster with the weight scoring larger than the preset threshold value in the first cluster as a second cluster.
10. The method of claim 8, wherein the determining a target cluster in the second cluster and binding the computing task to the target cluster comprises:
determining weight scoring of each cluster in the second cluster according to the scheduling result;
and taking the cluster with the highest weight value in the second cluster as the target cluster, and binding the computing task to the target cluster.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311666674.0A CN117632505A (en) | 2023-12-06 | 2023-12-06 | Heterogeneous calculation power intelligent scheduling system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311666674.0A CN117632505A (en) | 2023-12-06 | 2023-12-06 | Heterogeneous calculation power intelligent scheduling system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117632505A true CN117632505A (en) | 2024-03-01 |
Family
ID=90037382
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311666674.0A Pending CN117632505A (en) | 2023-12-06 | 2023-12-06 | Heterogeneous calculation power intelligent scheduling system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117632505A (en) |
-
2023
- 2023-12-06 CN CN202311666674.0A patent/CN117632505A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8689231B2 (en) | System and method for ordering tasks with complex interrelationships | |
| US20190188012A1 (en) | Method, device, terminal and storage medium for processing application | |
| EP4080362A2 (en) | Method and apparatus for creating a container | |
| CN113986402A (en) | Function calling method and device, electronic equipment and storage medium | |
| US11610155B2 (en) | Data processing system and data processing method | |
| JP2021121921A (en) | Method and apparatus for management of artificial intelligence development platform, and medium | |
| US20110078710A1 (en) | Value container propagation in development tools, business process management, and business rules management solutions | |
| CN117827392A (en) | Distributed computing system, method, electronic equipment and medium for computing tasks | |
| CN117632505A (en) | Heterogeneous calculation power intelligent scheduling system and method | |
| CN112748997A (en) | Workflow scheduling method and system | |
| CN116302448A (en) | Task scheduling method and system | |
| CN115880508A (en) | Image data processing method, device, equipment and storage medium | |
| CN115658287A (en) | Method, apparatus, medium, and program product for scheduling execution units | |
| CN109165257A (en) | Data query method and related system, equipment and storage medium | |
| CN111985740B (en) | Task processing method and device | |
| US11366648B2 (en) | Compiling monoglot function compositions into a single entity | |
| CN107301089A (en) | A kind of APP deployment and call method and terminal | |
| CN113672557A (en) | Method, system, device, medium, and article of manufacture for migrating data to a distributed system | |
| CN113849285A (en) | Delay task processing method and device, electronic equipment and storage medium | |
| CN113986495A (en) | Task execution method, device, equipment and storage medium | |
| CN106934555B (en) | Workflow organization method and device, computer equipment and storage medium | |
| CN110928876A (en) | Credit data storage method and device | |
| US20240220472A1 (en) | Systems and methods for executing and hashing modeling flows | |
| CN108762871B (en) | Page component preloading method and device and display terminal | |
| CN118656222A (en) | Method, device and medium for constructing machine resource queuing in concurrent execution based on pipeline |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |