CN117056058B - Task scheduling method, system, equipment and storage medium based on state awareness - Google Patents
Task scheduling method, system, equipment and storage medium based on state awareness Download PDFInfo
- Publication number
- CN117056058B CN117056058B CN202311310819.3A CN202311310819A CN117056058B CN 117056058 B CN117056058 B CN 117056058B CN 202311310819 A CN202311310819 A CN 202311310819A CN 117056058 B CN117056058 B CN 117056058B
- Authority
- CN
- China
- Prior art keywords
- scheduling
- algorithm
- target
- data source
- target weather
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 272
- 238000012545 processing Methods 0.000 claims description 15
- 230000000903 blocking effect Effects 0.000 claims description 7
- 238000012216 screening Methods 0.000 claims description 5
- 230000005856 abnormality Effects 0.000 claims description 4
- 239000002699 waste material Substances 0.000 abstract description 6
- 230000008447 perception Effects 0.000 abstract description 2
- 238000011144 upstream manufacturing Methods 0.000 description 14
- 238000004891 communication Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000012805 post-processing Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000006855 networking Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000032683 aging Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
A task scheduling method, system, equipment and storage medium based on state perception relate to the technical field of meteorological task scheduling, and comprise the steps of counting and updating data source arrival rates corresponding to various data source messages; and when detecting that the arrival rate of the target data source corresponding to the target data source message is greater than or equal to an arrival rate threshold corresponding to the target data source message, generating a target weather algorithm scheduling instruction based on algorithm execution information and resource state information corresponding to the target data source message, so that a scheduling client starts to operate a target weather algorithm based on the target weather algorithm scheduling instruction. Through the method and the device, the task scheduling efficiency can be effectively improved, and resource waste can be avoided.
Description
Technical Field
The application relates to the technical field of meteorological task scheduling, in particular to a task scheduling method, system, equipment and storage medium based on state awareness.
Background
With the development of meteorological service, a user has higher requirements on the operation efficiency of the system and the product generation timeliness of the system, wherein the operation efficiency and the product generation timeliness depend on the calculation efficiency of an algorithm on one hand and on the starting timeliness of the algorithm on the other hand. Under the condition of unchanged calculation efficiency, if the starting time of the algorithm can be advanced, the product generation aging can also be advanced, and a more intelligent scheduling method is needed to enable the algorithm to be started at the first time when the condition is met. Data processing algorithms in meteorological service are often operated through data driving, namely, the algorithm is operated to require that data reach certain integrity, otherwise, the quality of the generated product is greatly affected.
In the current weather service, the algorithm is usually operated in a timing scheduling mode, or in order to improve the timeliness of calculation, the algorithm is operated in a resident memory mode, but the two modes have larger defects in scheduling timeliness, resource utilization rationality and the like. The timing scheduling generally discovers data in a directory scanning mode, namely the data discovery mode in the timing scheduling is passive, so that the data cannot be completely fitted to the time of reporting, on one hand, the algorithm cannot be started at the first time of the data to the report, so that the starting timeliness of the algorithm is reduced, and on the other hand, the directory scanning also causes additional resource waste, particularly when the number of files under the directory is large; in addition, although the algorithm resident memory can calculate the first time of the data to the report by utilizing the method of monitoring the message, the algorithm resident memory can occupy the resources for a long time, thereby causing the resource waste.
Therefore, a task scheduling method with high timeliness is needed in the weather business mainly driven by data, so that efficient scheduling operation of the weather business is guaranteed, and the purpose of 'data arrival algorithm operation' is achieved.
Disclosure of Invention
The application provides a task scheduling method, a system, equipment and a storage medium based on state awareness, which can solve the technical problems of low task scheduling efficiency and resource waste in the prior art.
In a first aspect, an embodiment of the present application provides a task scheduling method based on state awareness, where the task scheduling method based on state awareness is applied to a scheduling server, and the method includes the following steps:
counting and updating the data source arrival rate corresponding to each data source message;
and when detecting that the arrival rate of the target data source corresponding to the target data source message is greater than or equal to an arrival rate threshold corresponding to the target data source message, generating a target weather algorithm scheduling instruction based on algorithm execution information and resource state information corresponding to the target data source message, so that a scheduling client starts to operate a target weather algorithm based on the target weather algorithm scheduling instruction.
With reference to the first aspect, in one implementation manner, the counting and updating the data source arrival rate corresponding to each data source message includes:
and counting and updating the data source to report rate corresponding to the data source message based on the four-level coding and the data observation time in the data source message for each data source message, and storing the updated data source to report rate into a Redis cache.
With reference to the first aspect, in one implementation manner, the algorithm execution information includes a target weather algorithm ID, a target weather algorithm parameter, a target weather algorithm execution command, a target weather algorithm execution user, and an executable computing node corresponding to the target weather algorithm, the resource status information includes first resource status information corresponding to each executable computing node and second resource status information corresponding to the target weather algorithm, the first resource status information includes a CPU usage amount, a memory usage amount, and a running algorithm number, and the second resource status information includes a CPU and a memory amount required to run the target weather algorithm.
With reference to the first aspect, in an implementation manner, the generating the target weather algorithm scheduling instruction based on the algorithm execution information and the resource status information corresponding to the target data source message includes:
screening a target computing node from executable computing nodes based on the first resource state information and the second resource state information;
and generating a target weather algorithm scheduling instruction based on the target computing node and the algorithm execution information.
With reference to the first aspect, in an implementation manner, the scheduling client starts to operate a target weather algorithm based on the target weather algorithm scheduling instruction, including:
The scheduling client analyzes the target weather algorithm scheduling instruction to obtain a target weather algorithm ID, target weather algorithm parameters, a target weather algorithm execution command, a target weather algorithm execution user and second resource state information;
the scheduling client determines whether to schedule a target weather algorithm or not based on the left resource information and the second resource state information;
if yes, the scheduling client performs user scheduling and starts to operate the target weather algorithm based on the target weather algorithm ID, the target weather algorithm parameters, the target weather algorithm execution command and the target weather algorithm;
if not, the scheduling client transmits the target weather algorithm scheduling failure information to the scheduling server through gRPC protocol.
With reference to the first aspect, in an embodiment, the method further includes:
and when the scheduling server receives the scheduling failure information of the target weather algorithm through the gRPC protocol, performing scheduling abnormality warning prompt, and storing the scheduling instruction of the target weather algorithm into a blocking queue in a Redis cache.
In a second aspect, an embodiment of the present application provides a task scheduling system based on state awareness, where the task scheduling system based on state awareness includes a scheduling server, where the scheduling server is configured to:
Counting and updating the data source arrival rate corresponding to each data source message;
and when detecting that the arrival rate of the target data source corresponding to the target data source message is greater than or equal to an arrival rate threshold corresponding to the target data source message, generating a target weather algorithm scheduling instruction based on algorithm execution information and resource state information corresponding to the target data source message, so that a scheduling client starts to operate a target weather algorithm based on the target weather algorithm scheduling instruction.
With reference to the second aspect, in one embodiment, the scheduling server is specifically configured to:
and counting and updating the data source to report rate corresponding to the data source message based on the four-level coding and the data observation time in the data source message for each data source message, and storing the updated data source to report rate into a Redis cache.
In a third aspect, an embodiment of the present application provides a state-aware-based task scheduling device, where the state-aware-based task scheduling device includes a processor, a memory, and a state-aware-based task scheduler stored on the memory and executable by the processor, where the state-aware-based task scheduler, when executed by the processor, implements the steps of the state-aware-based task scheduling method as described above.
In a fourth aspect, embodiments of the present application provide a computer readable storage medium having stored thereon a state-based task scheduler, wherein the state-based task scheduler, when executed by a processor, implements the steps of the state-based task scheduling method as described above.
The beneficial effects that technical scheme that this application embodiment provided include at least:
the arrival state of the meteorological data is known and mastered in real time by counting and updating the arrival rate of the data source corresponding to each data source message of the upstream business, and when the demand of the downstream business is met, namely when the arrival rate of the data source reaches a set arrival rate threshold value, an algorithm scheduling instruction is rapidly sent out, so that the meteorological algorithm can be started to run at the first time, timely scheduling of the meteorological business algorithm and serial connection of the upstream business and the downstream business are effectively realized, directory scanning is not needed, resident memory of the algorithm is not needed, and resource waste is effectively avoided while the scheduling efficiency and starting timeliness of the meteorological algorithm are improved.
Drawings
FIG. 1 is a schematic flow chart of an embodiment of a task scheduling method based on state awareness;
FIG. 2 is a schematic diagram of functional modules of an embodiment of a task scheduling system based on state awareness of the present application;
fig. 3 is a schematic hardware structure diagram of a task scheduling device based on state awareness according to an embodiment of the present application.
Detailed Description
In order to make the present application solution better understood by those skilled in the art, the following description will clearly and completely describe the technical solution in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
The terms "comprising" and "having" and any variations thereof in the description and claims of the present application and in the foregoing drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus. The terms "first," "second," and "third," etc. are used for distinguishing between different objects and not necessarily for describing a sequential or chronological order, and are not limited to the fact that "first," "second," and "third" are not identical.
In the description of embodiments of the present application, "exemplary," "such as," or "for example," etc., are used to indicate an example, instance, or illustration. Any embodiment or design described herein as "exemplary," "such as" or "for example" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "such as" or "for example," etc., is intended to present related concepts in a concrete fashion.
In the description of the embodiments of the present application, unless otherwise indicated, "/" means or, for example, a/B may represent a or B; the text "and/or" is merely an association relation describing the associated object, and indicates that three relations may exist, for example, a and/or B may indicate: the three cases where a exists alone, a and B exist together, and B exists alone, and in addition, in the description of the embodiments of the present application, "plural" means two or more than two.
In some of the processes described in the embodiments of the present application, a plurality of operations or steps occurring in a particular order are included, but it should be understood that these operations or steps may be performed out of the order in which they occur in the embodiments of the present application or in parallel, the sequence numbers of the operations merely serve to distinguish between the various operations, and the sequence numbers themselves do not represent any order of execution. In addition, the processes may include more or fewer operations, and the operations or steps may be performed in sequence or in parallel, and the operations or steps may be combined.
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
In a first aspect, an embodiment of the present application provides a task scheduling method based on state awareness.
In an embodiment, referring to fig. 1, fig. 1 is a flow chart of an embodiment of a task scheduling method based on state awareness in the present application. As shown in fig. 1, the task scheduling method based on state awareness is applied to a scheduling server, and the task scheduling method based on state awareness includes the following steps:
step S10: counting and updating the data source arrival rate corresponding to each data source message;
by way of example, it will be appreciated that the upstream and downstream relationships of a weather service rely primarily on the flow of weather data, i.e., the output product of the upstream service serves as the input data for the downstream service. However, the current downstream service cannot grasp the product generation state of the upstream service in real time, so when the upstream service has completed product generation, the problem that the downstream service cannot be started in time due to the fact that the downstream service cannot grasp the product generation state of the upstream service in real time often occurs, namely, the phenomenon of weather service disconnection occurs.
In this embodiment, the scheduling server side uses the characteristics of the weather big data cloud platform message bus, and on the basis of globally grasping the scheduling policy of each algorithm, acquires and analyzes the reporting states of various data source messages corresponding to the upstream service in real time, so as to monitor various states, and connects links on the service chain in series through state information. When the data source trigger task is started by the task scheduling trigger engine in the scheduling server, the data source trigger task is matched with the data source information corresponding to the upstream service from Kafka (Kafka is a high throughput distributed publishing and subscribing information system which can process all action stream data of a consumer in a website) and count the arrival condition of the data source information, and statistics and updating are carried out on the arrival rate of the data source corresponding to each data source information, so that the product generation state of the upstream service can be mastered in real time, and related algorithms of the downstream service can be started in time when the requirement of the downstream service is met, and real-time linkage of the upstream service and the downstream service is realized.
It should be noted that, the task scheduling trigger engine includes an external trigger engine and an internal trigger engine, where the external trigger engine is mainly a custom trigger engine, and the internal trigger engine mainly includes a state sensing trigger engine, a timing trigger engine, a workflow trigger engine, a manual trigger engine, and the like.
Step S20: and when detecting that the arrival rate of the target data source corresponding to the target data source message is greater than or equal to an arrival rate threshold corresponding to the target data source message, generating a target weather algorithm scheduling instruction based on algorithm execution information and resource state information corresponding to the target data source message, so that a scheduling client starts to operate a target weather algorithm based on the target weather algorithm scheduling instruction.
For example, in this embodiment, for the data source message corresponding to each weather algorithm, the corresponding algorithm scheduling configuration information is configured for and stored in the configuration information base, where it is to be noted that the algorithm scheduling configuration information includes, but is not limited to, four-level codes corresponding to each data source message, a preset arrival rate threshold value, an arrival rate observation time, and the like. Wherein, the four-level coding is composed of four levels, and the structure is: x.yyy.nnnn.azzz, wherein "X" identifies a first class classification of meteorological data, represented by a capital letter; "YYYY" identifies the two-level classification of meteorological data, consisting of 4 digits or capital letters, the first two digits being digits, ranging from 00 to 99, the last two digits being digits or capital letters; "NNNN" identifies three-level classification of meteorological data, consisting of 4 digits, ranging from 0001 to 9999; the AZZZ identifies the business link where the meteorological data is located, "A" is a business link identification, the code is represented by 1-bit capital letters (such as R represents a transmission link, P represents a processing link, S represents a storage link and M represents a service link), the ZZZ is a code of four-level data under the business link, and the code consists of 3-bit numbers or capital letters, for example, a certain four-level code is J.0001.0002.S001; the arrival rate threshold may be an arrival data count threshold, an arrival data percentage threshold, or other forms of threshold expression, and the specific value setting may be determined according to actual requirements, for example, the arrival rate threshold is set to 95% by taking the arrival data percentage threshold as an example.
The time-to-report rate observation time refers to that the observation time corresponding to the data in the data source message must be the same as a preset certain time, for example, for the data source message a corresponding to the weather algorithm X, assuming that the time-to-report rate observation time corresponding to the data source message a is 2023, 9, 21, and 08 (i.e. 2023092108), the data A1 observed in the data source message a must also be 2023, 9, 21, and 08, and the data A1 may be considered as the data required for executing the weather algorithm X.
It should be appreciated that the configuration information base is used to store not only the algorithm scheduling configuration information, but also support information required for the operation of the system, such as algorithm execution information, resource status information of the computing nodes, and algorithm execution status information.
In this embodiment, after statistics and updating of the data source to reporting rate corresponding to each data source message, for each data source message, it is further required to determine a size relationship between the updated data source to reporting rate corresponding to each data source message and the corresponding reporting rate threshold, and determine whether to generate an algorithm scheduling instruction based on the determination result, so that a corresponding algorithm can be started by first time scheduling, and further high-timeliness task scheduling is achieved.
Specifically, taking a target data source message corresponding to a certain target weather algorithm as an example, judging whether the updated arrival rate of the target data source corresponding to the target data source message is greater than or equal to an arrival rate threshold corresponding to the target data source message, if so, generating a target weather algorithm scheduling instruction based on algorithm execution information and resource state information corresponding to the target data source message, and sending the target weather algorithm scheduling instruction to a corresponding scheduling client through a message middleware so that the scheduling client starts to operate the target weather algorithm based on the target weather algorithm scheduling instruction; if not, continuing to monitor and update the target data source message to the reporting state. It should be noted that, the message middleware is used for communication between the scheduling server and the scheduling client, and specifically includes updating the algorithm execution state, the computing node resource state, and the like. Therefore, the scheduling efficiency of the weather algorithm is remarkably improved, the serial connection of the upstream business and the downstream business is realized according to the natural time sequence and the business logic, and the problem of the dislocation of the weather business is solved to a certain extent.
It should be noted that, after the execution of the target weather algorithm is finished, the scheduling client sends the execution state (i.e. execution success or execution failure), the algorithm execution finishing time, the computing node resource state, and the like of the target weather algorithm to the scheduling server through the rabitmq (which is implemented based on the open source of the advanced message queue protocol and is used for storing and forwarding the message in the distributed system) cluster, so that the scheduling server performs the algorithm scheduling post-processing including updating the algorithm execution state, updating the computing node resource state, and the like, and the whole scheduling flow is finished.
In summary, the embodiment calculates and updates the arrival rate of the data source corresponding to each data source message of the upstream service to acquire and grasp the arrival state of the meteorological data in real time, and when the requirement of the downstream service is met, that is, when the arrival rate of the data source reaches the set arrival rate threshold, an algorithm scheduling instruction is rapidly sent out, so that the meteorological algorithm can be started to run at the first time, timely scheduling of the meteorological service algorithm and serial connection of the upstream service and the downstream service are effectively realized, directory scanning is not needed, and the algorithm is not required to be resident in a memory, so that the scheduling efficiency and the starting timeliness of the meteorological algorithm are improved, and meanwhile, the resource waste is effectively avoided.
Further, in an embodiment, the counting and updating the data source to report rate corresponding to each data source message includes:
and counting and updating the data source to report rate corresponding to the data source message based on the four-level coding and the data observation time in the data source message for each data source message, and storing the updated data source to report rate into a Redis cache.
Illustratively, in this embodiment, for each data source message, the scheduling server will match the data source message from Kafka and count the arrival of the data source message. Specifically, taking a target data source message as an example, the target data source message contains a four-level code and data observation time, whether the four-level code in the target data source message is consistent with the four-level code corresponding to the target data source message stored in the configuration information base or not is judged, and if not, the target data source message is directly discarded; if so, continuing to judge whether the data observation time in the target data source message is consistent with the reporting rate observation time corresponding to the target data source message stored in the configuration information base, if not, directly discarding the target data source message, and if so, successfully matching, updating the reporting rate of the data source corresponding to the target data source message, and storing the real-time updated reporting rate of the data source into the Redis cache. It should be noted that, in this embodiment, low time delay of scheduling can be effectively guaranteed through the dis cache, that is, time delay of algorithm scheduling is greatly improved.
Further, in an embodiment, the algorithm execution information includes a target weather algorithm ID, a target weather algorithm parameter, a target weather algorithm execution command, a target weather algorithm execution user, and an executable computing node corresponding to the target weather algorithm, the resource status information includes first resource status information corresponding to each executable computing node and second resource status information corresponding to the target weather algorithm, the first resource status information includes a CPU usage amount, a memory usage amount, and a running algorithm amount, and the second resource status information includes a CPU and a memory amount required to run the target weather algorithm.
Exemplary, in this embodiment, algorithm execution information corresponding to each data source message and resource status information of each computing node are stored in a configuration information base; therefore, when the arrival rate of the target data source reaches the corresponding arrival rate threshold value, the corresponding algorithm execution information and resource state information are acquired from the configuration information base, so that the corresponding algorithm scheduling instruction can be generated by the algorithm execution information and the resource state information.
The algorithm execution information includes, but is not limited to, a target weather algorithm ID, a target weather algorithm parameter, a target weather algorithm execution command, a target weather algorithm execution user, and an executable computing node corresponding to the target weather algorithm, the resource state information includes, but is not limited to, first resource state information corresponding to each executable computing node, including, but not limited to, a CPU usage amount, a memory usage amount, and a running algorithm amount, and second resource state information corresponding to the target weather algorithm, including, but not limited to, a CPU and a memory amount required to run the target weather algorithm. It should be noted that, the above examples of the algorithm execution information and the resource status information are merely presented in embodiments, and related information in the algorithm execution information and the resource status information may be added or deleted according to actual requirements.
Further, in an embodiment, the generating the target weather algorithm scheduling instruction based on the algorithm execution information and the resource status information corresponding to the target data source message includes:
screening a target computing node from executable computing nodes based on the first resource state information and the second resource state information;
and generating a target weather algorithm scheduling instruction based on the target computing node and the algorithm execution information.
In this embodiment, after the scheduling server determines in real time that the arrival rate of the target data source corresponding to the target data source message has reached the configured arrival rate threshold, the algorithm execution information including the ID of the target weather algorithm, the target weather algorithm parameter, the target weather algorithm execution command, the target weather algorithm execution user, the target weather algorithm executable computing node, and the like is read from the configuration information base, and simultaneously, first resource state information corresponding to the target weather algorithm executable computing node including the CPU usage amount, the memory usage amount, the number of running algorithms, and the like, and second resource state information including the CPU and the memory amount required for the current target weather algorithm running are obtained; and then, judging the resource load based on the first resource state information and the second resource state information to determine a target computing node (namely a scheduling client) for executing the target weather algorithm, further ensuring optimal resource allocation when the algorithm is scheduled in a cluster environment, and generating a target weather algorithm scheduling instruction through the target computing node and the acquired algorithm execution information.
Further, in an embodiment, the scheduling client starts to operate the target weather algorithm based on the target weather algorithm scheduling instruction, including:
the scheduling client analyzes the target weather algorithm scheduling instruction to obtain a target weather algorithm ID, target weather algorithm parameters, a target weather algorithm execution command, a target weather algorithm execution user and second resource state information;
the scheduling client determines whether to schedule a target weather algorithm or not based on the left resource information and the second resource state information;
if yes, the scheduling client performs user scheduling and starts to operate the target weather algorithm based on the target weather algorithm ID, the target weather algorithm parameters, the target weather algorithm execution command and the target weather algorithm;
if not, the scheduling client transmits the target weather algorithm scheduling failure information to the scheduling server through gRPC protocol.
Exemplary, in this embodiment, the task scheduling client is mainly configured to execute an algorithm scheduling instruction issued by the scheduling server, collect an algorithm execution state, and calculate a memory and a CPU state of the node. Therefore, after the scheduling server generates the scheduling information of the target weather algorithm, the scheduling server sends the information dropping instruction of the target weather algorithm to the scheduling client corresponding to the target computing node based on the message middleware through gRPC (gRPC is a high-performance, open-source and language-independent general RPC framework which is designed based on the HTTP2 Protocol and uses PB (Protocol Buffer) in sequence; the scheduling client analyzes the target weather algorithm scheduling instruction to obtain information such as corresponding target weather algorithm ID, target weather algorithm parameters, target weather algorithm execution command, CPU and memory required by target weather algorithm execution user and operation, and schedules and starts execution of the target weather algorithm based on the obtained information. It should be noted that, in this embodiment, not only the low delay of the scheduling but also the high stability of the scheduling can be ensured by the gRPC communication.
Specifically, after the scheduling client analyzes the target weather algorithm scheduling instruction to obtain the CPU and the memory amount required by the operation of the target weather algorithm, further judging whether the current residual resources can meet the CPU and the memory amount required by the operation of the target weather algorithm, if so, scheduling and starting to execute the target weather algorithm, and transmitting the scheduling success information of the target weather algorithm to the scheduling server through gRPC protocol; if the scheduling failure information does not meet the scheduling failure information, the scheduling failure information of the target weather algorithm is sent to a scheduling server through the gRPC protocol, and a scheduling result is returned in time. When the target weather algorithm is scheduled, if the scheduling fails due to a system failure or the like, the scheduling failure information of the target weather algorithm is also sent to the scheduling server through the gRPC protocol.
Further, in an embodiment, the method further comprises:
and when the scheduling server receives the scheduling failure information of the target weather algorithm through the gRPC protocol, performing scheduling abnormality warning prompt, and storing the scheduling instruction of the target weather algorithm into a blocking queue in a Redis cache.
In this embodiment, after receiving the target weather algorithm scheduling failure information through the gRPC protocol, the scheduling server not only carries out scheduling exception alarm prompting, but also sends the target weather algorithm scheduling instruction to the blocking queue of the Redis cache, that is, stores the scheduling information when the resources of the scheduling client are insufficient through the blocking queue of the Redis, and reschedules the scheduling information based on the target weather algorithm scheduling instruction stored in the blocking queue, and records a log at the same time, if the retries are still unsuccessful, the target weather algorithm scheduling is directly discarded, and no scheduling is carried out any more. It can be understood that, in this embodiment, in order to ensure the reliability and fault tolerance requirements of the scheduling server, the Redis cluster is used to persistence the algorithm information to be executed in the system memory and the scheduling intermediate result, so as to avoid the loss of the scheduling information when the scheduling server is restarted or abnormally terminated.
The following describes, in combination with a specific embodiment, that the scheduling efficiency of the algorithm can be effectively improved in this embodiment: taking a radar data processing flow as an example, the flow comprises two links of basic data processing and networking jigsaw, the task scheduling method based on state perception provided by the embodiment is used for replacing a processing mode of scanning a catalogue in a minute-by-minute timing manner to realize scheduling start of a basic data processing algorithm and generate a networking flat product, and the time efficiency of basic data processing can be averagely improved by about 20 seconds through statistics, and the time efficiency of generating the networking jigsaw product is obviously improved from 8 minutes to 3 minutes, so that a foundation is laid for the time efficiency improvement of downstream business such as short-term forecasting, live-action and the like. When the embodiment is applied to the data source triggering scheduling of supporting a processing assembly line in a weather big data cloud platform of the China weather department, the real-time scheduling operation of 150 algorithms is realized, scheduling timeliness is remarkably improved, and the scheduling success rate can reach 100%.
In summary, the embodiment is based on the message bus of the weather big data cloud platform, acquires the arrival status of each data source message in real time, and performs efficient processing and analysis, and when the arrival rate of the data source reaches the set arrival rate threshold, the embodiment can rapidly send out an algorithm scheduling instruction to start an execution algorithm; and the low time delay and high stability of the scheduling are ensured by applying technologies such as Redis cache, gRPC communication, containers and the like, for example, the scheduling time delay is controlled at millisecond level when 300 algorithms are scheduled simultaneously.
In a second aspect, the embodiment of the application also provides a task scheduling system based on state awareness.
In an embodiment, referring to fig. 2, fig. 2 is a schematic functional block diagram of an embodiment of a task scheduling system based on state awareness in the present application. As shown in fig. 2, the task scheduling system based on state awareness includes a scheduling server for:
counting and updating the data source arrival rate corresponding to each data source message;
and when detecting that the arrival rate of the target data source corresponding to the target data source message is greater than or equal to an arrival rate threshold corresponding to the target data source message, generating a target weather algorithm scheduling instruction based on algorithm execution information and resource state information corresponding to the target data source message, so that a scheduling client starts to operate a target weather algorithm based on the target weather algorithm scheduling instruction.
It can be understood that the scheduling server is mainly used for realizing statistical analysis of data source to report conditions, calculation resource load judgment, algorithm scheduling instruction generation and issuing and task scheduling post-processing, and comprises a task scheduling trigger engine, a task scheduling plug-in, scheduling instruction distribution, task scheduling load balancing, task execution queue management, a task execution engine, a task scheduling post-processing component, redis cache and other modules. The task scheduling trigger engine is used for generating scheduling information according to scheduling rules preconfigured by an algorithm; the task scheduling plug-in is used for generating an algorithm scheduling instruction according to the defined algorithm information and the scheduling information; the scheduling instruction distribution is used for sending the generated algorithm scheduling instruction to the scheduling client; the task scheduling load balancing is used for screening out the computing node with the minimum load according to the CPU and the memory of the server to execute the algorithm; the task execution queue management is used for managing ready and waiting scheduling messages based on a Redis cache, wherein the Redis cache guarantees scheduling timeliness and stability with high-efficiency reading and writing performance; the task execution engine is used for being responsible for communication with a scheduling client of the computing node in a gRPC mode and the like; the task scheduling post-processing component is used for processing task scheduling results and executing results.
Referring to fig. 2, the system in this embodiment further includes a configuration information base, message middleware, and a scheduling client. The configuration information base is mainly used for storing supporting information required by system operation, such as information of an algorithm operation frame, required resource size, operation nodes, scheduling strategies and the like; the message middleware is used for supporting communication between the scheduling server and the scheduling client through gRPC protocol, rabbitMQ cluster and the like, and specifically comprises issuing of scheduling instructions, callback of algorithm execution state, returning of calculation node state and the like; the dispatching client is mainly used for executing an algorithm dispatching instruction issued by the dispatching server, collecting an algorithm execution state, calculating the memory and CPU state of the node and the like.
Further, in an embodiment, the scheduling server is specifically configured to:
and counting and updating the data source to report rate corresponding to the data source message based on the four-level coding and the data observation time in the data source message for each data source message, and storing the updated data source to report rate into a Redis cache.
Further, in an embodiment, the algorithm execution information includes a target weather algorithm ID, a target weather algorithm parameter, a target weather algorithm execution command, a target weather algorithm execution user, and an executable computing node corresponding to the target weather algorithm, the resource status information includes first resource status information corresponding to each executable computing node and second resource status information corresponding to the target weather algorithm, the first resource status information includes a CPU usage amount, a memory usage amount, and a running algorithm amount, and the second resource status information includes a CPU and a memory amount required to run the target weather algorithm.
Further, in an embodiment, the scheduling server is specifically further configured to:
screening a target computing node from executable computing nodes based on the first resource state information and the second resource state information;
and generating a target weather algorithm scheduling instruction based on the target computing node and the algorithm execution information.
Further, in an embodiment, the task scheduling system further includes a scheduling client, which is specifically configured to:
analyzing the target weather algorithm scheduling instruction to obtain a target weather algorithm ID, target weather algorithm parameters, a target weather algorithm execution command, a target weather algorithm execution user and second resource state information;
determining whether to schedule a target weather algorithm based on the remaining resource information and the second resource status information;
if yes, executing user scheduling and starting to operate the target weather algorithm based on the target weather algorithm ID, the target weather algorithm parameters, the target weather algorithm execution command and the target weather algorithm;
if not, the target weather algorithm scheduling failure information is sent to the scheduling server through the gRPC protocol.
Further, in an embodiment, the scheduling server is further configured to:
And after receiving the scheduling failure information of the target weather algorithm through the gRPC protocol, performing scheduling abnormality warning prompt, and storing the scheduling instruction of the target weather algorithm into a blocking queue in a Redis cache.
The function implementation of each part in the task scheduling system based on state awareness corresponds to each step in the task scheduling method embodiment based on state awareness, and the function and implementation process of each part are not described in detail herein.
In a third aspect, an embodiment of the present application provides a task scheduling device based on state awareness, which may be a device with a data processing function, such as a personal computer (personal computer, PC), a notebook computer, a server, or the like.
Referring to fig. 3, fig. 3 is a schematic hardware structure diagram of a task scheduling device based on state awareness according to an embodiment of the present application. In the embodiment of the application, the task scheduling device based on state awareness can include a processor, a memory, a communication interface and a communication bus.
The communication bus may be of any type for implementing the processor, memory, and communication interface interconnections.
The communication interfaces include input/output (I/O) interfaces, physical interfaces, logical interfaces, and the like for implementing device interconnections within the state-aware-based task scheduling device, and interfaces for implementing state-aware-based task scheduling device interconnections with other devices (e.g., other computing devices or user devices). The physical interface may be an ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device may be a Display, a Keyboard (Keyboard), or the like.
The memory may be various types of storage media such as random access memory (randomaccess memory, RAM), read-only memory (ROM), nonvolatile RAM (non-volatileRAM, NVRAM), flash memory, optical memory, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (electrically erasable PROM, EEPROM), and the like.
The processor may be a general-purpose processor, and the general-purpose processor may call a task scheduler based on state awareness stored in the memory and execute the task scheduling method based on state awareness provided in the embodiments of the present application. For example, the general purpose processor may be a central processing unit (central processing unit, CPU). The method executed when the task scheduler based on state awareness is invoked may refer to various embodiments of the task scheduling method based on state awareness of the present application, which are not described herein.
Those skilled in the art will appreciate that the hardware configuration shown in fig. 3 is not limiting of the application and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.
In a fourth aspect, embodiments of the present application also provide a computer-readable storage medium.
The application readable storage medium stores a task scheduling program based on state awareness, wherein the task scheduling program based on state awareness realizes the steps of the task scheduling method based on state awareness when being executed by a processor.
The method implemented when the task scheduler based on state awareness is executed may refer to various embodiments of the task scheduling method based on state awareness of the present application, which are not described herein.
It should be noted that, the foregoing embodiment numbers are merely for describing the embodiments, and do not represent the advantages and disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) as described above, comprising several instructions for causing a terminal device to perform the method described in the various embodiments of the present application.
The foregoing description is only of the preferred embodiments of the present application, and is not intended to limit the scope of the claims, and all equivalent structures or equivalent processes using the descriptions and drawings of the present application, or direct or indirect application in other related technical fields are included in the scope of the claims of the present application.
Claims (9)
1. The task scheduling method based on state awareness is characterized by being applied to a scheduling server and comprising the following steps of:
counting and updating the data source arrival rate corresponding to each data source message;
when detecting that the arrival rate of a target data source corresponding to a target data source message is greater than or equal to an arrival rate threshold corresponding to the target data source message, generating a target weather algorithm scheduling instruction based on algorithm execution information and resource state information corresponding to the target data source message, wherein the algorithm execution information comprises a target weather algorithm ID, target weather algorithm parameters, a target weather algorithm execution command, a target weather algorithm execution user and an executable computing node corresponding to the target weather algorithm, the resource state information comprises second resource state information corresponding to the target weather algorithm, and the second resource state information comprises a CPU (central processing unit) and a memory amount required for running the target weather algorithm;
The scheduling client starts to operate a target weather algorithm based on the target weather algorithm scheduling instruction, and the method comprises the following steps:
the scheduling client analyzes the target weather algorithm scheduling instruction to obtain a target weather algorithm ID, target weather algorithm parameters, a target weather algorithm execution command, a target weather algorithm execution user and second resource state information;
the scheduling client determines whether to schedule a target weather algorithm or not based on the left resource information and the second resource state information;
if yes, the scheduling client performs user scheduling and starts to operate the target weather algorithm based on the target weather algorithm ID, the target weather algorithm parameters, the target weather algorithm execution command and the target weather algorithm;
if not, the scheduling client transmits the target weather algorithm scheduling failure information to the scheduling server through gRPC protocol.
2. The state-aware-based task scheduling method according to claim 1, wherein the counting and updating the data source-to-report rate corresponding to each data source message comprises:
and counting and updating the data source to report rate corresponding to the data source message based on the four-level coding and the data observation time in the data source message for each data source message, and storing the updated data source to report rate into a Redis cache.
3. The state-aware-based task scheduling method of claim 1, wherein: the resource state information further includes first resource state information corresponding to each executable computing node, the first resource state information including a CPU usage amount, a memory usage amount, and a number of algorithms being run.
4. The state-aware-based task scheduling method of claim 3, wherein the generating the target weather algorithm scheduling instructions based on the algorithm execution information and the resource state information corresponding to the target data source message comprises:
screening a target computing node from executable computing nodes based on the first resource state information and the second resource state information;
and generating a target weather algorithm scheduling instruction based on the target computing node and the algorithm execution information.
5. The state-aware-based task scheduling method of claim 1, further comprising:
and when the scheduling server receives the scheduling failure information of the target weather algorithm through the gRPC protocol, performing scheduling abnormality warning prompt, and storing the scheduling instruction of the target weather algorithm into a blocking queue in a Redis cache.
6. The task scheduling system based on state awareness is characterized by comprising a scheduling server, wherein the scheduling server is used for:
counting and updating the data source arrival rate corresponding to each data source message;
when detecting that the arrival rate of a target data source corresponding to a target data source message is greater than or equal to an arrival rate threshold corresponding to the target data source message, generating a target weather algorithm scheduling instruction based on algorithm execution information and resource state information corresponding to the target data source message, wherein the algorithm execution information comprises a target weather algorithm ID, target weather algorithm parameters, a target weather algorithm execution command, a target weather algorithm execution user and an executable computing node corresponding to the target weather algorithm, the resource state information comprises second resource state information corresponding to the target weather algorithm, and the second resource state information comprises a CPU (central processing unit) and a memory amount required for running the target weather algorithm;
wherein, the scheduling client is specifically configured to:
Analyzing the target weather algorithm scheduling instruction to obtain a target weather algorithm ID, target weather algorithm parameters, a target weather algorithm execution command, a target weather algorithm execution user and second resource state information;
determining whether to schedule a target weather algorithm based on the remaining resource information and the second resource status information;
if yes, executing user scheduling and starting to operate the target weather algorithm based on the target weather algorithm ID, the target weather algorithm parameters, the target weather algorithm execution command and the target weather algorithm;
if not, the target weather algorithm scheduling failure information is sent to the scheduling server through the gRPC protocol.
7. The state-aware-based task scheduling system according to claim 6, wherein the scheduling server is specifically configured to:
and counting and updating the data source to report rate corresponding to the data source message based on the four-level coding and the data observation time in the data source message for each data source message, and storing the updated data source to report rate into a Redis cache.
8. A state-aware based task scheduling device, characterized in that it comprises a processor, a memory and a state-aware based task scheduler stored on the memory and executable by the processor, wherein the state-aware based task scheduler, when executed by the processor, implements the steps of the state-aware based task scheduling method according to any of claims 1 to 5.
9. A computer readable storage medium, wherein a state-aware based task scheduler is stored thereon, wherein the state-aware based task scheduler, when executed by a processor, implements the steps of the state-aware based task scheduling method according to any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311310819.3A CN117056058B (en) | 2023-10-11 | 2023-10-11 | Task scheduling method, system, equipment and storage medium based on state awareness |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311310819.3A CN117056058B (en) | 2023-10-11 | 2023-10-11 | Task scheduling method, system, equipment and storage medium based on state awareness |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117056058A CN117056058A (en) | 2023-11-14 |
| CN117056058B true CN117056058B (en) | 2024-02-27 |
Family
ID=88669557
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311310819.3A Active CN117056058B (en) | 2023-10-11 | 2023-10-11 | Task scheduling method, system, equipment and storage medium based on state awareness |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117056058B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109815019A (en) * | 2019-02-03 | 2019-05-28 | 普信恒业科技发展（北京）有限公司 | Method for scheduling task, device, electronic equipment and readable storage medium storing program for executing |
| WO2020140683A1 (en) * | 2019-01-04 | 2020-07-09 | 深圳壹账通智能科技有限公司 | Task scheduling method and apparatus, computer device, and storage medium |
| CN111767433A (en) * | 2020-06-23 | 2020-10-13 | Oppo（重庆）智能科技有限公司 | Data processing method, device, storage medium and terminal |
| CN113946636A (en) * | 2020-07-15 | 2022-01-18 | 华风爱科气象科技(北京)有限公司 | Meteorological data processing method and device and readable storage medium |
| CN115525680A (en) * | 2022-09-21 | 2022-12-27 | 京信数据科技有限公司 | Data processing job scheduling method and device, computer equipment and storage medium |
-
2023
- 2023-10-11 CN CN202311310819.3A patent/CN117056058B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020140683A1 (en) * | 2019-01-04 | 2020-07-09 | 深圳壹账通智能科技有限公司 | Task scheduling method and apparatus, computer device, and storage medium |
| CN109815019A (en) * | 2019-02-03 | 2019-05-28 | 普信恒业科技发展（北京）有限公司 | Method for scheduling task, device, electronic equipment and readable storage medium storing program for executing |
| CN111767433A (en) * | 2020-06-23 | 2020-10-13 | Oppo（重庆）智能科技有限公司 | Data processing method, device, storage medium and terminal |
| CN113946636A (en) * | 2020-07-15 | 2022-01-18 | 华风爱科气象科技(北京)有限公司 | Meteorological data processing method and device and readable storage medium |
| CN115525680A (en) * | 2022-09-21 | 2022-12-27 | 京信数据科技有限公司 | Data processing job scheduling method and device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117056058A (en) | 2023-11-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10303509B2 (en) | Distributed task system based on internet of things and service processing method for distributed tasks based on the same | |
| US8930521B2 (en) | Method, apparatus, and computer program product for enabling monitoring of a resource | |
| CN111625331B (en) | Task scheduling method, device, platform, server and storage medium | |
| EP2503733A1 (en) | Data collecting method, data collecting apparatus and network management device | |
| CN110581773A (en) | automatic service monitoring and alarm management system | |
| US8538793B2 (en) | System and method for managing real-time batch workflows | |
| CN112882813A (en) | Task scheduling method, device and system and electronic equipment | |
| CN112749221A (en) | Data task scheduling method and device, storage medium and scheduling tool | |
| CN110231998B (en) | Detection method and device for distributed timing task and storage medium | |
| CN112579304A (en) | Resource scheduling method, device, equipment and medium based on distributed platform | |
| CN110611707A (en) | Task scheduling method and device | |
| WO2021204013A1 (en) | Intelligent dispatching method, apparatus and device, and storage medium | |
| CN111026602A (en) | Health inspection scheduling management method and device of cloud platform and electronic equipment | |
| US20170078049A1 (en) | Freshness-sensitive message delivery | |
| US20220150221A1 (en) | Utilizing machine learning models to determine customer care actions for telecommunications network providers | |
| CN110322143B (en) | Model materialization management method, device, equipment and computer storage medium | |
| CN108199920A (en) | Monitor the method and server of the performance of the server in cloud platform | |
| CN109409948B (en) | Transaction abnormity detection method, device, equipment and computer readable storage medium | |
| CN117056058B (en) | Task scheduling method, system, equipment and storage medium based on state awareness | |
| CN112788010B (en) | Script processing method, device, medium and equipment based on threat event | |
| CN113760634A (en) | Data processing method and device | |
| CN115470025A (en) | Intelligent root cause analysis method, device, medium and equipment in distributed cloud scene | |
| CN111211938B (en) | Biological information software monitoring system and method | |
| CN111082964B (en) | Distribution method and device of configuration information | |
| CN113986495A (en) | Task execution method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |