CN116401133A - Method, device, equipment and storage medium for predicting end time - Google Patents

Method, device, equipment and storage medium for predicting end time Download PDF

Info

Publication number
CN116401133A
CN116401133A CN202310375867.4A CN202310375867A CN116401133A CN 116401133 A CN116401133 A CN 116401133A CN 202310375867 A CN202310375867 A CN 202310375867A CN 116401133 A CN116401133 A CN 116401133A
Authority
CN
China
Prior art keywords
task
time
running
current monitoring
current
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310375867.4A
Other languages
Chinese (zh)
Inventor
唐坤
王托
陈晓倩
冯朝明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Century TAL Education Technology Co Ltd
Original Assignee
Beijing Century TAL Education Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Century TAL Education Technology Co Ltd filed Critical Beijing Century TAL Education Technology Co Ltd
Priority to CN202310375867.4A priority Critical patent/CN116401133A/en
Publication of CN116401133A publication Critical patent/CN116401133A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
    • G06F11/3419—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment by assessing time

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The present disclosure relates to an end time prediction method, apparatus, device, and storage medium. The method comprises the following steps: when the current running state of the current monitoring task is the running state, reading the first starting time of the current monitoring task from the first running characteristic of the current monitoring task, and processing the first running characteristic and the second running characteristic of the related task of the current monitoring task by utilizing a pre-trained running time prediction model to obtain the first running time of the current monitoring task, and further predicting the ending time of the current monitoring task based on the first running time and the first starting time. Therefore, when the task running environment changes or an upstream task of the task running environment is abnormal, the running characteristics are processed by using the model to accurately determine the ending time, and when the monitoring time dynamically changes, the ending time can be dynamically determined, so that the reliability and the flexibility of the ending time prediction are ensured.

Description

Method, device, equipment and storage medium for predicting end time
Technical Field
The disclosure relates to the field of computer technology, and in particular, to a method, a device, equipment and a storage medium for predicting end time.
Background
In the big data development scenario, there are many tasks scheduled in the task scheduling system (for example, airlow), and in order to maintain the tasks, so as to ensure the normal output of the task data, the end time of each task needs to be monitored or predicted.
In the related art, only the historical running time of a task is simply used for predicting the ending time of the task, however, when the running environment of the task changes or an abnormality occurs in an upstream task, the prediction accuracy is not high, and the real-time dynamic prediction of the ending time cannot be realized. Therefore, providing a method for predicting the ending time with high accuracy and real-time dynamic performance is a technical problem that needs to be solved at present.
Disclosure of Invention
In order to solve the technical problems, the present disclosure provides an end time prediction method, an apparatus, a device and a storage medium.
In a first aspect, the present disclosure provides an end time prediction method, the method comprising:
acquiring a current running state of a current monitoring task at any monitoring time, wherein the current monitoring task is any task in a pre-constructed monitored task chain, and the monitored task chain also comprises related tasks with dependency relationship with the current monitoring task;
If the current running state is the running state, reading a first starting time of the current monitoring task from a first running characteristic of the current monitoring task;
processing the first operation feature and the second operation feature of the related task by using a pre-trained operation duration prediction model to obtain a first operation duration of the current monitoring task;
and predicting the ending time of the current monitoring task based on the first running duration and the first starting time.
In a second aspect, the present disclosure provides an end time prediction apparatus, comprising:
the system comprises an acquisition module, a monitoring module and a control module, wherein the acquisition module is used for acquiring the current running state of a current monitoring task at any monitoring time, the current monitoring task is any task in a pre-constructed monitored task chain, and the monitored task chain also comprises related tasks with dependency relationship with the current monitoring task;
the reading module is used for reading the first starting time of the current monitoring task from the first operation characteristic of the current monitoring task if the current operation state is the running state;
the determining module is used for processing the first operation characteristic and the second operation characteristic of the related task by utilizing a pre-trained operation duration prediction model to obtain the first operation duration of the current monitoring task;
And the prediction module is used for predicting the ending time of the current monitoring task based on the first running time and the first starting time.
In a third aspect, embodiments of the present disclosure further provide an electronic device, including:
a processor;
a memory for storing executable instructions;
wherein the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method provided in the first aspect.
In a fourth aspect, embodiments of the present disclosure further provide a computer readable storage medium having a computer program stored thereon, wherein the storage medium stores the computer program, which when executed by a processor, causes the processor to implement the method provided in the first aspect above.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages:
the method, the device, the equipment and the storage medium for predicting the ending time acquire the current running state of a current monitoring task at any monitoring time, wherein the current monitoring task is any task in a pre-constructed monitored task chain, and the monitored task chain also comprises related tasks with dependency relationship with the current monitoring task; if the current running state is the running state, reading a first starting time of the current monitoring task from a first running characteristic of the current monitoring task; processing the first operation characteristics and the second operation characteristics of related tasks by using a pre-trained operation duration prediction model to obtain the first operation duration of the current monitoring task; and predicting the ending time of the current monitoring task based on the first running duration and the first starting time. Therefore, when the task running environment changes or an upstream task of the task running environment is abnormal, the running characteristics of a plurality of tasks with dependency relations on the whole monitored task chain can be utilized, the first running time of the current monitoring task is determined by combining a running time prediction model, and then the ending time of the current monitoring task is determined based on the first starting time and the first running time, so that the accuracy of ending time prediction is improved, when the monitoring time changes, the first starting time and the first running time are determined again, a new ending time is obtained, the effect of dynamically determining the ending time of the current monitoring task is achieved, and the reliability and the flexibility of ending time prediction are ensured.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments of the present disclosure or the solutions in the prior art, the drawings that are required for the description of the embodiments or the prior art will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.
Fig. 1 is a flowchart of an end time prediction method according to an embodiment of the present disclosure;
FIG. 2 is a flowchart of another method for predicting an end time according to an embodiment of the disclosure;
FIG. 3 is a schematic diagram of a tree-structured monitored task chain according to an embodiment of the present disclosure;
FIG. 4 is a logic diagram of a splice feature generation provided by an embodiment of the present disclosure;
fig. 5 is a logic schematic diagram of a task scheduling system monitoring method according to an embodiment of the present disclosure;
fig. 6 is a schematic structural diagram of an end time prediction apparatus according to an embodiment of the present disclosure;
Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure have been shown in the accompanying drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order and/or performed in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
The term "including" and variations thereof as used herein are intended to be open-ended, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments. Related definitions of other terms will be given in the description below. It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units.
It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise.
The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
In order to accurately and dynamically predict the end time of a task, an end time prediction method provided by an embodiment of the present disclosure is described below with reference to fig. 1 to 5. In the embodiments of the present disclosure, the end time prediction method may be performed by an electronic device or a server. The electronic device may include a mobile phone, a tablet computer, a desktop computer, a notebook computer, and other devices with communication functions. The server may be a cloud server or a server cluster, or other devices with storage and computing functions. Note that the following embodiments are exemplarily explained with the electronic device as an execution subject.
Fig. 1 shows a flowchart of an end time prediction method according to an embodiment of the present disclosure.
As shown in fig. 1, the end time prediction method may include the following steps.
S110, acquiring a current running state of a current monitoring task at any monitoring time, wherein the current monitoring task is any task in a pre-constructed monitored task chain, and the monitored task chain also comprises related tasks with dependency relation with the current monitoring task.
In this embodiment, when it is required to monitor the end time of a task in a task scheduling system (for example, airlow), a pre-built monitored task chain is obtained, any task on the monitored task chain is used as a current monitoring task, and an operation state at any monitoring time is obtained as the current operation state.
The monitored task chain is a complete task chain formed by each task in the task scheduling system. Specifically, the monitored task chain includes a current monitoring task and related tasks having a dependency relationship with the current monitoring task. Optionally, the related task may be an upstream task of the current monitoring task, or may be a downstream task of the current monitoring task.
In this embodiment, optionally, S110 specifically includes: reading current identification information carried by a pre-configured target interface; and matching the current identification information with the identification information respectively corresponding to the predetermined operation states, and determining the current operation state from the predetermined operation states according to the matching result.
The target interface is a virtual interface which is preconfigured and used for acquiring the running state of the task. The identification information is used for representing the running state of the task. Optionally, the running state of the task may include: the identification information corresponding to the waiting operation state is 0, the identification information corresponding to the running state is 1, and the identification information corresponding to the running completion state is 2.
Specifically, starting from a task of the last node of the monitored task chain, reading current identification information, matching the current identification information with identification information corresponding to a plurality of predetermined running states respectively, further determining the running state of the first node task according to a matching result, and repeating the operation based on the dependency relationship until the running state of the current monitored task is read, so as to obtain the current running state.
S120, if the current running state is the running state, reading a first starting time of the current monitoring task from a first running characteristic of the current monitoring task.
It will be appreciated that if the current identification information is determined to be 1 in the manner described above, then the current running state is determined to be the running state, and that during the running of the current monitoring task, the running start time and other relevant features can be stored in a database (e.g., mySQL database) so that the first start time of the current monitoring task is read directly from the database.
S130, processing the first operation feature and the second operation feature of the related task by using a pre-trained operation duration prediction model to obtain the first operation duration of the current monitoring task.
In this embodiment, before the end time prediction is performed, the running time prediction model may be trained based on the historical features of each task and the running time of each task on the monitored task chain, so that after the current monitored task is determined to be in the running state, the running time prediction model is directly used to process the first running feature of the current monitored task and the second running feature of the related task, so as to determine the first running time of the current monitored task.
The first operation characteristic is the operation information of the current monitoring task from the starting operation time to the monitoring time, and the second operation characteristic is the operation information of the related task monitored in the last round. Optionally, the first operating characteristic and the second operating characteristic may each include: the number of memories allocated to the task, the number of CPUs allocated to the task, the number of used memories in the queue, the number of used CPUs in the queue, the maximum memory in the queue, the number of containers occupied by the operation of the maximum CPU in the queue, the number of waiting containers, the number of remaining containers, the queue environment, the allocation resources and the like.
Therefore, when the task operation environment is changed or an upstream task of the task operation environment is abnormal, the operation characteristics of each task on the whole monitored task chain can be utilized, and the first operation time length of the current monitored task can be determined by combining an operation time length prediction model.
And S140, predicting the ending time of the current monitoring task based on the first running duration and the first starting time.
In this embodiment, S140 specifically includes: and adding the first running time and the first starting time to obtain the ending time of the current monitoring task.
Alternatively, the end time of the current monitoring task may be calculated as follows:
T end ＝T start +T run
wherein T is start Is the first starting time, T run Is a first operation time length, T end Is the end time of the current monitoring task.
The embodiment of the disclosure provides an end time prediction method, which is used for acquiring a current running state of a current monitoring task at any monitoring time, wherein the current monitoring task is any task in a pre-constructed monitored task chain, and the monitored task chain also comprises related tasks with dependency relationship with the current monitoring task; if the current running state is the running state, reading a first starting time of the current monitoring task from a first running characteristic of the current monitoring task; processing the first operation characteristics and the second operation characteristics of related tasks by using a pre-trained operation duration prediction model to obtain the first operation duration of the current monitoring task; and predicting the ending time of the current monitoring task based on the first running duration and the first starting time. Therefore, when the task running environment changes or an upstream task of the task running environment is abnormal, the running characteristics of a plurality of tasks with dependency relations on the whole monitored task chain can be utilized, the first running time of the current monitoring task is determined by combining a running time prediction model, and then the ending time of the current monitoring task is determined based on the first starting time and the first running time, so that the accuracy of ending time prediction is improved, when the monitoring time changes, the first starting time and the first running time are determined again, a new ending time is obtained, the effect of dynamically determining the ending time of the current monitoring task is achieved, and the reliability and the flexibility of ending time prediction are ensured.
Further, in order to ensure the safety of the scheduling system, and when the scheduling system is abnormal, the user can maintain or adjust the scheduling system early, and after executing S140, the method further includes: judging whether the ending time of the current monitoring task is within a preset alarm time range or not; if the ending time of the current monitoring task is not within the preset alarm time range, generating alarm information and sending the alarm information to a user.
The alarm time range is an alarm time line set for the task. It can be understood that if the ending time of the current monitoring task is not within the preset alarm time range, the ending time of the current monitoring task is beyond the alarm time line, alarm information is generated, and the user is informed of the alarm information in time.
Therefore, after the ending time of the current monitoring task is determined, the ending time of the current monitoring task is judged to exceed the alarm time line based on the abnormal feedback mechanism, and if the ending time of the current monitoring task exceeds the alarm time line, alarm information is sent to a user at the first time, so that the safety of a dispatching system is ensured, and the user can maintain or adjust the dispatching system early.
In another embodiment of the present disclosure, the current running state may also be a waiting running state or an ending running state, and the ending time of the current monitoring task is predicted by adopting a respective corresponding prediction mode.
In some embodiments of the present disclosure, after performing S110, the method further comprises:
s150, if the current running state is a waiting running state and the related tasks comprise upstream tasks of the current monitoring task, determining second starting running time of the current monitoring task based on first running features corresponding to the current monitoring task and second running features corresponding to the upstream tasks;
s160, if the current running state is a waiting running state and the related tasks do not comprise the upstream tasks of the current monitoring task, reading the second starting time of the current monitoring task from the first running characteristics corresponding to the current monitoring task;
s170, determining a second operation duration of the current monitoring task based on the operation durations of a plurality of time intervals in the first operation characteristic;
and S180, adding the second initial running time and the second running time to obtain the ending time of the current monitoring task.
Wherein, S150 specifically includes: acquiring a set starting time of a current monitoring task from a first operation characteristic, and acquiring an actual ending time of an upstream task from a second operation characteristic corresponding to the upstream task; the maximum of the start time and the actual end time is set as the second start run time.
The set starting time is the starting running time specified for the current monitoring task, the actual ending time is the time when the actual running of the upstream task ends, and the set starting time is greater than or equal to the actual ending time of the last running upstream task.
It will be appreciated that the current monitoring task may be run immediately after the end of the upstream task run, or at a set start time of the current monitoring task for a period of time after the end of the upstream task run. Alternatively, the maximum value of the set start time and the actual end time may be the set start time, or may be the set start time or the actual end time of the last running upstream task.
Wherein, S170 specifically includes: sequencing the operation time lengths of the time intervals according to a preset sequence, and determining sequencing positions corresponding to the operation time lengths of the time intervals; the operation duration at the median position is selected from the operation durations of the plurality of time intervals as the second operation duration.
Specifically, the past operation time of the current monitoring task is divided into a plurality of time intervals, each time interval can be any value of 5 minutes, 10 minutes and the like, then the operation time periods in the time intervals are arranged in a reverse order or a positive order, the ordering position corresponding to the operation time period of each time interval is determined, and finally the operation time period at the middle position is selected as the second operation time period.
In other embodiments, S170 specifically includes: and averaging the running time lengths of the time intervals, and taking the calculated average value as a second running time length.
In some embodiments of the present disclosure, after performing S110, the method further comprises:
and if the current running state is a running completion state, reading the ending time of the current monitoring task from the first running characteristic.
It can be understood that, during the running process of the current monitoring task, the corresponding first running characteristic is stored in the database (for example, mySQL database) in real time, and when the current monitoring task is detected to be in the running completion state, the ending time of the current monitoring task is directly read from the stored first running characteristic in the database.
Thus, the end time of the current monitoring task is adaptively determined based on the stored data and/or the set data while the current operating state is in the waiting operating state or the operation completion state.
In yet another embodiment of the present disclosure, in order to be able to effectively use the run-length prediction model to determine the first run-length of the current monitored task, a monitored task chain needs to be built and a run-time prediction model needs to be trained based on historical features of each task on the monitored task chain before the current monitored task is predicted.
Fig. 2 shows a flowchart of another end time prediction method provided by an embodiment of the present disclosure.
As shown in fig. 2, the end time prediction method may include the following steps.
S210, taking a history monitoring task as a root node, taking an upstream task of the history monitoring task as a leaf node, and constructing a monitored task chain of a tree structure based on the dependency relationship between the history monitoring task and the upstream task.
Wherein, the historical monitoring task refers to a monitored task in a historical period of time before the monitoring time.
Specifically, when the history monitoring task is obtained, an upstream task with a dependency relationship with the history monitoring task is obtained from a database (e.g. a Hadoop database) according to a task identifier (e.g. an identity Identifier (ID)) of the history monitoring task, and a monitored task chain is formed based on the history monitoring task, the upstream task and the dependency relationship. Furthermore, layering is carried out on the monitored task chain, specifically, a historical monitoring task is taken as a root node, an upstream task of the historical monitoring task is taken as a leaf node, and the monitored task chain with a tree structure is formed by constructing the monitored task chain layer by layer in a recursion mode according to the dependency relationship.
For ease of understanding, fig. 3 shows a schematic diagram of a tree structured monitored task chain. As shown in fig. 3, the history monitoring task a is taken as a root node, the upstream tasks a_11, a_12, a_21, a_22, a_23 and a_31 are taken as leaf nodes, and connection relations between different nodes are generated based on the dependency relations, so that a monitored task chain with a tree structure is obtained.
S220, extracting the historical characteristics of each task and the running time of each task from the historical characteristics corresponding to the monitored task chain of the tree structure.
In this embodiment, the history features include a lateral feature, a longitudinal feature, and a cross feature, and then S220 specifically includes: according to a first time window, a plurality of first type features are obtained from historical features corresponding to the monitored task chains of the tree structure and serve as transverse features; according to a second time window, a plurality of second type features are obtained from historical features corresponding to the monitored task chains of the tree structure and serve as longitudinal features; and carrying out characteristic cross processing on the transverse characteristics and the longitudinal characteristics by using a multi-layer perceptron to obtain cross characteristics.
The historical features corresponding to the monitored task chains of the tree structure are stored in a database (MySQL database) in advance, information acquisition is carried out from the database for one time according to a first time window (for example, 20 minutes) to obtain transverse features, information acquisition is carried out from the database for one time according to a second time window (for example, 2 days, 5 days and 7 days) to obtain longitudinal features, then the transverse features and the longitudinal features are used as input data of a multi-layer perceptron (Multilayer Perceptron, MLP), and feature cross processing is carried out on the transverse features and the longitudinal features by utilizing feature cross capability of the multi-layer perceptron to obtain cross features.
Optionally, the transverse feature may include information such as the number of memories allocated to the task, the number of CPUs allocated to the task, the number of used memories in the queue, the number of used CPUs in the queue, the maximum memory in the queue, the number of containers occupied by the operation of the maximum number of CPUs in the queue, the number of waiting containers, the number of remaining containers, and the like.
Alternatively, the longitudinal characteristics may include information on queue environment, allocated resources, etc.
S230, training a preset network based on historical characteristics of each task and the running time of each task to obtain a running time prediction model.
In this embodiment, optionally, S230 includes: splicing the transverse features, the longitudinal features and the cross features according to the row numbers corresponding to the transverse features, the longitudinal features and the cross features respectively to generate splicing features of each task; training a preset network based on the splicing characteristics of each task and the running time of each task to obtain a running time prediction model.
Specifically, the rows of the transverse features, the longitudinal features and the cross features are the same, but the columns of the transverse features, the longitudinal features and the cross features are different, and a plurality of columns corresponding to the transverse features, the longitudinal features and the cross features respectively can be spliced according to the rows of the three types of features to generate the spliced features of each task, so that the features for model training are obtained.
Alternatively, the splice characteristics of each task may be determined as follows:
Fea final ＝concat(Fea row ，Fea col ，Fea cross )
wherein Fea final Is the splicing characteristic of each task, fea row Is a transverse feature, fea row Is a longitudinal feature, fea cross Is a cross feature.
For easy understanding, referring to the logic schematic diagram generated by the stitching features shown in fig. 4, first, after the transverse features and the longitudinal features are acquired, the transverse features and the longitudinal features are input into the multi-layer perceptron to obtain intersecting features, and then, the transverse features, the intersecting features and the longitudinal features are subjected to feature stitching to obtain the stitching features of each task.
Furthermore, iterative training is carried out on the preset network by utilizing the splicing characteristics of each task and the running time of each task to obtain a running time prediction model. Alternatively, the preset network may include, but is not limited to, a LightGBM model.
In order to improve the robustness of the operation duration prediction model, 80% of samples can be used as training samples, the remaining 20% can be used as verification samples when the operation duration prediction model is trained, the model is evaluated on a verification set and a test set according to evaluation indexes, and the operation duration prediction model is obtained when the evaluation indexes meet preset conditions. Alternatively, the evaluation index may include, but is not limited to: determining coefficients (Coefficient of determination, R) 2 ) Mean absolute error (Mean Absolute Error, MAE), mean square error (Mean Square Error, RMAE), root mean square error (Root Mean Square Error, MSE), mean absolute percent error (Mean Absolute Percentage Error, MAPE).
Therefore, before the current monitoring task is predicted, a monitored task chain of a tree structure is constructed according to the historical monitoring task, an upstream task of the historical monitoring task and a dependency relationship, and based on historical characteristics corresponding to the monitored task chain of the tree structure, a running time prediction model is trained, and further the trained running time prediction model is utilized to predict the first running time of the current monitoring task.
S240, acquiring a current running state of a current monitoring task at any monitoring time, wherein the current monitoring task is any task in a pre-constructed monitored task chain, and the monitored task chain further comprises related tasks with dependency relation with the current monitoring task.
S250, if the current running state is the running state, reading the first starting time of the current monitoring task from the first running characteristic of the current monitoring task.
S260, processing the first operation feature and the second operation feature of the related task by using a pre-trained operation duration prediction model to obtain the first operation duration of the current monitoring task.
S270, predicting the ending time of the current monitoring task based on the first running duration and the first starting time.
Wherein, S240-S270 are similar to S110-S140, and are not described herein.
In yet another embodiment of the present disclosure, the above scheme is explained in its entirety. Fig. 5 shows a logic schematic diagram of a task scheduling system monitoring method according to an embodiment of the present disclosure, where, as shown in fig. 5, the task scheduling system monitoring method includes: the method comprises a monitored task chain construction and model training stage, an ending time prediction stage and an early warning stage.
The monitored task chain construction and model training stage comprises the following steps:
s1, constructing a monitored task chain with a tree structure.
Specifically, a history monitoring task is taken as a root node, an upstream task of the history monitoring task is taken as a leaf node, and a monitored task chain of a tree structure is constructed based on the dependency relationship between the history monitoring task and the upstream task
S2, training a running time prediction model.
Specifically, extracting historical features of each task and the running time of each task from historical features corresponding to a monitored task chain of a tree structure; training a preset network based on the historical characteristics of each task and the running time of each task to obtain a running time prediction model.
An end time prediction phase comprising:
s3, determining the current running state of the current monitoring task at any monitoring time.
Specifically, current identification information carried by a pre-configured target interface is read, the current identification information is matched with identification information corresponding to a plurality of pre-determined running states respectively, and the current running state is determined from the plurality of pre-determined running states according to a matching result.
S4, the current running state is the running state, and the first starting time of the current monitoring task is read from the first running characteristics of the current monitoring task.
S5, processing the first operation feature and the second operation feature of the related task by using a pre-trained operation time length prediction model to obtain the first operation time length of the current monitoring task, and predicting the ending time of the current monitoring task based on the first operation time length and the first starting time.
Specifically, the first operation duration and the first starting time are added to obtain the ending time of the current monitoring task.
S6, determining the second initial running time of the current monitoring task when the current running state is the waiting running state.
Specifically, if the related task includes an upstream task of the current monitoring task, acquiring a set starting time of the current monitoring task from a first operation feature, acquiring an actual ending time of the upstream task from a second operation feature corresponding to the upstream task, and taking the maximum value of the set starting time and the actual ending time as a second starting operation time; and if the related task does not comprise the upstream task of the current monitoring task, reading the second starting time of the current monitoring task from the first operation characteristic corresponding to the current monitoring task.
S7, determining a second operation duration of the current monitoring task based on the operation durations of the plurality of time intervals in the first operation characteristic, and predicting the ending time of the current monitoring task based on the second operation duration and the second starting time.
Specifically, the second starting running time and the second running time are added to obtain the ending time of the current monitoring task.
S8, the current running state is a running completion state, and the ending time of the current monitoring task is read from the first running characteristic.
An early warning stage comprising:
and S9, judging whether the ending time of the current monitoring task is within a preset alarm time range.
Specifically, if the ending time of the current monitoring task is within the preset alarm time range, executing S10, otherwise, ending.
S10, generating alarm information and sending the alarm information to a user.
The embodiment of the present disclosure further provides an end time prediction apparatus for implementing the end time prediction method described above, and is described below with reference to fig. 6. In an embodiment of the disclosure, the ending time prediction apparatus may be an electronic device or a server. The electronic device may include a mobile phone, a tablet computer, a desktop computer, a notebook computer, and other devices with communication functions. The server may be a cloud server or a server cluster, or other devices with storage and computing functions.
Fig. 6 shows a schematic structural diagram of an end time prediction apparatus according to an embodiment of the present disclosure.
As shown in fig. 6, the end time prediction apparatus 600 may include:
an obtaining module 610, configured to obtain a current running state of a current monitoring task at any monitoring time, where the current monitoring task is any task in a pre-built monitored task chain, and the monitored task chain further includes a related task having a dependency relationship with the current monitoring task;
a reading module 620, configured to read a first start time of the current monitoring task from a first operation feature of the current monitoring task if the current operation state is an ongoing operation state;
a determining module 630, configured to process the first operation feature and the second operation feature of the related task by using a pre-trained operation duration prediction model, so as to obtain a first operation duration of the current monitoring task;
a prediction module 640, configured to predict an end time of the current monitoring task based on the first running duration and the first start time.
The embodiment of the disclosure provides an end time prediction device, which is used for acquiring a current running state of a current monitoring task at any monitoring time, wherein the current monitoring task is any task in a pre-constructed monitored task chain, and the monitored task chain also comprises related tasks with dependency relationship with the current monitoring task; if the current running state is the running state, reading a first starting time of the current monitoring task from a first running characteristic of the current monitoring task; processing the first operation characteristics and the second operation characteristics of related tasks by using a pre-trained operation duration prediction model to obtain the first operation duration of the current monitoring task; and predicting the ending time of the current monitoring task based on the first running duration and the first starting time. Therefore, when the task running environment changes or an upstream task of the task running environment is abnormal, the running characteristics of a plurality of tasks with dependency relations on the whole monitored task chain can be utilized, the first running time of the current monitoring task is determined by combining a running time prediction model, and then the ending time of the current monitoring task is determined based on the first starting time and the first running time, so that the accuracy of ending time prediction is improved, when the monitoring time changes, the first starting time and the first running time are determined again, a new ending time is obtained, the effect of dynamically determining the ending time of the current monitoring task is achieved, and the reliability and the flexibility of ending time prediction are ensured.
In some alternative embodiments, the acquisition module 610 includes:
the identification reading unit is used for reading the current identification information carried by the pre-configured target interface;
the identification matching unit is used for matching the current identification information with the identification information respectively corresponding to a plurality of preset running states, and determining the current running state from the preset running states according to a matching result.
In some alternative embodiments, the prediction module 640 is specifically configured to:
and adding the first operation time length and the first starting time to obtain the ending time of the current monitoring task.
In some alternative embodiments, the apparatus further comprises:
the first determining module is configured to determine, if the current running state is a waiting running state and the related task includes an upstream task of the current monitoring task, a second starting running time of the current monitoring task based on a first running feature corresponding to the current monitoring task and a second running feature corresponding to the upstream task;
the second determining module is used for reading a second starting time of the current monitoring task from the first operation characteristics corresponding to the current monitoring task if the current operation state is a waiting operation state and the related task does not comprise an upstream task of the current monitoring task;
The third determining module is used for determining a second operation duration of the current monitoring task based on the operation durations of a plurality of time intervals in the first operation characteristic;
and a fourth determining module, configured to add the second starting running time and the second running time to obtain an end time of the current monitoring task.
In some alternative embodiments, the first determining module is specifically configured to:
acquiring a set starting time of the current monitoring task from the first operation characteristic, and acquiring an actual ending time of the upstream task from a second operation characteristic corresponding to the upstream task;
and taking the maximum value of the set starting time and the actual ending time as the second starting running time.
In some alternative embodiments, the third determining module is specifically configured to: sequencing the operation time lengths of the time intervals according to a preset sequence, and determining sequencing positions corresponding to the operation time lengths of the time intervals;
and selecting the operation duration at the median position from the operation durations of the time intervals as the second operation duration.
In some alternative embodiments, the apparatus further comprises:
And the ending time reading module is used for reading the ending time of the current monitoring task from the first operation characteristic if the current operation state is an operation completion state.
In some alternative embodiments, the apparatus further comprises:
the construction module is used for taking a history monitoring task as a root node, taking an upstream task of the history monitoring task as a leaf node and constructing a monitored task chain of a tree structure based on the dependency relationship between the history monitoring task and the upstream task;
the extraction module is used for extracting the historical characteristics of each task and the running time of each task from the historical characteristics corresponding to the monitored task chain of the tree structure;
the training module is used for training a preset network based on the historical characteristics of each task and the running time of each task to obtain the running time prediction model.
In some alternative embodiments, the history features include lateral features, longitudinal features, and cross features, and the extracting module includes:
the first acquisition unit is used for acquiring a plurality of first type features from the historical features corresponding to the monitored task chains of the tree structure according to a first time window, and the first type features are used as the transverse features;
The second acquisition unit is used for acquiring a plurality of second type features from the historical features corresponding to the monitored task chains of the tree structure according to a second time window, and the second type features are used as the longitudinal features;
and the third acquisition unit is used for carrying out characteristic cross processing on the transverse characteristics and the longitudinal characteristics by using a multi-layer perceptron to obtain the cross characteristics.
In some alternative embodiments, the training module includes:
the splicing unit is used for splicing the transverse features, the longitudinal features and the cross features according to the row numbers corresponding to the transverse features, the longitudinal features and the cross features respectively to generate splicing features of each task;
the training unit is used for training the preset network based on the splicing characteristics of the tasks and the running time of the tasks to obtain the running time prediction model.
In some alternative embodiments, the apparatus further comprises:
the judging module is used for judging whether the ending time of the current monitoring task is in a preset alarm time range or not;
and the alarm module is used for generating alarm information and sending the alarm information to a user if the ending time of the current monitoring task is not in the preset alarm time range.
It should be noted that, the ending time prediction apparatus 600 shown in fig. 6 may perform the steps in the method embodiments shown in fig. 1 to 5, and implement the processes and effects in the method embodiments shown in fig. 1 to 5, which are not described herein.
The exemplary embodiments of the present disclosure also provide an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor. The memory stores a computer program executable by the at least one processor for causing the electronic device to perform a method according to embodiments of the present disclosure when executed by the at least one processor.
The present disclosure also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is for causing the computer to perform a method according to an embodiment of the present disclosure.
The present disclosure also provides a computer program product comprising a computer program, wherein the computer program, when executed by a processor of a computer, is for causing the computer to perform a method according to an embodiment of the present disclosure.
Referring to fig. 7, a block diagram of a structure of an electronic device 700 that may be a server or a client of the present disclosure, which is an example of a hardware device that may be applied to aspects of the present disclosure, will now be described, and the electronic device 700 may be the above-described electronic device. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 7, the electronic device 700 includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 702 or a computer program loaded from a storage unit 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data required for the operation of the electronic device 700 may also be stored. The computing unit 701, the ROM702, and the RAM703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
Various components in the electronic device 700 are connected to the I/O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 may be any type of device capable of inputting information to the electronic device 700, and the input unit 706 may receive input numeric or character information and generate key signal inputs related to user settings and/or function controls of the electronic device. The output unit 707 may be any type of device capable of presenting information and may include, but is not limited to, a display, speakers, video/audio output terminals, vibrators, and/or printers. Storage unit 708 may include, but is not limited to, magnetic disks, optical disks. The communication unit 709 allows the electronic device 700 to exchange information/data with other devices through computer networks, such as the internet, and/or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and/or chipsets, such as bluetooth (TM) devices, wiFi devices, wiMax devices, cellular communication devices, and/or the like.
The computing unit 701 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 701 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the end time prediction method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 700 via the ROM702 and/or the communication unit 709. In some embodiments, the computing unit 701 may be configured to perform the end time prediction method by any other suitable means (e.g. by means of firmware).
Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
It should be noted that in this document, relational terms such as "first" and "second" and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.
The above is merely a specific embodiment of the disclosure to enable one skilled in the art to understand or practice the disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (14)

1. An end time prediction method, comprising:
acquiring a current running state of a current monitoring task at any monitoring time, wherein the current monitoring task is any task in a pre-constructed monitored task chain, and the monitored task chain also comprises related tasks with dependency relationship with the current monitoring task;
if the current running state is the running state, reading a first starting time of the current monitoring task from a first running characteristic of the current monitoring task;
processing the first operation feature and the second operation feature of the related task by using a pre-trained operation duration prediction model to obtain a first operation duration of the current monitoring task;
And predicting the ending time of the current monitoring task based on the first running duration and the first starting time.
2. The method of claim 1, wherein the obtaining the current running state of the current monitoring task at any monitoring time comprises:
reading current identification information carried by a pre-configured target interface;
and matching the current identification information with the identification information respectively corresponding to a plurality of predetermined running states, and determining the current running state from the plurality of predetermined running states according to a matching result.
3. The method of claim 1, wherein predicting the end time of the current monitoring task based on the first run length and the first start time comprises:
and adding the first operation time length and the first starting time to obtain the ending time of the current monitoring task.
4. The method according to claim 1, wherein the method further comprises:
if the current running state is a waiting running state and the related task comprises an upstream task of the current monitoring task, determining a second starting running time of the current monitoring task based on a first running characteristic corresponding to the current monitoring task and a second running characteristic corresponding to the upstream task;
If the current running state is a waiting running state and the related task does not comprise an upstream task of the current monitoring task, reading a second starting time of the current monitoring task from a first running characteristic corresponding to the current monitoring task;
determining a second operation duration of the current monitoring task based on the operation durations of a plurality of time intervals in the first operation feature;
and adding the second initial running time and the second running time to obtain the ending time of the current monitoring task.
5. The method of claim 4, wherein the determining a second start-up run time of the current monitoring task based on the first run feature corresponding to the current monitoring task and the second run feature corresponding to the upstream task comprises:
acquiring a set starting time of the current monitoring task from the first operation characteristic, and acquiring an actual ending time of the upstream task from a second operation characteristic corresponding to the upstream task;
and taking the maximum value of the set starting time and the actual ending time as the second starting running time.
6. The method of claim 4, wherein determining the second operational duration of the current monitoring task based on the operational durations of the plurality of time intervals in the first operational characteristic comprises:
sequencing the operation time lengths of the time intervals according to a preset sequence, and determining sequencing positions corresponding to the operation time lengths of the time intervals;
and selecting the operation duration at the median position from the operation durations of the time intervals as the second operation duration.
7. The method as recited in claim 1, further comprising:
and if the current running state is a running completion state, reading the ending time of the current monitoring task from the first running characteristic.
8. The method of claim 1, wherein prior to said obtaining a current operational state of a current monitoring task at any monitoring time, the method further comprises:
taking a history monitoring task as a root node, taking an upstream task of the history monitoring task as a leaf node, and constructing a monitored task chain of a tree structure based on the dependency relationship between the history monitoring task and the upstream task;
Extracting historical characteristics of each task and operation time of each task from historical characteristics corresponding to a monitored task chain of the tree structure;
training a preset network based on the historical characteristics of each task and the running time of each task to obtain the running time prediction model.
9. The method according to claim 8, wherein the history features include a lateral feature, a longitudinal feature, and a cross feature, and the extracting the history features of each task from the history features corresponding to the monitored task chain of the tree structure includes:
according to a first time window, a plurality of first type features are obtained from historical features corresponding to the monitored task chains of the tree structure and serve as the transverse features;
according to a second time window, a plurality of second type features are obtained from historical features corresponding to the monitored task chains of the tree structure and serve as the longitudinal features;
and carrying out characteristic cross processing on the transverse characteristic and the longitudinal characteristic by using a multi-layer perceptron to obtain the cross characteristic.
10. The method according to claim 9, wherein training a preset network based on the historical features of each task and the running time of each task to obtain the running time prediction model includes:
Splicing the transverse features, the longitudinal features and the cross features according to the row numbers corresponding to the transverse features, the longitudinal features and the cross features respectively to generate splicing features of each task;
training the preset network based on the splicing characteristics of each task and the running time of each task to obtain the running time prediction model.
11. The method according to any one of claims 1 to 10, further comprising:
judging whether the ending time of the current monitoring task is within a preset alarm time range or not;
if the ending time of the current monitoring task is not within the preset alarm time range, generating alarm information and sending the alarm information to a user.
12. An end time prediction apparatus, comprising:
the system comprises an acquisition module, a monitoring module and a control module, wherein the acquisition module is used for acquiring the current running state of a current monitoring task at any monitoring time, the current monitoring task is any task in a pre-constructed monitored task chain, and the monitored task chain also comprises related tasks with dependency relationship with the current monitoring task;
The reading module is used for reading the first starting time of the current monitoring task from the first operation characteristic of the current monitoring task if the current operation state is the running state;
the determining module is used for processing the first operation characteristic and the second operation characteristic of the related task by utilizing a pre-trained operation duration prediction model to obtain the first operation duration of the current monitoring task;
and the prediction module is used for predicting the ending time of the current monitoring task based on the first running time and the first starting time.
13. An electronic device, comprising:
a processor;
a memory for storing executable instructions;
wherein the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method of any of the preceding claims 1-11.
14. A computer readable storage medium, on which a computer program is stored, characterized in that the storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the method of any of the preceding claims 1-11.
CN202310375867.4A 2023-04-10 2023-04-10 Method, device, equipment and storage medium for predicting end time Pending CN116401133A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310375867.4A CN116401133A (en) 2023-04-10 2023-04-10 Method, device, equipment and storage medium for predicting end time

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310375867.4A CN116401133A (en) 2023-04-10 2023-04-10 Method, device, equipment and storage medium for predicting end time

Publications (1)

Publication Number Publication Date
CN116401133A true CN116401133A (en) 2023-07-07

Family

ID=87011916

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310375867.4A Pending CN116401133A (en) 2023-04-10 2023-04-10 Method, device, equipment and storage medium for predicting end time

Country Status (1)

Country Link
CN (1) CN116401133A (en)

Similar Documents

Publication Publication Date Title
CN109992473B (en) Application system monitoring method, device, equipment and storage medium
US10832150B2 (en) Optimized re-training for analytic models
CN111967917B (en) Method and device for predicting user loss
US9471383B2 (en) Task allocation in a computing environment
CN112925587A (en) Method and apparatus for initializing applications
EP4060496A2 (en) Method, apparatus, device and storage medium for running inference service platform
CN117033146B (en) Identification method, device, equipment and medium for appointed consensus contract execution process
CN116401133A (en) Method, device, equipment and storage medium for predicting end time
CN116432069A (en) Information processing method, service providing method, device, equipment and medium
US11394631B2 (en) Determining changes in a performance of a server
CN114157578A (en) Network state prediction method and device
CN113052509A (en) Model evaluation method, model evaluation apparatus, electronic device, and storage medium
US20200250572A1 (en) Implementing a computer system task involving nonstationary streaming time-series data by removing biased gradients from memory
CN117130873B (en) Task monitoring method and device
CN114816758B (en) Resource allocation method and device
CN113704016B (en) Cloud function component diagnosis method, device, equipment and storage medium
CN113420227B (en) Training method of click rate estimation model, click rate estimation method and device
CN113535187B (en) Service online method, service updating method and service providing method
CN113705594B (en) Image identification method and device
US20240028669A1 (en) Experimentally validating causal graphs
CN114356513B (en) Task processing method and device for cluster mode
US20240193538A1 (en) Temporal supply-related forecasting using artificial intelligence techniques
CN113127159B (en) Application processing method and device, electronic equipment and storage medium
CN111767085B (en) Storm platform parameter configuration method and apparatus
CN114237856A (en) Operation type identification method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination