CN113778776A - Method and device for early warning task abnormity and storage medium - Google Patents
Method and device for early warning task abnormity and storage medium Download PDFInfo
- Publication number
- CN113778776A CN113778776A CN202010579102.9A CN202010579102A CN113778776A CN 113778776 A CN113778776 A CN 113778776A CN 202010579102 A CN202010579102 A CN 202010579102A CN 113778776 A CN113778776 A CN 113778776A
- Authority
- CN
- China
- Prior art keywords
- task
- time period
- data volume
- current time
- early warning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 230000002159 abnormal effect Effects 0.000 claims abstract description 23
- 238000004364 calculation method Methods 0.000 claims description 48
- 238000004590 computer program Methods 0.000 claims description 12
- 230000008569 process Effects 0.000 abstract description 8
- 238000013024 troubleshooting Methods 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 17
- 238000013507 mapping Methods 0.000 description 10
- 230000005856 abnormality Effects 0.000 description 8
- 238000012545 processing Methods 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000010223 real-time analysis Methods 0.000 description 3
- 238000012937 correction Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000000779 smoke Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3017—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is implementing multitasking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The disclosure provides a method and a device for early warning task abnormity and a storage medium, and relates to the field of computers. The method comprises the steps of determining the predicted data volume of a task in the current time period according to the actual data volume of the task in the past time period, and giving an early warning on whether the task is abnormal or not according to the difference value between the predicted data volume and the actual data volume of the task in the current time period, so that the task is timely warned in the task running process, and problem troubleshooting and related intervention can be conveniently carried out at the first time when the problem occurs.
Description
Technical Field
The present disclosure relates to the field of computers, and in particular, to a method and an apparatus for performing an early warning on task exception, and a storage medium.
Background
In the big data era, enterprises run a large number of computing tasks every day to realize different business scenarios.
In order to guarantee the agreed quality of service, the tasks need to be monitored and analyzed. At present, a post-analysis mode is adopted for task analysis by a big data platform, namely, after an abnormal condition of a task is found, task information, log information and the like corresponding to the task are analyzed.
The inventor finds that the post analysis mode has certain delay, and the problem investigation and the related intervention cannot be carried out at the first time when the problem occurs.
Disclosure of Invention
According to the method and the device, the predicted data volume of the task in the current time period is determined according to the actual data volume of the task in the past time period, and whether the task is abnormal or not is pre-warned according to the difference value between the predicted data volume and the actual data volume of the task in the current time period, so that the task is pre-warned in time in the task running process, and problem troubleshooting and related intervention can be performed at the first time when the problem occurs.
Some embodiments of the present disclosure provide a method for early warning task exception, including:
acquiring the actual data volume of the task in the past time period;
determining the predicted data volume of the task in the current time period according to the actual data volume of the task in the past time period;
acquiring the actual data volume of the task in the current time period;
and according to the difference value between the predicted data volume and the actual data volume of the task in the current time period, early warning is carried out on whether the task is abnormal.
In some embodiments, the past time period comprises: a preset first number of past time periods closest to the current time period.
In some embodiments, determining the predicted amount of data for the task at the current time period comprises:
performing weighted average calculation on the actual data volume of a preset first number of past time periods closest to the current time period, wherein the closer the past time period is to the current time period, the greater the corresponding weight of the actual data volume of the past time period is;
and determining the predicted data volume of the task in the current time period according to the calculation result.
In some embodiments, determining the predicted data amount of the task in the current time period according to the calculation result comprises:
judging whether the difference value between the calculation result and the actual data volume of the past time period closest to the current time period is larger than a preset threshold value or not;
if not, determining the calculation result as the predicted data volume of the task in the current time period;
and if so, correcting the calculation result, and determining the corrected calculation result as the predicted data volume of the task in the current time period.
In some embodiments, modifying the calculation comprises:
calculating an average relative error between the actual data volume and the predicted data volume of a preset second number of past time segments nearest to the current time segment;
and correcting the calculation result by using the average relative error.
In some embodiments, the early warning whether the task is abnormal includes:
if the difference value between the predicted data volume and the actual data volume of the task in the current time period exceeds a corresponding early warning threshold value of the task set by an early warning rule, sending out an early warning that the task is abnormal; wherein the pre-warning rules are configurable.
In some embodiments, the pre-warning rules include: the types of the tasks needing early warning and the early warning threshold value corresponding to each type of the tasks needing early warning.
In some embodiments, the types of tasks that require pre-warning include: one or more of a map task, a map read task, a map write task, a reduce read task, a reduce write task.
In some embodiments, obtaining the actual data volume of the task at the current time period comprises: and acquiring the actual data volume of the task in the current time period in real time through an application program main control assembly or a webpage interface.
Some embodiments of the present disclosure provide an apparatus for performing an early warning on task exception, including:
a memory; and
a processor coupled to the memory, the processor configured to perform a method of pre-warning for task exceptions based on instructions stored in the memory.
Some embodiments of the present disclosure provide an apparatus for early warning task exception, including
A history data acquisition unit configured to acquire an actual data amount of the task in a past time period;
a prediction unit configured to determine a predicted data amount of the task in a current time period according to an actual data amount of the task in a past time period;
a real-time data acquisition unit configured to acquire an actual data volume of the task in a current time period;
and the early warning unit is configured to early warn whether the task is abnormal or not according to the difference value between the predicted data volume and the actual data volume of the task in the current time period.
Some embodiments of the present disclosure provide a non-transitory computer readable storage medium having stored thereon a computer program that, when executed by a processor, performs the steps of a method of pre-warning for task exceptions.
Drawings
The drawings that will be used in the description of the embodiments or the related art will be briefly described below. The present disclosure can be understood more clearly from the following detailed description, which proceeds with reference to the accompanying drawings.
It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without undue inventive faculty.
Fig. 1 illustrates a flow diagram of a method of early warning of task anomalies according to some embodiments of the present disclosure.
FIG. 2 illustrates a schematic diagram of determining a predicted data amount of a task at a current time period according to a calculation result according to some embodiments of the present disclosure.
FIG. 3 illustrates a comparison diagram of mapping actual data volume to predicted data volume of a read file according to some embodiments of the present disclosure.
Fig. 4 is a flowchart illustrating a method for early warning of task abnormalities according to further embodiments of the present disclosure.
Fig. 5 shows a schematic diagram of an apparatus for early warning of task abnormalities, according to some embodiments of the present disclosure.
Fig. 6 is a schematic diagram of an apparatus for warning task abnormalities according to further embodiments of the present disclosure.
Detailed Description
The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure.
Unless otherwise specified, "first", "second", and the like in the present disclosure are described to distinguish different objects, and are not intended to mean size, timing, or the like.
Fig. 1 illustrates a flow diagram of a method of early warning of task anomalies according to some embodiments of the present disclosure.
As shown in fig. 1, the method of this embodiment includes: step 110-.
At step 110, the actual data volume of the task over the past time period is obtained.
Wherein, the past time period includes, for example: a preset first number of past time periods closest to the current time period. The data volume of the current time period is predicted based on the actual data volume of the past time period closest to the current time period, and the prediction accuracy is improved.
Tasks (Job) include, for example, Map (Map) tasks, Reduce (Reduce) tasks, and may be further subdivided into Map Read (Map Read) tasks, Map write (Map Written) tasks, Reduce Read (Reduce Read) tasks, and Reduce write (Reduce Written) tasks, to name but not limited to these examples.
At step 120, a predicted data volume for the task at the current time period is determined based on the actual data volume for the task at the past time period.
In some embodiments, the determining 120 the predicted data amount of the task at the current time period includes:
in step 121, a weighted average calculation is performed on the actual data volumes of the past time periods of the preset first number, which are closest to the current time period, wherein the weights of the actual data volumes of the past time periods are larger as the past time periods are closer to the current time period.
The formula is expressed as follows:
wherein the actual data amount y of the past time periods t, t-1, …, t-i +1, …, t-N +1 of the preset first number N nearest to the current time period t +1t,yt-1,…,yt-i+1,…,yt-N+1，w1，w2,…,wi,…,wNAre each yt,yt-1,…,yt-i+1,…,yt-N+1Weight of (A), Mt,NThe calculation result of the weighted average is represented. Wherein the preset first number is one or more. E.g. N ═ 7, respectively, w1～w7For example, 5, 4, 3, 1, 1, 1, 1, or, w, respectively1～w7For example, 7, 6, 5, 4, 3, 2, 1, respectively, but not limited to the illustrated examples.
In step 122, the predicted data amount of the task in the current time period is determined according to the calculation result.
The above prediction method is referred to figuratively as a weighted moving average method. And the data volume of the current time period is predicted by a weighted moving average method, so that the prediction accuracy is further improved.
In some embodiments, as shown in fig. 2, the determining the predicted data amount of the task in the current time period according to the calculation result in step 122 includes:
in step 1221, it is determined whether the difference between the calculation result and the actual data amount of the past time period closest to the current time period is greater than a preset threshold value, thereby determining the rationality of the prediction result.
If not, the calculation result is determined as the predicted data amount of the task in the current time period.
y′t+1＝Mt,N
Wherein, y't+1Representing the predicted data amount, M, of the current time period t +1t,NThe calculation result of weighted average of the actual data amount of the past time period t-N +1 of the preset first number N closest to the current time period t +1 is shown.
In step 1223, if the prediction result is not reasonable, the calculation result is corrected, and the corrected calculation result is determined as the predicted data size of the task in the current time period.
y′t+1＝M′t,N
Wherein, y't+1Representing the predicted data quantity, M ', of the current time period t + 1't,NRepresents the calculation result Mt,NThe correction value of (2).
The correction of the calculation result includes, for example: calculating an average relative error between the actual data volume and the predicted data volume of a preset second number of past time segments closest to the current time segment; and correcting the calculation result by using the average relative error. For example, the average relative error is subtracted from the calculation result as a corrected calculation result. The preset second number and the preset first number may be the same or different.
The accuracy of prediction is further improved by correcting the prediction result.
In step 130, the actual data volume of the task in the current time period is obtained.
For example, the actual data volume of the task in the current time period is acquired in real time through an application program main control component or through a webpage interface.
In step 140, according to the difference between the predicted data volume and the actual data volume of the task in the current time period, whether the task is abnormal or not is pre-warned.
And if the difference value between the predicted data volume and the actual data volume of the task in the current time period exceeds a corresponding early warning threshold value of the task set by the early warning rule, sending out an early warning that the task is abnormal.
Wherein the pre-warning rules are configurable. The warning rules include, for example: the type of the tasks needing early warning, and early warning threshold values and other configurable information corresponding to each type of the tasks needing early warning. The types of tasks that need to be warned include, for example: one or more of a map, a task, a map read task, a map write task, a reduce read task, a reduce write task, but not limited to, the examples given. And the early warning mode is quickly adjusted by configuring the early warning rule.
For example, the task is a map read task, N is 7, w1～w7For example, 5, 4, 3, 1, 1, 1, 1, 1, respectively, according to the actual data volume of the mapping read file from 2019/12/16-2019/12/22, the predicted data volume of the mapping read file from 2019/12/23 is calculated to be 867.37 according to the formula in step 121, the actual data volume of the mapping read file from 2019/12/23 obtained in the running process is 872.18, and it is assumed that the early warning threshold is 30MB and i 872.18-867.37 is equal to y<30, the mapping reading task of 2019/12/23 is normal without warning. By analogy, according to the actual data volume of the mapping read files of 2019/12/17-2019/12/23 and the formula in step 121, the predicted data volume of the mapping read file of 2019/12/24 is calculated to be 867.74, the actual data volume of the mapping read file of 2019/12/24 obtained in the running process is 892.12, |892.12-867.74|<30, the mapping reading task of 2019/12/24 is normal without warning. Similarly, |981.28-886.66| does not smoke at 2019/12/27>And 30, the mapping reading task of 2019/12/27 is abnormal, and an early warning is given. FIG. 3 is a diagram showing the comparison between the actual data amount and the predicted data amount of the mapped read file during periods 2019/12/23-2019/12/31, wherein the solid line represents the actual data amount, the dotted line represents the predicted data amount, the abscissa represents the date, and the ordinate represents the data amount.
In the embodiment, the predicted data volume of the task in the current time period is determined according to the actual data volume of the task in the past time period, and whether the task is abnormal or not is pre-warned according to the difference value between the predicted data volume and the actual data volume of the task in the current time period, so that the task is pre-warned in time in the task running process, and problem troubleshooting and related intervention can be performed at the first time when the problem occurs.
With reference to fig. 4, a method for performing early warning on task exception is described below by taking Hadoop as an example.
Fig. 4 is a flowchart illustrating a method for early warning of task abnormalities according to further embodiments of the present disclosure.
The method comprises the steps of determining the predicted data volume of a task in the current time period according to the actual data volume of the task in the past time period, comparing the actual data volume of the task in the current time period with the predicted data volume in the task running process, and when the difference value of the actual data volume of the task in the current time period and the predicted data volume exceeds an early warning threshold value, for example, the data volume is sharply increased or sharply decreased, indicating that the task is possibly abnormal, alarming in the task running process, and informing relevant personnel of the abnormal situation so as to perform problem troubleshooting and relevant intervention at the first time when the problem occurs.
As shown in fig. 4, in the offline analysis, the job history service JobHistory records the historical data of the completion of the job operation, the database HBase counts the actual data volume of the job in each past time period, determines the predicted data volume of the job in the current time period by prediction according to the actual data volume of the job in each past time period, and Kafka stores the predicted data volume of the job in the current time period and provides the data volume as a data source for real-time analysis to Storm.
As shown in fig. 4, in the real-time analysis, the task data is sent to the Kafka message queue in real time through an Application Master (Application Master) component in the Resource manager Yarn (Another Resource coordinator) or through a crawled web Interface (WebUI), and the sending frequency is, for example, 1 time/30 seconds. The Kafka counts the actual data volume of the task in the current time period and provides the actual data volume to the real-time computing system as another data source for real-time analysis. In a distributed early warning system (such as Apache Eagle), a real-time computing system (such as Storm, flex, spark and the like) and a configurable Complex Event Processing (CEP) engine (such as Siddhi) are combined, and monitoring and early warning of task abnormity can be realized by configuring an early warning rule without developing a combination program by using the distributed early warning system. For example, the complex event processing engine configures various early warning rules, the real-time computing system obtains the predicted data volume and the actual data volume of various tasks in the current time period, performs data format conversion (if necessary), compares the difference between the predicted data volume and the actual data volume of the corresponding tasks in the current time period according to the various early warning rules, analyzes and outputs whether the tasks are abnormal, and performs early warning when the tasks are abnormal. The analysis results can be stored, for example, in the Kafka, Mysql, HBase databases. The configuration of the early warning rule can be realized by writing an SQL rule according to the Structured Query Language (SQL) syntax of Siddhi. Different early warning rules can be configured according to different monitoring scenes. In addition, the Siddhi can accept a plurality of data sources and a plurality of data formats, and the real-time computing system converts data streams in various formats into Siddhi streams and inputs the Siddhi streams into the Siddhi engine.
Fig. 5 shows a schematic diagram of an apparatus for early warning of task abnormalities, according to some embodiments of the present disclosure.
As shown in fig. 5, the apparatus 500 for warning task abnormality of this embodiment includes:
a history data acquisition unit 510 configured to acquire an actual data amount of the task in a past time period;
a prediction unit 520 configured to determine a predicted data amount of the task in the current time period according to an actual data amount of the task in a past time period;
a real-time data acquiring unit 530 configured to acquire an actual data amount of the task in a current time period;
and the early warning unit 540 is configured to early warn whether the task is abnormal according to the difference value between the predicted data volume and the actual data volume of the task in the current time period.
In some embodiments, the past time period comprises: a preset first number of past time periods closest to the current time period.
In some embodiments, the prediction unit 520 determines the predicted data amount of the task in the current time period, including: carrying out weighted average calculation on the actual data volume of the past time periods with the preset first quantity and the nearest distance from the current time period, wherein the closer the past time period is to the current time period, the greater the corresponding weight of the actual data volume of the past time period is; and determining the predicted data volume of the task in the current time period according to the calculation result.
In some embodiments, the predicting unit 520 determines the predicted data amount of the task in the current time period according to the calculation result, including: judging whether the difference value between the calculation result and the actual data volume of the past time period closest to the current time period is larger than a preset threshold value or not; if not, determining the calculation result as the predicted data volume of the task in the current time period; and if so, correcting the calculation result, and determining the corrected calculation result as the predicted data volume of the task in the current time period.
In some embodiments, the prediction unit 520 modifies the calculation results, including: calculating an average relative error between the actual data volume and the predicted data volume of a preset second number of past time segments closest to the current time segment; and correcting the calculation result by using the average relative error.
In some embodiments, the real-time data obtaining unit 530 is configured to obtain the actual data amount of the task in the current time period in real time through the application main control component or through the web interface.
In some embodiments, the warning unit 540 warns whether the task is abnormal, including: if the difference value between the predicted data volume and the actual data volume of the task in the current time period exceeds a corresponding early warning threshold value of the task set by an early warning rule, sending out an early warning that the task is abnormal; wherein the pre-warning rules are configurable. The early warning rules include: the types of the tasks needing early warning and the early warning threshold value corresponding to each type of the tasks needing early warning. The types of tasks to be pre-warned include: one or more of a map task, a map read task, a map write task, a reduce read task, a reduce write task.
Fig. 6 is a schematic diagram of an apparatus for warning task abnormalities according to further embodiments of the present disclosure.
As shown in fig. 6, the apparatus 600 for warning task abnormality according to this embodiment includes: a memory 610 and a processor 620 coupled to the memory 610, the processor 620 being configured to perform the method of pre-warning for task exceptions in any of the foregoing embodiments based on instructions stored in the memory 610.
For example, the actual data volume of the task in the past time period is acquired; determining the predicted data volume of the task in the current time period according to the actual data volume of the task in the past time period; acquiring the actual data volume of the task in the current time period; and according to the difference value between the predicted data volume and the actual data volume of the task in the current time period, early warning whether the task is abnormal or not.
For another example, the weighted average calculation is performed on the actual data volume of the past time period of the preset first number closest to the current time period, wherein the closer the past time period is to the current time period, the greater the weight corresponding to the actual data volume of the past time period is; and determining the predicted data volume of the task in the current time period according to the calculation result.
For another example, it is determined whether a difference between the calculation result and an actual data amount of a past time period closest to the current time period is greater than a preset threshold value; if not, determining the calculation result as the predicted data volume of the task in the current time period; and if so, correcting the calculation result, and determining the corrected calculation result as the predicted data volume of the task in the current time period. Calculating the average relative error between the actual data volume and the predicted data volume of a preset second number of past time periods closest to the current time period; and correcting the calculation result by using the average relative error.
The apparatus 600 may also include an input-output interface 630, a network interface 640, a storage interface 650, and the like. These interfaces 630, 640, 650 and the connections between the memory 610 and the processor 620 may be, for example, via a bus 660. The input/output interface 630 provides a connection interface for input/output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 640 provides a connection interface for various networking devices. The storage interface 650 provides a connection interface for external storage devices such as an SD card and a usb disk.
The disclosed embodiments also provide a non-transitory computer-readable storage medium on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the method for early warning of task exceptions in any of the embodiments.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more non-transitory computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer program code embodied therein.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only exemplary of the present disclosure and is not intended to limit the present disclosure, so that any modification, equivalent replacement, or improvement made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.
Claims (11)
1. A method for early warning task abnormity is characterized by comprising the following steps
Acquiring the actual data volume of the task in the past time period;
determining the predicted data volume of the task in the current time period according to the actual data volume of the task in the past time period;
acquiring the actual data volume of the task in the current time period;
and according to the difference value between the predicted data volume and the actual data volume of the task in the current time period, early warning is carried out on whether the task is abnormal.
2. The method of claim 1,
the past time period includes: a preset first number of past time periods closest to the current time period.
3. The method of claim 2, wherein determining a predicted amount of data for the task at a current time period comprises:
performing weighted average calculation on the actual data volume of a preset first number of past time periods closest to the current time period, wherein the closer the past time period is to the current time period, the greater the corresponding weight of the actual data volume of the past time period is;
and determining the predicted data volume of the task in the current time period according to the calculation result.
4. The method of claim 3, wherein determining the predicted amount of data for the task at the current time period based on the calculation comprises:
judging whether the difference value between the calculation result and the actual data volume of the past time period closest to the current time period is larger than a preset threshold value or not;
if not, determining the calculation result as the predicted data volume of the task in the current time period;
and if so, correcting the calculation result, and determining the corrected calculation result as the predicted data volume of the task in the current time period.
5. The method of claim 4, wherein modifying the computed result comprises:
calculating an average relative error between the actual data volume and the predicted data volume of a preset second number of past time segments nearest to the current time segment;
and correcting the calculation result by using the average relative error.
6. The method of any one of claims 1-5, wherein pre-warning whether the task is abnormal comprises:
if the difference value between the predicted data volume and the actual data volume of the task in the current time period exceeds a corresponding early warning threshold value of the task set by an early warning rule, sending out an early warning that the task is abnormal;
wherein the pre-warning rules are configurable.
7. The method of claim 6,
the early warning rule comprises the following steps: the type of the task needing early warning and the early warning threshold value corresponding to each type of the task needing early warning;
the types of tasks needing early warning include: one or more of a map task, a map read task, a map write task, a reduce read task, a reduce write task.
8. The method of any one of claims 1-5, wherein obtaining the actual amount of data for the task at the current time period comprises:
and acquiring the actual data volume of the task in the current time period in real time through an application program main control assembly or a webpage interface.
9. An apparatus for early warning of task anomalies, comprising:
a memory; and
a processor coupled to the memory, the processor configured to perform the method of pre-warning of task exceptions of any of claims 1-8 based on instructions stored in the memory.
10. A device for early warning task abnormity comprises
A history data acquisition unit configured to acquire an actual data amount of the task in a past time period;
a prediction unit configured to determine a predicted data amount of the task in a current time period according to an actual data amount of the task in a past time period;
a real-time data acquisition unit configured to acquire an actual data volume of the task in a current time period;
and the early warning unit is configured to early warn whether the task is abnormal or not according to the difference value between the predicted data volume and the actual data volume of the task in the current time period.
11. A non-transitory computer readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the method of pre-warning of task exceptions of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010579102.9A CN113778776A (en) | 2020-06-23 | 2020-06-23 | Method and device for early warning task abnormity and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010579102.9A CN113778776A (en) | 2020-06-23 | 2020-06-23 | Method and device for early warning task abnormity and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113778776A true CN113778776A (en) | 2021-12-10 |
Family
ID=78835206
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010579102.9A Pending CN113778776A (en) | 2020-06-23 | 2020-06-23 | Method and device for early warning task abnormity and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113778776A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115168474A (en) * | 2022-07-26 | 2022-10-11 | 成都智元汇信息技术股份有限公司 | Internet of things center station system building method based on big data model |
| CN115292065A (en) * | 2022-07-29 | 2022-11-04 | 成都智元汇信息技术股份有限公司 | Event confirmation method, system and device based on stream architecture |
| WO2024072579A1 (en) * | 2022-09-27 | 2024-04-04 | Microsoft Technology Licensing, Llc | System and method for ml-aided anomaly detection and end-to-end comparative analysis of the execution of spark jobs within a cluster |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108965826A (en) * | 2018-08-21 | 2018-12-07 | 北京旷视科技有限公司 | Monitoring method, device, processing equipment and storage medium |
| CN111090939A (en) * | 2019-12-17 | 2020-05-01 | 上海汉中诺软件科技有限公司 | Early warning method and system for abnormal working condition of petrochemical device |
-
2020
- 2020-06-23 CN CN202010579102.9A patent/CN113778776A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108965826A (en) * | 2018-08-21 | 2018-12-07 | 北京旷视科技有限公司 | Monitoring method, device, processing equipment and storage medium |
| CN111090939A (en) * | 2019-12-17 | 2020-05-01 | 上海汉中诺软件科技有限公司 | Early warning method and system for abnormal working condition of petrochemical device |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115168474A (en) * | 2022-07-26 | 2022-10-11 | 成都智元汇信息技术股份有限公司 | Internet of things center station system building method based on big data model |
| CN115168474B (en) * | 2022-07-26 | 2023-04-14 | 成都智元汇信息技术股份有限公司 | Internet of things central station system building method based on big data model |
| CN115292065A (en) * | 2022-07-29 | 2022-11-04 | 成都智元汇信息技术股份有限公司 | Event confirmation method, system and device based on stream architecture |
| WO2024072579A1 (en) * | 2022-09-27 | 2024-04-04 | Microsoft Technology Licensing, Llc | System and method for ml-aided anomaly detection and end-to-end comparative analysis of the execution of spark jobs within a cluster |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020259421A1 (en) | Method and apparatus for monitoring service system | |
| CN109634801B (en) | Data trend analysis method, system, computer device and readable storage medium | |
| CN110309009B (en) | Situation-based operation and maintenance fault root cause positioning method, device, equipment and medium | |
| CN113778776A (en) | Method and device for early warning task abnormity and storage medium | |
| US10579453B2 (en) | Stream-processing data | |
| JP2018523195A (en) | Data quality analysis | |
| CN111045894B (en) | Database abnormality detection method, database abnormality detection device, computer device and storage medium | |
| US9286735B1 (en) | Generating cumulative wear-based indicators for vehicular components | |
| CN109684162B (en) | Equipment state prediction method, system, terminal and computer readable storage medium | |
| US11120051B2 (en) | Dimension optimization in singular value decomposition-based topic models | |
| US20210097431A1 (en) | Debugging and profiling of machine learning model training | |
| US20200234581A1 (en) | Vehicle traffic information analysis and traffic jam management | |
| US20210124661A1 (en) | Diagnosing and remediating errors using visual error signatures | |
| WO2017214613A1 (en) | Streaming data decision-making using distributions with noise reduction | |
| Alevizos et al. | Complex event recognition under uncertainty: A short survey | |
| US11468365B2 (en) | GPU code injection to summarize machine learning training data | |
| CN110889597A (en) | Method and device for detecting abnormal business timing sequence indexes | |
| WO2021067385A1 (en) | Debugging and profiling of machine learning model training | |
| CN115033412A (en) | Task log merging method and device | |
| CN114706893A (en) | Fault detection method, device, equipment and storage medium | |
| CN113220551A (en) | Index trend prediction and early warning method and device, electronic equipment and storage medium | |
| US11138512B2 (en) | Management of building energy systems through quantification of reliability | |
| CN112860779A (en) | Batch data importing method and device | |
| CN111813631A (en) | Resource situation visualization and analysis method for cloud data center | |
| US9459939B2 (en) | In-memory approach to extend semantic event processing with domain insights |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |