CN117609918A - Abnormal task identification method and related device - Google Patents
Abnormal task identification method and related device Download PDFInfo
- Publication number
- CN117609918A CN117609918A CN202311577033.8A CN202311577033A CN117609918A CN 117609918 A CN117609918 A CN 117609918A CN 202311577033 A CN202311577033 A CN 202311577033A CN 117609918 A CN117609918 A CN 117609918A
- Authority
- CN
- China
- Prior art keywords
- abnormal
- task
- training
- data
- operation data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000002159 abnormal effect Effects 0.000 title claims abstract description 324
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000012549 training Methods 0.000 claims abstract description 145
- 238000012545 processing Methods 0.000 claims abstract description 117
- 238000000605 extraction Methods 0.000 claims abstract description 21
- 238000002372 labelling Methods 0.000 claims description 32
- 230000008569 process Effects 0.000 claims description 11
- 230000006870 function Effects 0.000 claims description 10
- 238000013507 mapping Methods 0.000 claims description 7
- 230000009467 reduction Effects 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 4
- 230000005856 abnormality Effects 0.000 description 15
- 238000012360 testing method Methods 0.000 description 11
- 238000004364 calculation method Methods 0.000 description 9
- 238000004422 calculation algorithm Methods 0.000 description 4
- 239000006185 dispersion Substances 0.000 description 4
- 238000009825 accumulation Methods 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 238000007635 classification algorithm Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses an abnormal task identification method and a related device, wherein the method comprises the steps of performing feature extraction processing on operation data generated by executing a task to be identified through an abnormal task identification model to obtain key features corresponding to the operation data; the abnormal task identification model is obtained by training marking data obtained by marking training operation data by using marking rules, and the marking rules are obtained by calculating the abnormal types of tasks; determining a recognition result corresponding to the task to be recognized according to the key features through the abnormal task recognition model; when the identification result indicates that the task to be identified is an abnormal task, the identification result comprises the abnormal type of the abnormal task. Because the abnormal task identification model is obtained through training operation data and calculating the abnormal type of the task, the abnormal task identification model can rapidly identify whether the task to be identified is abnormal or not through the operation data, determine the abnormal type of the abnormal task and improve the identification efficiency and the identification precision of the abnormal task.
Description
Technical Field
The present disclosure relates to the field of big data processing technologies, and in particular, to a method and an apparatus for identifying an abnormal task.
Background
The large data platform can be used as an infrastructure for enterprise digital transformation, and at present, the large data platform realizes the functions of calculating, storing, scheduling and the like aiming at data by arranging a plurality of calculation tasks.
However, since the developer has limitation on knowledge of the technical bottom layer, the computing task set by the developer may have abnormality, resulting in reduced reliability of the big data platform.
In order to solve the technical problem, in the related art, developers generally accumulate and determine whether the computing task is abnormal through professional knowledge and experience, but the number of the computing tasks is large, so that the recognition efficiency of manually recognizing the abnormal task is low and the recognition precision is low.
Disclosure of Invention
Based on the problems, the application provides an abnormal task identification method and a related device, which solve the problems of low efficiency and low identification precision of manually identifying the abnormal task.
The embodiment of the application discloses the following technical scheme:
in a first aspect, an embodiment of the present application provides a method for identifying an abnormal task, where the method includes:
acquiring operation data of a task to be identified; the operation data are data generated by executing the task to be identified;
performing feature extraction processing on the operation data through an abnormal task identification model to obtain key features corresponding to the operation data; the abnormal task identification model is obtained based on training marking data; the training marking data is obtained by marking training operation data by using marking rules; the labeling rule is obtained based on the abnormal type of the computing task;
determining an identification result corresponding to the task to be identified according to the key characteristics through the abnormal task identification model; when the identification result indicates that the task to be identified is an abnormal task, the identification result comprises an abnormal type of the abnormal task.
Optionally, the labeling rule is obtained by:
acquiring a plurality of execution stages of the computing task;
determining at least one exception type corresponding to each of the plurality of execution phases;
determining the abnormal conditions corresponding to the abnormal types to obtain an abnormal condition set;
and constructing the labeling rule according to the abnormal type and the abnormal condition set corresponding to each of the plurality of stages.
Optionally, the plurality of execution phases includes a task level phase, a mapping phase, a reduction phase, and a processing phase;
the determining at least one exception type corresponding to each of the plurality of execution phases includes:
at least one exception type is determined for each of the task level stage, the map stage, the reduce stage, and the process stage.
Optionally, the abnormal task identification model is obtained through training in the following way:
acquiring training operation data; the training operation data is operation data generated by executing abnormal tasks;
carrying out standardization processing on the training operation data to obtain training standard data;
marking the training standard data by using the marking rule to obtain training marking data; the training annotation data comprises annotation anomaly types; the marked abnormal type characterizes the abnormal type of the abnormal task;
determining the training abnormal type of the abnormal task corresponding to the training marking data according to the training marking data through the abnormal task identification model to be trained;
constructing a loss function according to the difference between the labeling anomaly type and the training anomaly type; and training the abnormal task identification model based on the loss function.
Optionally, after acquiring the operation data of the task to be identified, the method further includes:
performing standardized processing on the operation data to obtain standard data; wherein the normalization process indicates to convert the operation data into data of a preset format;
the feature extraction processing is performed on the operation data through the abnormal task identification model to obtain key features corresponding to the operation data, including:
and carrying out feature extraction processing on the standard data through an abnormal task identification model to obtain key features corresponding to the operation data.
Optionally, when the identification result indicates that the task to be identified is an abnormal task, the method further includes:
matching the abnormal type corresponding to the abnormal task with a preset abnormal task processing table to obtain a matching result; the matching result comprises a target processing mode of the abnormal task; the preset abnormal task processing table comprises processing modes corresponding to a plurality of abnormal tasks respectively; the preset abnormal task processing table comprises the target processing mode.
Optionally, after the abnormal task returns to normal, the method further includes:
acquiring an actual processing mode for processing the abnormal task;
comparing the actual processing mode with the target processing mode;
and when the comparison result indicates that the actual processing mode is different from the target processing mode, updating the target processing mode in the preset abnormal task processing table by using the actual processing mode.
In a second aspect, an embodiment of the present application provides an apparatus for identifying an abnormal task, where the apparatus includes:
the acquisition module is used for acquiring the operation data of the task to be identified; the operation data are data generated by executing the task to be identified;
the extraction module is used for carrying out feature extraction processing on the operation data through the abnormal task identification model to obtain key features corresponding to the operation data; the abnormal task identification model is obtained based on training marking data; the training marking data is obtained by marking training operation data by using marking rules; the labeling rules are obtained based on a plurality of stages of executing tasks;
the identification module is used for determining an identification result corresponding to the task to be identified according to the key characteristics through the abnormal task identification model; when the identification result indicates that the task to be identified is an abnormal task, the identification result comprises an abnormal type of the abnormal task.
In a third aspect, embodiments of the present application provide a computer device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the method of identifying abnormal tasks according to any one of the first aspects when the computer program is executed.
In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, where an instruction is stored, where the instruction when executed on a terminal device causes the terminal device to execute the method for identifying an abnormal task according to any one of the first aspect.
Compared with the prior art, the application has the following beneficial effects:
according to the abnormal task identification method, feature extraction processing is carried out on operation data generated by executing a task to be identified through an abnormal task identification model, so that key features corresponding to the operation data are obtained; the abnormal task identification model is obtained by training marking data obtained by marking training operation data by using marking rules, wherein the marking rules are obtained by calculating the abnormal types of tasks; determining a recognition result corresponding to the task to be recognized according to the key features through the abnormal task recognition model; when the identification result indicates that the task to be identified is an abnormal task, the identification result comprises the abnormal type of the abnormal task. The abnormal task identification model is obtained through training operation data and the abnormal type of the calculation task, so that the abnormal task identification model can rapidly identify whether the task to be identified is abnormal or not through the operation data of the task to be identified, and when the task to be identified is abnormal, the abnormal type of the abnormal task is determined, and the identification efficiency and the identification accuracy of the abnormal task are improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive faculty for a person skilled in the art.
FIG. 1 is a flowchart of a method for identifying abnormal tasks according to an embodiment of the present application;
FIG. 2 is a flowchart of a training method of an abnormal task recognition model according to an embodiment of the present application;
FIG. 3 is a flowchart of another method for identifying abnormal tasks according to an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of an abnormal task recognition device according to an embodiment of the present application.
Detailed Description
As described above, in research on computing tasks for a large data platform, it is found that, currently, the large data platform performs functions of computing, storing, scheduling, and the like for data by setting a plurality of computing tasks.
However, since the developer has limitation on knowledge of the technical bottom layer, the computing task set by the developer may have abnormality, resulting in reduced reliability of the big data platform. As an example, assume that the task demand a is a form of outputting input data in a table, the computing task corresponding to the task demand a is B, and the table output when the computing task B is executed is empty, that is, the computing task B has an abnormality.
In order to solve the technical problem, in the related art, developers generally accumulate and determine whether the computing task is abnormal through professional knowledge and experience, but the number of the computing tasks is large, so that the recognition efficiency of manually recognizing the abnormal task is low and the recognition precision is low.
As an example, assuming that the output table when executing the computing task B is empty, the cause of abnormality of the computing task B may be a problem with the input data, a problem with the code of the computing task, a problem with the filtering condition of the input data, or the like, but these causes of abnormality need to be determined by the developer through expertise and experience accumulation, resulting in difficulty in quickly determining the cause of abnormality of the computing task, that is, difficulty in quickly determining the type of abnormality of the computing task; meanwhile, because the number of calculation tasks is large, the time for identifying the abnormal tasks manually is long, the identification efficiency is reduced, and because the manual identification depends on the accumulation of expertise and experience, the identification error of the abnormal tasks can be caused, so that the identification precision of the abnormal tasks is reduced.
In order to solve the above problems, an embodiment of the present application provides a method and a related device for identifying an abnormal task. The method comprises the following steps: performing feature extraction processing on operation data generated by executing a task to be identified through an abnormal task identification model to obtain key features corresponding to the operation data; the abnormal task identification model is obtained by training marking data obtained by marking training operation data by using marking rules, wherein the marking rules are obtained by calculating the abnormal types of tasks; determining a recognition result corresponding to the task to be recognized according to the key features through the abnormal task recognition model; when the identification result indicates that the task to be identified is an abnormal task, the identification result comprises the abnormal type of the abnormal task.
Therefore, the abnormal task identification model is obtained through training operation data and the abnormal type of the calculation task, so that whether the task to be identified is abnormal or not can be rapidly identified through the operation data of the task to be identified, the abnormal type of the abnormal task is determined when the task to be identified is abnormal, and the identification efficiency and the identification accuracy of the abnormal task are improved.
In order to make the present application solution better understood by those skilled in the art, the following description will clearly and completely describe the technical solution in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
In the application, the relevant data collection and processing should be strictly according to the requirements of relevant national laws and regulations when the examples are applied, the informed consent or independent consent of the main body of the personal information is obtained, and the subsequent data use and processing behaviors are developed within the authorized range of the laws and regulations and the main body of the personal information.
Referring to fig. 1, the figure is a flowchart of a method for identifying an abnormal task according to an embodiment of the present application.
Referring to fig. 1, the method for identifying an abnormal task provided in the embodiment of the present application may include:
s101: and acquiring the operation data of the task to be identified.
The running data is data generated by executing the task to be identified, such as log data, metadata and the like for executing the task to be identified.
Different log data exists in different execution stages of the computing task, such as log data of task stage including but not limited to task execution state, execution start time, execution end time, execution input and output data amount, execution resource consumption, etc.; the log data for the mapping stage, the reduction stage, and the processing stage includes, but is not limited to, the start-end time of each stage, the amount of input and output data, the number of concurrent instances, the start-end time of each instance, the amount of input and output data for each instance, and the like.
Metadata refers to data related to data or information called data or information used to describe the data, and may be understood as the smallest unit of data. Metadata may be data that states its elements or attributes (name, size, data type, etc.), or its structure (length, field, data column), or its associated data (location, contact, use object).
It should be understood that, since the operation data can reflect the execution process of the task to be identified and reflect the input data, the intermediate data, the output data, and the like in the execution process, the operation data can be used as the input of whether the task to be identified has an abnormality.
S102: and carrying out feature extraction processing on the operation data through an abnormal task identification model to obtain key features corresponding to the operation data.
The abnormal task identification model is a model for identifying whether an abnormal computing task exists or not, and is obtained by training based on training marking data; the training marking data is obtained by marking training operation data by using marking rules; the labeling rules are derived based on the type of anomaly of the computing task.
The training operation data are data generated by executing training abnormal tasks, and the training abnormal tasks are obtained through a training data set. The labeling rule means a rule for labeling abnormal types of training operation data, and the labeling rule comprises abnormal types corresponding to a plurality of abnormal tasks. The training annotation data means training operation data annotated with an anomaly type.
Since the operation data includes a plurality of data, it is necessary to extract key features in the operation data through the abnormal task identification model, so as to determine whether the task to be identified is abnormal through the key features.
Based on the method for identifying abnormal tasks provided in the foregoing embodiment, after obtaining the operation data of the task to be identified, the method may further include: performing standardized processing on the operation data to obtain standard data; wherein the normalization process indicates to convert the operation data into data of a preset format. It should be understood that, in this embodiment, since the data structure of the operation data is complex, the operation data may be converted into data in a preset format through standardized processing, so as to improve the efficiency of abnormal task identification, and correspondingly, the feature extraction processing may be performed on the standard data through an abnormal task identification model, so as to obtain the key feature corresponding to the operation data.
S103: and determining a recognition result corresponding to the task to be recognized according to the key features through the abnormal task recognition model.
When the identification result indicates that the task to be identified is an abnormal task, the identification result comprises an abnormal type of the abnormal task. That is, in the embodiment of the present application, the abnormal task identification model is trained by the training marking data for marking the abnormal task, so that the abnormal task to be identified can be identified.
According to the abnormal task identification method, feature extraction processing is carried out on operation data generated by executing a task to be identified through an abnormal task identification model, so that key features corresponding to the operation data are obtained; the abnormal task identification model is obtained by training marking data obtained by marking training operation data by using marking rules, wherein the marking rules are obtained by calculating the abnormal types of tasks; determining a recognition result corresponding to the task to be recognized according to the key features through the abnormal task recognition model; when the identification result indicates that the task to be identified is an abnormal task, the identification result comprises the abnormal type of the abnormal task. The abnormal task identification model is obtained through training operation data and the abnormal type of the calculation task, so that the abnormal task identification model can rapidly identify whether the task to be identified is abnormal or not through the operation data of the task to be identified, and when the task to be identified is abnormal, the abnormal type of the abnormal task is determined, and the identification efficiency and the identification accuracy of the abnormal task are improved.
Based on the method for identifying abnormal tasks provided by the foregoing embodiment, in some possible implementation manners, the labeling rule for labeling training operation data may be obtained by:
step A1: a plurality of execution phases of the computing task are acquired.
A computing task means a task for processing data, and the computing task may be divided into a plurality of execution stages, for example, divided in whole, and then the computing task may be a task-level stage; the computing tasks may be divided into a Map (Map) stage, a Reduce (Reduce) stage, and a process (Join) stage, according to the order in which the computing tasks are executed. Wherein, the task to be identified means a computing task to be identified.
Step A2: at least one exception type corresponding to each of the plurality of execution phases is determined.
After determining different execution stages of the computing task, the types of anomalies possibly occurring in each execution stage can be further determined according to the different execution stages, for example, the types of anomalies existing in the mapping stage can comprise excessive small files and oversized file input, and the types of anomalies existing in the reduction stage can comprise data tilting and computing long tails.
It should be noted that, the exception types of each execution stage may be expanded according to the type of the computing task and the exception type of the exception task, and the new exception type is added to the labeling rule.
In some possible implementations, when the plurality of execution phases includes a task level phase, a mapping phase, a reduction phase, and a processing phase; step A2 may determine at least one exception type corresponding to each of the task level stage, the mapping stage, the reduction stage, and the processing stage.
Step A3: and determining the abnormal conditions corresponding to the abnormal types to obtain an abnormal condition set.
After determining the exception type of each execution stage, determining the condition for determining that the computing task is an exception task by determining the exception condition for generating the exception type. As one example, assuming that the abnormality type is too many small files, the abnormality condition corresponding to the abnormality type is that more than 1 ten thousand files less than 10 megafiles are input to the computing task, that is, when more than 1 ten thousand files less than 10 megafiles are input to the computing task, it may be determined that the abnormality type of the abnormality task is too many small files.
Step A4: and constructing the labeling rule according to the abnormal type and the abnormal condition set corresponding to each of the plurality of stages.
It should be understood that after determining the set of abnormal conditions, labeling rules are constructed according to each execution stage of the computing task, the type of abnormality corresponding to each execution stage, and the set of abnormal conditions, see table 1 below:
table 1: marking rule table
In the embodiment of the application, the abnormal types possibly existing in each execution stage of the calculation task are divided according to each execution stage of the calculation task, the abnormal conditions corresponding to each abnormal type are determined according to the abnormal types of each stage, so that an abnormal condition set is formed, and the marking rule is constructed based on the abnormal condition set, so that the abnormal types of the calculation task can be combed, the follow-up batch marking of training operation data is facilitated, the time of marking by manually relying on professional knowledge and accumulation is saved, the marking efficiency is improved, and the cost of data marking is reduced.
Based on the labeling rules provided in the foregoing embodiments, reference is made to fig. 2, which is a flowchart of a training method for an abnormal task recognition model provided in an embodiment of the present application.
Referring to fig. 2, the training method for an abnormal task identification model provided in the embodiment of the present application may include:
s201: training operation data is acquired.
The training operation data are operation data generated by executing abnormal tasks.
The training operation data may be obtained by analyzing log data and metadata of the abnormal tasks, for example, log data of all abnormal tasks in 1 cluster within 1 year is obtained as the training operation data.
S202: and carrying out standardization processing on the training operation data to obtain training standard data.
It should be appreciated that since the training operation data includes a plurality of data, the format of the different data is inconsistent, and in order to facilitate training and recognition, the training operation data needs to be standardized to obtain training standard data.
S203: and marking the training standard data by using the marking rule to obtain training marking data.
The training annotation data comprises an annotation anomaly type, and the annotation anomaly type characterizes the anomaly type of the anomaly task. In one possible implementation manner, the labeling rules constructed by the foregoing embodiments may be used for batch labeling.
S204: and determining the training abnormal type of the abnormal task corresponding to the training marking data according to the training marking data through the abnormal task identification model to be trained.
In this embodiment of the present application, training annotation data may be divided into a training set and a testing set according to a preset proportion, where the training set is used for training an abnormal task recognition model to be trained, and the testing set is used for testing the abnormal task recognition model.
In some possible implementation manners, the abnormal task recognition model may be trained in a semi-supervised learning manner, that is, training standard data which is not marked by a marking rule may be added, and the unmarked training standard data and the training marking data are used together as input of the abnormal task recognition model.
In other possible implementations, the abnormal task recognition model may be trained by adopting an alternating mode of supervised learning and unsupervised learning, that is, training label data is firstly used as input of the abnormal task recognition model, training is performed on the abnormal task recognition model, and unlabeled training standard data is used as input of the abnormal task recognition model after iteration is performed for a preset number of times.
Training anomaly type means an anomaly type identified by an anomaly task identification model to be trained.
As an example, the abnormal task recognition model to be trained may perform feature extraction on training annotation data to obtain training key features, where the training key features include, but are not limited to, task execution duration features, data input and output features, concurrency quantity features, discrete degree features of concurrent instance execution duration, and discrete degree features of concurrent instance input and output.
In this embodiment of the present application, the feature extraction manner may be binning, feature hashing, nesting method, feature scaling, standardization, feature interaction, and the like, which is not limited herein specifically.
S205: constructing a loss function according to the difference between the labeling anomaly type and the training anomaly type; and training the abnormal task identification model based on the loss function.
In this embodiment of the present application, the training annotation data may be trained according to requirements using a suitable machine learning algorithm to obtain a trained abnormal task annotation model, where the machine learning algorithm includes, but is not limited to, algorithms such as a vector machine (SVM), bayesian classification, decision tree, and the like.
Taking a naive bayes classification algorithm as an example, training key features of training annotation data are: the task execution duration feature, the data input and output feature, the concurrency quantity feature, the discrete degree feature of the concurrency instance execution duration and the discrete degree feature of the concurrency instance input and output are defined as follows:
X＝df[['ffeature_exetime',
'ffeature_inoutput',
'feature_concurrency',
'feature_exetime_dispersion',
'feature_inoutput_dispersion']]
wherein feature_exetime represents a task execution duration feature, feature_input represents a task input feature, feature_concurrency represents a concurrency feature, feature_exetime_dispersion represents a discrete degree feature of a concurrency instance execution duration, feature_input_dispersion represents a discrete degree feature of a concurrency instance input and output, and a target variable is defined as y=df [ 'class' ].
Wherein, training set and test set are defined as respectively: x_train, x_test, y_train, y_test=train_test_split (X, y, test_size=0.2, random_state=42).
Model training gaussian nb (). Fit (x_train, y_train) is performed by using a naive bayes classifier in the Scikit-learn library, and after the abnormal task recognition model training is completed, model performance is evaluated by calculating the accuracy of the abnormal task recognition model on the test set, that is, the accuracy of the abnormal task recognition model on the test set is calculated by y_pred=gnb.prediction (x_test), and accuracy=accuracy_score (y_test, y_pred) to determine the performance of the abnormal task recognition model.
After the abnormal task recognition model is trained based on the mode, in some possible implementation modes, the trained abnormal task recognition model can be deployed to a large data platform online environment, when a task to be recognized starts to be executed, operation data generated by the execution of the task to be recognized is read online and used as an abnormal task recognition model to be input, whether the task is abnormal or not is diagnosed in real time, and recognition results are generated.
Based on the method for identifying an abnormal task provided in the foregoing embodiment, when the identification result indicates that the task to be identified is an abnormal task, the method may further include: matching the abnormal type corresponding to the abnormal task with a preset abnormal task processing table to obtain a matching result; the matching result comprises a target processing mode of the abnormal task; the preset abnormal task processing table comprises processing modes corresponding to a plurality of abnormal tasks respectively; the preset abnormal task processing table comprises the target processing mode.
It should be understood that after determining that the task to be identified is an abnormal task, the abnormal type of the abnormal task can be determined through an abnormal task identification model, and in order to facilitate processing of the abnormal task of the abnormal type, a processing mode can be obtained through a preset abnormal task processing table, so that the processing efficiency of a developer on the abnormal task is improved.
The preset abnormal task processing table may include processing manners of abnormal types corresponding to each stage of the computing task, as shown in the following table 2:
table 2: presetting an abnormal task processing table
As a possible implementation manner, after the abnormal task returns to normal, the method may further include:
b1: and acquiring an actual processing mode for processing the abnormal task.
The actual handling means the actual solution of the abnormal task by the developer.
B2: and comparing the actual processing mode with the target processing mode.
The target processing mode means a processing mode matched with the abnormal type of the abnormal task in a preset abnormal task processing table.
B3: and when the comparison result indicates that the actual processing mode is different from the target processing mode, updating the target processing mode in the preset abnormal task processing table by using the actual processing mode.
It should be understood that, in this embodiment, when the actual processing manner is different from the target processing manner, the actual processing manner is added to the preset abnormal task processing table, so that the processing schemes of the abnormal task can be enriched, and the processing efficiency of the abnormal task can be improved.
Based on the method for identifying an abnormal task provided in the foregoing embodiment, referring to fig. 3, which is a flowchart of another method for identifying an abnormal task provided in an embodiment of the present application, with reference to fig. 3, the method for identifying an abnormal task provided in an embodiment of the present application may include:
training annotation data preparation:
s301: and acquiring training operation data generated by executing the training calculation task.
S302: and carrying out standardization processing on the training operation data to obtain training standard data.
S303: and constructing labeling rules.
S304: and labeling the training standard data by using labeling rules to obtain training labeling data.
It should be noted that, the ways of acquiring the training annotation data in steps S301, S302, and S304 are the same as those of acquiring the training annotation data in steps S201 to S203 in the above embodiment, and the ways of constructing the annotation rule in step S303 are the same as those of constructing the annotation rule in steps A1 to A4 in the above embodiment, and the relevant explanation content will be referred to the above embodiments and will not be repeated here.
Training the abnormal task identification model:
s305: the training annotation data is divided into a training set and a testing set.
S306: and extracting features of training annotation data of the training set through the abnormal task identification model to obtain key features.
S307: and training the abnormal task identification model by using a machine learning algorithm.
It should be understood that steps S305 to S307 are the same as the training process of the abnormal task recognition model provided in the above embodiment, and the explanation thereof will be referred to the above embodiment and will not be repeated here.
Abnormal task identification model application stage:
s308: and deploying the trained abnormal task identification model to an online environment, and determining the abnormal type of the task to be identified.
S309: and matching the abnormal type with a preset abnormal task processing table, and determining a target processing mode of the abnormal task.
S310: pushing the identification result and the target processing mode to relevant personnel for confirmation.
In order to ensure the accuracy of the recognition result and the target processing mode, the recognition result and the target processing mode can be further sent to related personnel for confirmation.
Abnormal task identification model optimization stage:
s311: and taking the abnormal task confirmed by the related personnel as a training abnormal task, and carrying out iterative training on the abnormal task identification model by utilizing the training abnormal task.
The abnormal task identified by the abnormal task identification model can be used as a training abnormal task, the operation data generated by executing the training abnormal task is used as training operation data, the training operation data is used for further iterative optimization of the abnormal task identification model, and the identification precision of the abnormal task identification model is improved.
Referring to fig. 4, which is a schematic structural diagram of an abnormal task identifying device provided in an embodiment of the present application, and in conjunction with fig. 4, an abnormal task identifying device 400 provided in an embodiment of the present application may include:
an acquisition module 401, configured to acquire operation data of a task to be identified; the operation data are data generated by executing the task to be identified;
the extracting module 402 is configured to perform feature extraction processing on the operation data through an abnormal task identification model, so as to obtain key features corresponding to the operation data; the abnormal task identification model is obtained based on training marking data; the training marking data is obtained by marking training operation data by using marking rules; the labeling rules are obtained based on a plurality of stages of executing tasks;
the recognition module 403 is configured to determine, according to the key feature, a recognition result corresponding to the task to be recognized through the abnormal task recognition model; when the identification result indicates that the task to be identified is an abnormal task, the identification result comprises an abnormal type of the abnormal task.
As an example, the labeling rules are obtained by:
the first acquisition module is used for acquiring a plurality of execution phases of the computing task;
the exception type determining module is used for determining at least one exception type corresponding to each of the plurality of execution stages;
the abnormal condition set determining module is used for determining abnormal conditions corresponding to the abnormal types to obtain an abnormal condition set;
and the labeling rule construction module is used for constructing the labeling rule according to the abnormal types and the abnormal condition sets corresponding to the stages.
As one example, the plurality of execution phases includes a task level phase, a mapping phase, a reduction phase, and a processing phase;
the abnormality type determining module is specifically configured to:
at least one exception type is determined for each of the task level stage, the map stage, the reduce stage, and the process stage.
As an example, the abnormal task identification model is trained by:
the second acquisition module is used for acquiring training operation data; the training operation data is operation data generated by executing abnormal tasks;
the training standardization module is used for carrying out standardization processing on the training operation data to obtain training standard data;
the marking module is used for marking the training standard data by utilizing the marking rule to obtain training marking data; the training annotation data comprises annotation anomaly types; the marked abnormal type characterizes the abnormal type of the abnormal task;
the training abnormal type determining module is used for determining the training abnormal type of the abnormal task corresponding to the training marking data according to the training marking data through the abnormal task identification model to be trained;
the training module is used for constructing a loss function according to the difference between the labeling anomaly type and the training anomaly type; and training the abnormal task identification model based on the loss function.
As an example, after acquiring the operation data of the task to be identified, the apparatus further includes:
the standardized module is used for carrying out standardized processing on the operation data to obtain standard data; wherein the normalization process indicates to convert the operation data into data of a preset format;
the extraction module is specifically configured to:
and carrying out feature extraction processing on the standard data through an abnormal task identification model to obtain key features corresponding to the operation data.
As an example, when the identification result indicates that the task to be identified is an abnormal task, the apparatus further includes:
the matching module is used for matching the abnormal type corresponding to the abnormal task with a preset abnormal task processing table to obtain a matching result; the matching result comprises a target processing mode of the abnormal task; the preset abnormal task processing table comprises processing modes corresponding to a plurality of abnormal tasks respectively; the preset abnormal task processing table comprises the target processing mode.
As an example, after the abnormal task is restored to normal, the apparatus further includes:
the third acquisition module is used for acquiring an actual processing mode for processing the abnormal task;
the comparison module is used for comparing the actual processing mode with the target processing mode;
and the updating module is used for updating the target processing mode in the preset abnormal task processing table by using the actual processing mode when the comparison result indicates that the actual processing mode is different from the target processing mode.
The device for identifying the abnormal task provided in the embodiment of the present application has the same beneficial effects as the method for identifying the abnormal task provided in the above embodiment, and therefore will not be described in detail.
The embodiment of the application also provides corresponding equipment and a computer storage medium, which are used for realizing the scheme provided by the embodiment of the application.
The device comprises a memory and a processor, wherein the memory is used for storing instructions or codes, and the processor is used for executing the instructions or codes so that the device can execute the abnormal task identification method according to any embodiment of the application.
The computer storage medium stores codes, and when the codes are executed, equipment for executing the codes realizes the identification method of the abnormal task in any embodiment of the application.
It should be noted that, in the present specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment is mainly described in a different point from other embodiments. In particular, for the apparatus and device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments for relevant points. The apparatus and device embodiments described above are merely illustrative, wherein elements illustrated as separate elements may or may not be physically separate, and elements presented as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
The "first" and "second" in the names of "first", "second" (where present) and the like in the embodiments of the present application are used for name identification only, and do not represent the first and second in sequence.
From the above description of embodiments, it will be apparent to those skilled in the art that all or part of the steps of the above described example methods may be implemented in software plus general hardware platforms. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which may be stored in a storage medium, such as a read-only memory (ROM)/RAM, a magnetic disk, an optical disk, or the like, including several instructions for causing a computer device (which may be a personal computer, a server, or a network communication device such as a router) to perform the methods described in the embodiments or some parts of the embodiments of the present application.
The foregoing is merely one specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A method for identifying an abnormal task, the method comprising:
acquiring operation data of a task to be identified; the operation data are data generated by executing the task to be identified;
performing feature extraction processing on the operation data through an abnormal task identification model to obtain key features corresponding to the operation data; the abnormal task identification model is obtained based on training marking data; the training marking data is obtained by marking training operation data by using marking rules; the labeling rule is obtained based on the abnormal type of the computing task;
determining an identification result corresponding to the task to be identified according to the key characteristics through the abnormal task identification model; when the identification result indicates that the task to be identified is an abnormal task, the identification result comprises an abnormal type of the abnormal task.
2. The method according to claim 1, characterized in that the labeling rules are obtained by:
acquiring a plurality of execution stages of the computing task;
determining at least one exception type corresponding to each of the plurality of execution phases;
determining the abnormal conditions corresponding to the abnormal types to obtain an abnormal condition set;
and constructing the labeling rule according to the abnormal type and the abnormal condition set corresponding to each of the plurality of stages.
3. The method of claim 2, wherein the plurality of execution phases includes a task level phase, a mapping phase, a reduction phase, and a processing phase;
the determining at least one exception type corresponding to each of the plurality of execution phases includes:
at least one exception type is determined for each of the task level stage, the map stage, the reduce stage, and the process stage.
4. The method of claim 1, wherein the abnormal task identification model is trained by:
acquiring training operation data; the training operation data is operation data generated by executing abnormal tasks;
carrying out standardization processing on the training operation data to obtain training standard data;
marking the training standard data by using the marking rule to obtain training marking data; the training annotation data comprises annotation anomaly types; the marked abnormal type characterizes the abnormal type of the abnormal task;
determining the training abnormal type of the abnormal task corresponding to the training marking data according to the training marking data through the abnormal task identification model to be trained;
constructing a loss function according to the difference between the labeling anomaly type and the training anomaly type; and training the abnormal task identification model based on the loss function.
5. The method according to any one of claims 1-4, characterized in that after acquiring the operational data of the task to be identified, the method further comprises:
performing standardized processing on the operation data to obtain standard data; wherein the normalization process indicates to convert the operation data into data of a preset format;
the feature extraction processing is performed on the operation data through the abnormal task identification model to obtain key features corresponding to the operation data, including:
and carrying out feature extraction processing on the standard data through an abnormal task identification model to obtain key features corresponding to the operation data.
6. The method according to any one of claims 1-4, wherein when the recognition result indicates that the task to be recognized is an abnormal task, the method further comprises:
matching the abnormal type corresponding to the abnormal task with a preset abnormal task processing table to obtain a matching result; the matching result comprises a target processing mode of the abnormal task; the preset abnormal task processing table comprises processing modes corresponding to a plurality of abnormal tasks respectively; the preset abnormal task processing table comprises the target processing mode.
7. The method of claim 6, wherein after the abnormal task is restored to normal, the method further comprises:
acquiring an actual processing mode for processing the abnormal task;
comparing the actual processing mode with the target processing mode;
and when the comparison result indicates that the actual processing mode is different from the target processing mode, updating the target processing mode in the preset abnormal task processing table by using the actual processing mode.
8. An apparatus for identifying an abnormal task, the apparatus comprising:
the acquisition module is used for acquiring the operation data of the task to be identified; the operation data are data generated by executing the task to be identified;
the extraction module is used for carrying out feature extraction processing on the operation data through the abnormal task identification model to obtain key features corresponding to the operation data; the abnormal task identification model is obtained based on training marking data; the training marking data is obtained by marking training operation data by using marking rules; the labeling rules are obtained based on a plurality of stages of executing tasks;
the identification module is used for determining an identification result corresponding to the task to be identified according to the key characteristics through the abnormal task identification model; when the identification result indicates that the task to be identified is an abnormal task, the identification result comprises an abnormal type of the abnormal task.
9. A computer device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the method of identifying abnormal tasks according to any one of claims 1-7 when the computer program is executed.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein instructions, which when run on a terminal device, cause the terminal device to perform the method of identifying abnormal tasks according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311577033.8A CN117609918A (en) | 2023-11-23 | 2023-11-23 | Abnormal task identification method and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311577033.8A CN117609918A (en) | 2023-11-23 | 2023-11-23 | Abnormal task identification method and related device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117609918A true CN117609918A (en) | 2024-02-27 |
Family
ID=89949165
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311577033.8A Pending CN117609918A (en) | 2023-11-23 | 2023-11-23 | Abnormal task identification method and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117609918A (en) |
-
2023
- 2023-11-23 CN CN202311577033.8A patent/CN117609918A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9542255B2 (en) | Troubleshooting based on log similarity | |
| US20090119313A1 (en) | Determining structure of binary data using alignment algorithms | |
| US10769043B2 (en) | System and method for assisting user to resolve a hardware issue and a software issue | |
| US20190095318A1 (en) | Test-assisted application programming interface (api) learning | |
| WO2020197624A1 (en) | Method for predicting the successfulness of the execution of a devops release pipeline | |
| US9043651B2 (en) | Systematic failure remediation | |
| CN110825827B (en) | Entity relationship recognition model training method and device and entity relationship recognition method and device | |
| CN112579462A (en) | Test case acquisition method, system, equipment and computer readable storage medium | |
| CN112052305A (en) | Information extraction method and device, computer equipment and readable storage medium | |
| CN114691525A (en) | Test case selection method and device | |
| CN115659175A (en) | Multi-mode data analysis method, device and medium for micro-service resources | |
| CN107579944B (en) | Artificial intelligence and MapReduce-based security attack prediction method | |
| CN114328277A (en) | Software defect prediction and quality analysis method, device, equipment and medium | |
| CN116088846A (en) | Processing method, related device and equipment for continuous integrated code format | |
| Li et al. | Automatic identification of assumptions from the hibernate developer mailing list | |
| CN110750637B (en) | Text abstract extraction method, device, computer equipment and storage medium | |
| CN117609918A (en) | Abnormal task identification method and related device | |
| CN111324344A (en) | Code statement generation method, device, equipment and readable storage medium | |
| EP4071599A1 (en) | System and method for modernization of legacy batch based on functional context | |
| CN113076217B (en) | Disk fault prediction method based on domestic platform | |
| CN115576789A (en) | Method and system for identifying lost user | |
| CN113517998B (en) | Processing method, device, equipment and storage medium of early warning configuration data | |
| CN115099344A (en) | Model training method and device, user portrait generation method and device, and equipment | |
| CN114821396A (en) | Normative detection method, device and storage medium for LNG unloading operation process | |
| CN113537349A (en) | Method, device, equipment and storage medium for identifying hardware fault of large host |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |