CN115344416A - Abnormal log screening method, system and device and computer readable storage equipment - Google Patents
Abnormal log screening method, system and device and computer readable storage equipment Download PDFInfo
- Publication number
- CN115344416A CN115344416A CN202211052221.4A CN202211052221A CN115344416A CN 115344416 A CN115344416 A CN 115344416A CN 202211052221 A CN202211052221 A CN 202211052221A CN 115344416 A CN115344416 A CN 115344416A
- Authority
- CN
- China
- Prior art keywords
- log
- fault
- abnormal
- logs
- running
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
- G06F11/0781—Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/079—Root cause analysis, i.e. error or fault diagnosis
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method, a system and a device for screening abnormal logs and a computer readable storage medium, which relate to the field of fault processing, wherein a semantic extraction model and a fault classification model are trained in advance, operation logs of all working modules in equipment are obtained, and the operation logs are input into the semantic extraction model trained in advance as input items to obtain semantic feature information; and inputting the semantic feature information serving as an input item into a pre-trained fault classification model to obtain an output item of the fault classification model, wherein the output item comprises judgment result information and a corresponding fault reason when the operation log is judged to be an abnormal log. Compared with the prior art, the method has the advantages that the abnormal logs generated when the working module has faults and corresponding fault reasons can be quickly positioned without manual checking and screening, omission of the abnormal logs is avoided, the automation degree is high, technicians can conveniently and quickly master the conditions, the operation and maintenance efficiency is improved, and the equipment operation safety is ensured.
Description
Technical Field
The present invention relates to the field of fault handling technologies, and in particular, to a method, a system, a device, and a computer readable storage device for screening an abnormal log.
Background
The method comprises the steps that a storage system and the like possibly have faults in the operation process, correspondingly, a certain number of abnormal logs can be generated, when the reason that the storage system has the faults is subsequently positioned, due to the fact that the scale of the stored logs is usually very large, the abnormal logs mixed in the stored logs bring large workload when technical personnel analyze the problems, and missed abnormal logs are likely to occur in the analysis process of the technical personnel, so that the accurate positioning of the faults is influenced.
Therefore, how to find an effective method to screen the abnormal logs is a problem to be solved urgently at present.
Disclosure of Invention
The invention aims to provide a method, a system and a device for screening abnormal logs and a computer readable storage medium, which can quickly locate the abnormal logs generated when a working module has faults and corresponding fault reasons without manual checking and screening, avoid the omission of the abnormal logs, have high automation degree, facilitate technicians to quickly master the conditions, improve the operation and maintenance efficiency and ensure the operation safety of equipment.
In order to solve the technical problem, the invention provides an abnormal log screening method, which comprises the following steps:
acquiring running logs of all working modules in equipment;
inputting the running log as an input item to a pre-trained semantic extraction model to obtain semantic feature information representing whether a working module corresponding to the running log has a fault;
and inputting the semantic feature information as an input item to a pre-trained fault classification model to obtain an output item of the fault classification model, wherein the output item comprises judgment result information representing whether the operation log is an abnormal log generated when a fault exists in a corresponding working module or not and a corresponding fault reason when the operation log is judged to be the abnormal log.
Preferably, after the obtaining of the operation log of each work module in the device, the method further includes:
calling a regular expression tool to filter the running logs so as to filter repeated running logs in the running logs;
and processing the filtered operation log in a standard format to obtain a standard operation log corresponding to a preset standard format, wherein the standard operation log comprises specific operation record information.
Preferably, the standard running log further includes timestamp information representing occurrence time of the standard running log, and/or address information representing a storage location of the standard running log, and/or module identification information representing an identity of a working module corresponding to the standard running log.
Preferably, when the judgment result information is that the operation log is the abnormal log;
the abnormal log screening method further comprises the following steps:
and determining a target fault working module corresponding to the abnormal log when the standard running log comprises the module identification information, and/or determining target fault time corresponding to the abnormal log when the standard running log comprises the timestamp information.
Preferably, the semantic extraction model is pre-trained and determined based on an ALBERT model, and/or the fault classification model is pre-trained and determined based on a KNN model.
Preferably, the pre-training process of the semantic extraction model and the fault classification model includes:
s21: inputting a first preset number of abnormal training logs marked with fault reasons as output items to a fault classification model to be trained;
s22: let i =1;
s23: inputting a second preset number of training log sets serving as input items to a semantic extraction model to be trained to establish an input-output corresponding relation, wherein the training log sets comprise normal training logs and the first preset number of abnormal training logs;
s24: performing ith training by using the semantic extraction model and the fault classification model, wherein the output item of the semantic extraction model is the input item of the fault classification model;
s25: judging whether the training accuracy of the fault classification model in the ith training is not less than a preset accuracy threshold, if so, entering S27; if not, entering S26;
s26: let i = i +1 and return to S23;
s27: stopping training and determining the current semantic extraction model and fault classification model as the trained semantic extraction model and fault classification model.
Preferably, the output item of the fault classification model further includes a fault level corresponding to the operation log when the operation log is an abnormal log;
the abnormal log screening method further comprises the following steps:
judging whether the fault level of the abnormal log reaches a preset alarm level threshold value or not;
and if so, controlling a prompt module in the equipment to give an alarm.
In order to solve the above technical problem, the present invention further provides an abnormal log screening system, including:
the acquisition unit is used for acquiring the running logs of all the working modules in the equipment;
the characteristic extraction unit is used for inputting the running log as an input item to a pre-trained semantic extraction model so as to obtain semantic characteristic information representing whether a working module corresponding to the running log has a fault or not;
and the abnormal log judging unit is used for inputting the semantic feature information into a fault classification model trained in advance as an input item to obtain an output item of the fault classification model, wherein the output item comprises judgment result information representing whether the running log is an abnormal log generated when the corresponding working module has a fault or not and a corresponding fault reason when the running log is an abnormal log.
In order to solve the above technical problem, the present invention further provides an abnormal log screening device, including:
a memory for storing a computer program;
a processor for performing the steps of the anomaly log screening method as described above.
To solve the above technical problem, the present invention also provides a computer-readable storage medium, comprising:
the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the abnormality log screening method as described above.
The application provides a method, a system, a device and a computer readable storage medium for screening abnormal logs, wherein a semantic extraction model and a fault classification model are trained in advance, so that operation logs of all working modules in equipment are obtained, and the operation logs are input into the pre-trained semantic extraction model as input items to obtain semantic feature information representing whether faults exist in the working modules corresponding to the operation logs; and inputting the semantic feature information as an input item to a pre-trained fault classification model to obtain an output item of the fault classification model, wherein the output item comprises judgment result information representing whether the operation log is an abnormal log generated when the corresponding working module has a fault or not and a corresponding fault reason when the operation log is judged to be the abnormal log. Compared with the prior art, the method has the advantages that the abnormal logs generated when the working module has faults and corresponding fault reasons can be quickly positioned without manual checking and screening, omission of the abnormal logs is avoided, the automation degree is high, technicians can conveniently and quickly master the conditions, the operation and maintenance efficiency is improved, and the equipment operation safety is ensured.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed in the prior art and the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
FIG. 1 is a flow chart of an abnormal log screening method provided by the present invention;
FIG. 2 is a schematic structural diagram of an anomaly log screening system provided by the present invention;
fig. 3 is a schematic structural diagram of an abnormal log screening device provided by the present invention.
Detailed Description
The core of the invention is to provide a method, a system and a device for screening abnormal logs and a computer readable storage medium, which can quickly locate the abnormal logs generated when the working module has faults and corresponding fault reasons without manual checking and screening, avoid the omission of the abnormal logs, have high automation degree, facilitate technicians to quickly master the conditions, improve the operation and maintenance efficiency and ensure the operation safety of equipment.
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a flowchart of an abnormal log screening method provided by the present invention.
In the embodiment, a certain number of abnormal logs are generated due to the fact that a storage system and the like are difficult to avoid to have faults in the operation process, the abnormal logs are mixed in the normal logs, large workload is brought to subsequent analysis and processing of technicians due to the fact that the stored logs are large in scale, omission occurs in the analysis process, and accurate positioning of the faults is influenced. In order to solve the technical problem, the application provides an abnormal log screening method which can quickly locate an abnormal log generated when a working module has a fault and a corresponding fault reason.
The abnormal log screening method comprises the following steps:
s11: acquiring running logs of all working modules in equipment;
specifically, the device includes, but is not limited to, various computer devices such as a server, a client, etc., and is not particularly limited herein, and more specifically, the device may obtain the operation log of each work module in a storage system in the device; the working module essentially refers to a functional module that can implement various functions in the Device, such as a Storage System as an example, the working module includes but is not limited to NFS (Network File System), LIB library, MDS (Meta Data Source, metadata management System), OSD (Object Storage Device), MON (Monitor, etc.), and the logs generated by the working module are the operation logs.
It should be further noted that, here, the operation logs of each working module may be obtained in real time and a subsequent analysis step is performed, or the operation logs of each working module may also be obtained periodically and a subsequent analysis step is performed to determine whether there is an abnormal log in the period, which is not limited herein; the number of the operation logs is usually plural.
S12: inputting the running log as an input item into a pre-trained semantic extraction model to obtain semantic feature information representing whether a working module corresponding to the running log has a fault or not;
specifically, a semantic extraction model is trained in advance, the operation log is used as an input item and input into the semantic extraction model to perform feature vectorization processing, and an output item of the model is the semantic feature information.
S13: and inputting the semantic feature information as an input item to a pre-trained fault classification model to obtain an output item of the fault classification model, wherein the output item comprises judgment result information representing whether the operation log is an abnormal log generated when the corresponding working module has a fault or not and a corresponding fault reason when the operation log is judged to be the abnormal log.
Specifically, the fault classification model is trained in advance, semantic feature information extracted from each running log is input to the fault classification model as an input item, the judgment result information corresponding to each running log can be obtained, namely whether the running log is an abnormal log or not, and a corresponding fault reason is given when the running log is judged to be the abnormal log, so that technicians can further refine and analyze the fault subsequently.
Compared with the prior art, the abnormal log screening method has the advantages that the abnormal logs generated when the working module has faults and corresponding fault reasons can be quickly located from a large number of running logs without manual checking and screening, omission of the abnormal logs is avoided, simplicity in implementation is realized, better text information classification capability is realized, the automation degree is high, technicians can conveniently and quickly master the conditions, the operation and maintenance efficiency is improved, and the equipment running safety is guaranteed.
On the basis of the above-described embodiment:
as a preferred embodiment, after obtaining the operation logs of the work modules in the device, the method further includes:
calling a regular expression tool to filter the running logs so as to filter repeated running logs in the running logs;
and processing the filtered operation log in a standard format to obtain a standard operation log corresponding to a preset standard format, wherein the standard operation log comprises specific operation record information.
In this embodiment, in order to implement standardized processing on the operation logs, after the operation logs of each working module in the device are obtained, the regular expression tool (essentially, pre-written program code) may be called to filter multiple operation logs first to filter repeated operation logs in the operation logs, and needless operation logs in the operation logs may be further filtered, where useless refers to information that is printed at a normal operation timing of the storage system, and then the obtained operation logs, and the failure judgment of the operation logs on abnormal logs may be regarded as useless.
Secondly, considering that the running log also comprises some non-value information such as punctuation marks, space information and the like, the filtered running log is processed in a standard format to obtain a standard running log corresponding to a preset standard format, and then the standard running log is input to the semantic extraction model as an input item for subsequent processing. It should be noted that the preset standard format is essentially designed to filter out worthless information such as punctuations and space information, so that the format of the standard running log is unified, and the standard running log is convenient to be subsequently input into a pre-trained model for analysis.
In a preferred embodiment, the standard running log further includes timestamp information indicating the occurrence time of the standard running log, and/or address information indicating the storage location of the standard running log, and/or module identification information indicating the identity of the working module corresponding to the standard running log.
In this embodiment, the standard operation log may further include the timestamp information and/or the address information and/or the module identification information. It can be understood that the three parts of content are the content carried by the operation log itself, that is, the standard format processing itself in the above embodiment does not change the content, but removes the redundant worthless information and retains the timestamp information, the address information and the module identification information. In addition, the storage positions of the running logs corresponding to different working modules are different, and the storage positions of the corresponding standard running logs are also different; the same working module can generate running logs at different time, the corresponding timestamp information of each running log is different, and the timestamp information of the corresponding standard running logs is also different; the identity identifiers of different working modules are different, and the running log stores the module identification information corresponding to the identity identifier, that is, the standard running log stores the module identification information.
As a preferred embodiment, when the determination result information is that the operation log is an abnormal log;
the abnormal log screening method further comprises the following steps:
and determining a target fault working module corresponding to the abnormal log when the standard running log comprises the module identification information, and/or determining target fault time corresponding to the abnormal log when the standard running log comprises the timestamp information.
In this embodiment, the inventor further considers that when the determination result information is that the operation log is an abnormal log, the identity information of the corresponding working module with the abnormality may be further provided, and/or the time when the abnormality occurs, so that, on the basis that the standard operation log has been determined as the abnormal log, if the current standard operation log includes module identification information, and the module identification information is module identification information representing the identity of the working module corresponding to the standard operation log, the working module corresponding to the abnormal log may be further determined to be a target fault working module based on the storage, so that a technician can quickly locate a fault source; if the current standard running log comprises timestamp information, and the timestamp information is timestamp information representing the occurrence time of the standard running log, then the timestamp information corresponding to the abnormal log can be further determined based on the storage to be the target fault time, so that a technician can quickly locate the fault occurrence time.
Therefore, the practicability of the abnormal log screening method is further improved through the setting of the execution logic.
As a preferred embodiment, the semantic extraction model is pre-trained and determined based on the ALBERT model, and/or the fault classification model is pre-trained and determined based on the KNN model.
In this embodiment, it is provided that the semantic extraction model may be trained and determined in advance based on the ALBERT model, and/or the fault classification model is trained and determined in advance based on the KNN model. Specifically, the BERT model (Bidirectional Encoder expressions from Transformer) is a representation of a Bidirectional Encoder based on Transformer, and is a pre-trained language characterization model; the ALBERT model (A Lite BERT) is a lightweight BERT model, and model parameters can be simplified by using two techniques of decomposing parameters of embedded layers and realizing parameter sharing among different levels. Therefore, the execution logic of the semantic extraction model in the application can be reliably realized; of course, the semantic extraction model may also select a model or algorithm such as a TF-IDF technology (Term Frequency-Inverse Document Frequency), a word2Vec model (word to Vector), a GloVe algorithm (Global Vector), a Doc2Vec algorithm, a Fasttext model, and the like, and perform pre-training to obtain the semantic extraction model for realizing feature extraction in the present application.
Based on the KNN (K-nearest neighbor, K nearest neighbor) model, the following decision can be implemented, namely: if a sample belongs to a certain class in the K most similar samples in the feature space (i.e., the nearest neighbors in the feature space), then the sample also belongs to this class. Therefore, the execution logic of the fault classification model in the application can be reliably realized; of course, the fault classification model may also select a NB model (naive bayes classifier), a random forest model, an SVM classification model, a K-means algorithm, and other models or algorithms to perform pre-training, so as to obtain the fault classification module that realizes classification and fault cause determination in the present application.
As a preferred embodiment, the pre-training process of the semantic extraction model and the fault classification model includes:
s21: inputting a first preset number of abnormal training logs marked with fault reasons as output items to a fault classification model to be trained;
s22: let i =1;
s23: inputting a second preset number of training log sets serving as input items to the semantic extraction model to be trained to establish an input-output corresponding relation, wherein the training log sets comprise normal training logs and a first preset number of abnormal training logs;
s24: performing ith training by using a semantic extraction model and a fault classification model, wherein an output item of the semantic extraction model is an input item of the fault classification model;
s25: judging whether the training accuracy of the fault classification model in the ith training is not less than a preset accuracy threshold, if so, entering S27; if not, entering S26;
s26: let i = i +1 and return to S23;
s27: stopping training and determining the current semantic extraction model and fault classification model as the trained semantic extraction model and fault classification model.
In this embodiment, a pre-training process for the semantic extraction model and the fault classification model is provided, which is specifically described above and will not be described herein again. It should be noted that, in essence, the semantic extraction model and the fault classification model are taken as a whole during training, so the training log set will be input to the semantic extraction model to be trained as an input item, the abnormal training log labeled with the fault cause will be input to the fault classification model to be trained as an output item, and the output item of the semantic extraction model is the input item of the fault classification model.
It should be further noted that the first preset number and the second preset number are set according to actual requirements, but in order to ensure the training effect, it is necessary to ensure that the training data volume is large enough; furthermore, the total number of normal training logs plus the first preset number equals the second preset number; and the preset accuracy threshold value can be set according to actual requirements.
It is understood that the abnormal training log and the normal training log may be logs processed by the filtering and standard format described in the above embodiments.
As a preferred embodiment, the output item of the fault classification model further includes a fault level corresponding to the operation log which is judged to be an abnormal log;
the abnormal log screening method further comprises the following steps:
judging whether the fault level of the abnormal log reaches a preset alarm level threshold value or not;
if yes, a prompt module in the control equipment gives an alarm.
In this embodiment, in order to alarm the occurring fault in time, the output item of the fault classification model may further include a fault level corresponding to the operation log determined as an abnormal log, and specifically, the fault level may include the following 5 levels:
a first grade: FATAL (FATAL error) for indicating a very serious event that may cause the application to terminate execution;
and a second stage: ERROR: for indicating an error event, but the application may still be able to continue running;
and a third grade: WARN: for indicating potentially dangerous conditions;
fourth level: INFO (information): the system is used for indicating description information and describing the application running process from a coarse granularity;
and a fifth grade: DEBUG: for specifying detailed event information, most useful for debugging applications, these 5 levels can be understood as being in a certain class of abnormal faults.
Therefore, the preset alarm level threshold value can be set corresponding to the fault level, so that technicians can master the situation as soon as possible; in addition, the specific manner of the alarm by the control prompting module includes but is not limited to: the voice prompt module is controlled to perform voice prompt alarm, and/or the display module is controlled to perform display alarm, and the display module includes but is not limited to a display screen, which is not limited herein.
It should be noted that, when the output item of the fault classification model further includes the fault level, corresponding to the pre-training process for the semantic extraction model and the fault classification model described in the above embodiment, the step S21 needs to be replaced with: and inputting the abnormal training logs with the first preset number and labeled fault reasons and fault grades as output items to the fault classification model to be trained.
Referring to fig. 2, fig. 2 is a schematic structural diagram of an abnormal log screening system according to the present invention.
The abnormal log screening system comprises:
an obtaining unit 31, configured to obtain an operation log of each work module in the device;
the feature extraction unit 32 is configured to input the running log as an input item to a pre-trained semantic extraction model to obtain semantic feature information indicating whether a fault exists in a working module corresponding to the running log;
and the abnormal log judging unit 33 is configured to input the semantic feature information as an input item to a fault classification model trained in advance to obtain an output item of the fault classification model, where the output item includes judgment result information representing whether the operation log is an abnormal log generated when the corresponding working module has a fault, and a corresponding fault reason when the operation log is judged to be the abnormal log.
For introduction of the abnormal log screening system provided by the present invention, please refer to the above embodiment of the abnormal log screening method, which is not described herein again.
As a preferred embodiment, the abnormality log screening system further includes:
the filtering unit is used for calling a regular expression tool to filter the running logs after the obtaining unit 31 so as to filter repeated running logs in the running logs;
and the preprocessing unit is used for processing the filtered running log in a standard format to obtain a standard running log corresponding to a preset standard format, wherein the standard running log comprises specific running record information.
As a preferred embodiment, when the determination result information is that the operation log is the abnormality log; the abnormal log screening system also comprises a fault module positioning unit and/or a fault time positioning unit;
the fault module positioning unit is used for determining a target fault working module corresponding to the abnormal log when the standard running log comprises the module identification information;
and the fault time positioning unit is used for determining the target fault time corresponding to the abnormal log when the standard running log comprises the timestamp information.
As a preferred embodiment, the abnormality log screening system further includes a pre-training unit, and the pre-training unit includes:
the first input unit is used for inputting a first preset number of abnormal training logs marked with fault reasons as output items to a fault classification model to be trained;
a first assignment unit for i =1;
the second input unit is used for inputting a second preset number of training log sets serving as input items to the semantic extraction model to be trained so as to establish an input-output corresponding relation, wherein the training log sets comprise normal training logs and the first preset number of abnormal training logs;
the training unit is used for performing the ith training by utilizing the semantic extraction model and the fault classification model, wherein the output item of the semantic extraction model is the input item of the fault classification model;
the first judging unit is used for judging whether the training accuracy of the fault classification model in the ith training is not less than a preset accuracy threshold value or not, and if so, the model determining unit is triggered; if not, triggering a second assignment unit;
the second assignment unit is used for enabling i = i +1 and returning to the second input unit;
and the model determining unit is used for stopping training and determining the current semantic extraction model and the fault classification model as the trained semantic extraction model and the trained fault classification model.
As a preferred embodiment, the output item of the fault classification model further includes a fault level corresponding to the operation log determined to be an abnormal log;
the abnormality log screening system further includes:
the second judging unit is used for judging whether the fault level of the abnormal log reaches a preset alarm level threshold value or not; if yes, triggering an alarm unit;
and the alarm unit is used for controlling a prompt module in the equipment to alarm.
Referring to fig. 3, fig. 3 is a schematic structural diagram of an abnormal log screening device provided in the present invention.
The abnormality log screening device includes:
a memory 41 for storing a computer program;
a processor 42 for performing the steps of the anomaly log screening method as described above.
For introduction of the abnormal log screening device provided by the present invention, please refer to the above embodiment of the abnormal log screening method, which is not described herein again.
The present invention also provides a computer-readable storage medium comprising:
the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the abnormality log screening method as described above.
For introduction of the computer-readable storage medium provided in the present invention, please refer to the above-mentioned embodiment of the method for screening an abnormal log, which is not described herein again.
In the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed in the embodiment corresponds to the method disclosed in the embodiment, so that the description is simple, and the relevant points can be referred to the description of the method part. Relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising one of 8230; \8230;" 8230; "does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. An abnormal log screening method, comprising:
acquiring running logs of all working modules in equipment;
inputting the running log as an input item to a pre-trained semantic extraction model to obtain semantic feature information representing whether a working module corresponding to the running log has a fault or not;
and inputting the semantic feature information as an input item to a pre-trained fault classification model to obtain an output item of the fault classification model, wherein the output item comprises judgment result information representing whether the operation log is an abnormal log generated when a fault exists in a corresponding working module or not and a corresponding fault reason when the operation log is judged to be the abnormal log.
2. The method for screening the abnormal log as claimed in claim 1, wherein after the obtaining of the operation log of each working module in the device, the method further comprises:
calling a regular expression tool to filter the running logs so as to filter repeated running logs in the running logs;
and processing the filtered operation log in a standard format to obtain a standard operation log corresponding to a preset standard format, wherein the standard operation log comprises specific operation record information.
3. The abnormal log screening method according to claim 2, wherein the standard running log further includes timestamp information representing occurrence time of the standard running log, and/or address information representing a storage location of the standard running log, and/or module identification information representing an identity of a working module corresponding to the standard running log.
4. The abnormality log screening method according to claim 3, wherein when the determination result information is that the running log is the abnormality log;
the abnormal log screening method further comprises the following steps:
and determining a target fault working module corresponding to the abnormal log when the standard running log comprises the module identification information, and/or determining target fault time corresponding to the abnormal log when the standard running log comprises the timestamp information.
5. The abnormality log screening method of claim 1, wherein said semantic extraction model is pre-trained and determined based on an ALBERT model and/or said fault classification model is pre-trained and determined based on a KNN model.
6. The abnormal log screening method of claim 1, wherein the pre-training process of the semantic extraction model and the fault classification model comprises:
s21: inputting a first preset number of abnormal training logs marked with fault reasons as output items to a fault classification model to be trained;
s22: let i =1;
s23: inputting a second preset number of training log sets serving as input items to a semantic extraction model to be trained to establish an input-output corresponding relation, wherein the training log sets comprise normal training logs and the first preset number of abnormal training logs;
s24: performing ith training by using the semantic extraction model and the fault classification model, wherein the output item of the semantic extraction model is the input item of the fault classification model;
s25: judging whether the training accuracy of the fault classification model in the ith training is not less than a preset accuracy threshold, if so, entering S27; if not, entering S26;
s26: let i = i +1 and return to S23;
s27: stopping training and determining the current semantic extraction model and fault classification model as the trained semantic extraction model and fault classification model.
7. The abnormal log screening method according to any one of claims 1 to 6, wherein the output item of the fault classification model further includes a fault level corresponding to the case where the operation log is an abnormal log;
the abnormal log screening method further comprises the following steps:
judging whether the fault level of the abnormal log reaches a preset alarm level threshold value or not;
and if so, controlling a prompt module in the equipment to give an alarm.
8. An anomaly log screening system, comprising:
the acquisition unit is used for acquiring the running logs of all the working modules in the equipment;
the characteristic extraction unit is used for inputting the running log as an input item to a pre-trained semantic extraction model so as to obtain semantic characteristic information representing whether a working module corresponding to the running log has a fault or not;
and the abnormal log judging unit is used for inputting the semantic feature information as an input item to a pre-trained fault classification model to obtain an output item of the fault classification model, wherein the output item comprises judgment result information representing whether the operation log is an abnormal log generated when the corresponding working module has a fault or not and a corresponding fault reason when the operation log is an abnormal log.
9. An abnormality log screening device, comprising:
a memory for storing a computer program;
a processor for performing the steps of the abnormality log screening method of any one of claims 1 to 7.
10. A computer-readable storage medium, comprising:
the computer readable storage medium has stored thereon a computer program which, when executed by a processor, carries out the steps of the abnormality log screening method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211052221.4A CN115344416A (en) | 2022-08-31 | 2022-08-31 | Abnormal log screening method, system and device and computer readable storage equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211052221.4A CN115344416A (en) | 2022-08-31 | 2022-08-31 | Abnormal log screening method, system and device and computer readable storage equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115344416A true CN115344416A (en) | 2022-11-15 |
Family
ID=83955051
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211052221.4A Pending CN115344416A (en) | 2022-08-31 | 2022-08-31 | Abnormal log screening method, system and device and computer readable storage equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115344416A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115576735A (en) * | 2022-12-06 | 2023-01-06 | 苏州浪潮智能科技有限公司 | Fault positioning method and device and computer readable storage medium |
| CN116149898A (en) * | 2023-04-17 | 2023-05-23 | 阿里云计算有限公司 | Method for determining abnormal type of kernel, electronic equipment and storage medium |
-
2022
- 2022-08-31 CN CN202211052221.4A patent/CN115344416A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115576735A (en) * | 2022-12-06 | 2023-01-06 | 苏州浪潮智能科技有限公司 | Fault positioning method and device and computer readable storage medium |
| CN116149898A (en) * | 2023-04-17 | 2023-05-23 | 阿里云计算有限公司 | Method for determining abnormal type of kernel, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhang et al. | Robust log-based anomaly detection on unstable log data | |
| CN110321371B (en) | Log data anomaly detection method, device, terminal and medium | |
| CN115344416A (en) | Abnormal log screening method, system and device and computer readable storage equipment | |
| US10795753B2 (en) | Log-based computer failure diagnosis | |
| US9652318B2 (en) | System and method for automatically managing fault events of data center | |
| JP2018045403A (en) | Abnormality detection system and abnormality detection method | |
| CN111435366A (en) | Equipment fault diagnosis method and device and electronic equipment | |
| CN108460397B (en) | Method and device for analyzing equipment fault type, storage medium and electronic equipment | |
| US9152604B2 (en) | System and method for event logging in a technical installation or a technical process | |
| CN109977146B (en) | Fault diagnosis method and device and electronic equipment | |
| CN116755992B (en) | Log analysis method and system based on OpenStack cloud computing | |
| CN111400435B (en) | Mail alarm convergence method, device, computer equipment and storage medium | |
| CN114118295A (en) | Anomaly detection model training method, anomaly detection device and medium | |
| CN112988509A (en) | Alarm message filtering method and device, electronic equipment and storage medium | |
| CN111858352B (en) | Method, device, equipment and storage medium for automatic test monitoring | |
| JP6919438B2 (en) | Fault analysis support device, incident management system, fault analysis support method and program | |
| CN110855489A (en) | Fault processing method and device and fault processing device | |
| CN115408236A (en) | Log data auditing system, method, equipment and medium | |
| CN114881112A (en) | System anomaly detection method, device, equipment and medium | |
| CN115098326A (en) | System anomaly detection method and device, storage medium and electronic equipment | |
| CN116635843A (en) | Apparatus, computing platform and method for analyzing log files of industrial plants | |
| EP2960837A1 (en) | System and method for supporting global effect analysis | |
| CN117909163A (en) | Device and method for managing database logs | |
| CN118113564A (en) | Alarm processing method, device, electronic equipment and storage medium | |
| EP3156867B1 (en) | Maintenance system and method for a reliability centered maintenance |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |