CN117608902A - System log-based system abnormality judgment method, device and equipment - Google Patents

Abstract

The embodiment of the specification provides a system exception judging method, device and equipment based on a system log, which can be applied to the technical field of artificial intelligence. The method comprises the following steps: acquiring target log data corresponding to a target system; extracting target log features from the target log data; determining an anomaly factor of the target log feature based on an anomaly log rule; the abnormal log rule is used for limiting log features corresponding to an abnormal system; the abnormal factors are used for representing abnormal conditions reflected by the target system based on the log; and judging whether the target system is abnormal or not based on the abnormal factors. The method automatically analyzes the log to find out the abnormality of the system in time, reduces the risk of system faults and improves the stability of the system. In addition, the system abnormality which is newly generated can be identified through analysis, so that the actual application effect of the method is ensured, the application scene is expanded, and further, the method can be rapidly deployed and monitored in real time according to the requirements of the production environment.

Description

System log-based system abnormality judgment method, device and equipment

Technical Field

The embodiment of the specification relates to the technical field of artificial intelligence, in particular to a system exception judging method, device and equipment based on a system log.

Background

With the development of computer technology, more and more transactions can be transferred to online transaction, and corresponding software systems are correspondingly generated. Based on the continuous improvement of the requirements, the number of software systems is increased, and meanwhile, the complexity of the system is also increased. In the actual operation process, abnormal conditions, such as performance degradation and even system faults, are unavoidable. For abnormal conditions of the system, even if the abnormality of the system is identified, the normal and effective operation of the system is ensured.

At present, when a system is analyzed to be abnormal, analysis is often carried out by means of system log data. Log data is an important information source that records the system operation state, user operations, error information, and the like, so that anomalies can be found by the log data. Since the log data has huge data volume and high complexity, abnormality recognition is generally performed by an automated method. However, the existing abnormality recognition method often sets a fixed judgment rule, lacks recognition capability for the newly-occurring abnormality, has higher maintenance cost, and makes the judgment rule itself more complex along with the operation of the system, so that recognition errors are very easy to occur. Therefore, a method for identifying system anomalies rapidly and accurately is needed.

Disclosure of Invention

The embodiment of the specification aims to provide a system log-based system abnormality judging method, device and equipment so as to solve the problem of how to quickly and accurately identify system abnormalities.

In order to solve the above technical problems, an embodiment of the present disclosure provides a system exception determination method based on a system log, including: acquiring target log data corresponding to a target system; extracting target log features from the target log data; determining an anomaly factor of the target log feature based on an anomaly log rule; the abnormal log rule is used for limiting log features corresponding to an abnormal system; the abnormal factors are used for representing abnormal conditions reflected by the target system based on the log; and judging whether the target system is abnormal or not based on the abnormal factors.

In some embodiments, before extracting the target log feature from the target log data, the method further includes: preprocessing the target log data; the preprocessing comprises at least one of irrelevant information cleaning, word segmentation and stem extraction.

In some implementations, the target log features include text features and supplemental features; the extracting the target log features from the target log data includes: vector conversion is carried out on the target log data to obtain text characteristics; converting the target log data into numerical data; extracting supplemental features from the numerical data; the supplemental features include at least one of log level, timestamp, event occurrence frequency, log level, thread information.

In some embodiments, the anomaly log rule corresponds to an anomaly identification model; the anomaly identification model is obtained by the following steps: acquiring sample log data; extracting features of the sample log data to obtain sample log features; dividing the sample log features into training log features and verification log features; adjusting model types and super parameters of a pre-constructed abnormal recognition model based on verification log features; and training the regulated abnormal recognition model by utilizing the training log characteristics until the abnormal recognition model accords with the model application conditions.

Based on the above embodiment, the training the adjusted anomaly identification model by using the training log feature until the anomaly identification model meets a model application condition includes: optimizing model parameters of the anomaly identification model by using training log features; calculating a loss function corresponding to the optimized abnormal recognition model; judging whether the abnormal recognition model accords with the model application condition or not based on the loss function; the model application condition includes that the decreasing amplitude of the loss function is smaller than a preset optimization threshold.

Based on the above embodiment, the dividing the sample log features into training log features and verification log features includes: dividing the sample log features into training log features, verification log features and test log features; correspondingly, after optimizing the anomaly identification model based on the verification result, the method further comprises the following steps: under the condition that the optimized abnormal recognition model accords with the model application condition, testing the abnormal recognition model by using the test log characteristics to obtain a test result; and evaluating the abnormal recognition model based on the test result to obtain an evaluation result.

Based on the above embodiment, the testing the anomaly identification model by using the test log feature to obtain a test result includes: calculating an evaluation index of the abnormality recognition model; the evaluation index comprises at least one of an accuracy rate, a recall rate and an F1 score.

In some embodiments, after the determining whether the target system is abnormal based on the abnormality factor, the method further includes: under the condition that the abnormality of the target system is determined, carrying out system abnormality positioning based on the abnormality factors so as to determine the cause of the system fault; and sending the system fault reasons to a system manager.

In some embodiments, the target log data includes real-time log data obtained from real-time detection of the target system.

The embodiment of the specification also provides a system abnormality judgment device based on the system log, which comprises: the log data acquisition module is used for acquiring target log data corresponding to a target system; the log feature extraction module is used for extracting target log features from the target log data; the abnormal factor determining module is used for determining abnormal factors of the target log characteristics based on an abnormal log rule; the abnormal log rule is used for limiting log features corresponding to an abnormal system; the abnormal factors are used for representing abnormal conditions reflected by the target system based on the log; and the abnormality judging module is used for judging whether the target system is abnormal or not based on the abnormality factors.

The embodiment of the specification also provides electronic equipment, which comprises a memory and a processor; the memory is used for storing computer programs/instructions; the processor is configured to execute the computer program/instructions to implement the steps of the system log-based system anomaly determination method described above.

The present specification embodiment also proposes a computer storage medium having stored thereon a computer program/instruction which, when executed, is adapted to implement the steps of the system log-based system abnormality determination method described above.

The embodiments of the present specification also propose a computer program product comprising computer programs/instructions which, when executed, are adapted to implement the steps of the system log based system anomaly determination method described above.

As can be seen from the technical solutions provided in the embodiments of the present disclosure, in the system exception judging method based on the system log in the embodiments of the present disclosure, by acquiring log data corresponding to a target system, extracting log features in the log data, further determining exception factors in the target log features according to exception log rules, and finally judging whether the target system is abnormal by analyzing the exception factors. The method can automatically analyze the log to find out the abnormality of the system in time, reduce the risk of system faults and improve the stability of the system. In addition, the system abnormality which is newly generated can be identified through analysis, so that the actual application effect of the method is ensured, the application scene is expanded, and further, the method can be rapidly deployed and monitored in real time according to the requirements of the production environment. The method can be widely applied to software systems of various scales, and has important significance for improving the operation and maintenance efficiency of the system, reducing the operation and maintenance cost and ensuring the stable operation of the system.

Drawings

In order to more clearly illustrate the embodiments of the present description or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments described in the present description, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flowchart of a system exception determination method based on a system log according to an embodiment of the present disclosure;

fig. 2 is a block diagram of a system abnormality determination apparatus based on a system log according to an embodiment of the present disclosure.

Detailed Description

The technical solutions of the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is apparent that the described embodiments are only some embodiments of the present specification, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure.

In order to solve the above technical problems, a system abnormality determination method based on a system log according to an embodiment of the present disclosure is described. The execution main body of the system log-based system abnormality judgment method is corresponding electronic equipment, and the electronic equipment comprises, but is not limited to, a server, an industrial personal computer, a PC (personal computer) and the like. As shown in fig. 1, the system log-based system anomaly determination method may include the following specific implementation steps.

S110: target log data corresponding to a target system is acquired.

The target system is a system that needs to perform exception analysis, and the target system may be, for example, a service processing system, an internal interaction system, or another type of system, which is not limited in this regard.

The target system can generate log data in the application process, the log data is mainly used for recording the working process of the system, and the log data can be generated when corresponding transactions are executed or the target system detects the target system. Log data may be a direct hint to some simple fixed anomalies, but some more complex anomalies generally need to be determined by analysis of the log data.

The target log data may be read directly from the system or may be obtained by a corresponding log collection tool, for example, logstash, flume may be used to collect log data from a Java system of an application, server or other device, and store the data to a suitable storage system, such as Hadoop HDFS, elastic search, etc.

In some embodiments, in order to ensure that the abnormality in the system can be quickly and accurately identified based on the log data in the later application process, a preprocessing operation may also be performed on the target log data. The main purpose of the preprocessing operation is to remove extraneous data from the log data to ensure the efficiency and accuracy of the subsequent analysis process.

Specifically, the preprocessing operation may include at least one of irrelevant information cleaning, word segmentation, and stem extraction.

Irrelevant information cleansing includes data cleansing and stop word removal. The data cleaning mainly extracts key information such as time stamp, log level, thread information, class name, log content and the like in log data through regular expressions or customized analysis rules, and deletes irrelevant information such as empty lines, notes and the like. Stop word removal mainly removes common stop words, such as "and", "in", and the like, from the word sequence after word segmentation. These words frequently appear in text, but provide less information for the anomaly detection task and can therefore be removed.

The word segmentation is used for carrying out word segmentation processing on the log content, and dividing the text into word sequences and meaningful vocabulary units. The implementation by Natural Language Processing (NLP) toolkit uses existing word segmentation tools such as NLTK, jieba, etc. The data after word segmentation can reduce the sparsity of the data and improve the training effect of the model.

The stem extraction is used for reducing words into stem forms thereof so as to reduce vocabulary and improve the effect of subsequent feature extraction. For example, words such as "running", "ran" are reduced to "run".

The preprocessing operation can also be realized by means of a corresponding machine learning model so as to accelerate the processing efficiency. The specific implementation process can be set according to the requirements of practical applications, and will not be described herein.

In some embodiments, the target log data may also be real-time log data obtained in real-time from a target system. Specifically, the real-time detection can be performed on the target system, so as to obtain the logs generated by the target system in real time, and analyze the logs as real-time log data. By extracting and analyzing the real-time log data, the timeliness of the data is guaranteed, and therefore the usability of the final identification effect is improved.

S120: and extracting target log features from the target log data.

After log data is obtained, target log features may be extracted therefrom. The target log features can be key information in log data, and analysis efficiency in a subsequent process can be accelerated through extraction of the target log features.

In some implementations, the log features can include text features and supplemental features. Text features may be features that are embodied in text form, with supplemental features being primarily described in terms of data values.

Specifically, for text features, text data is converted into a vector with a fixed length by using a Word embedding technology according to the preprocessed log content, for example, word2Vec, gloVe, fastText and other Word embedding technologies are used. Word embedding can map words with similar semantics to a similar vector space, which is beneficial to capturing semantic information of text data by a model.

The supplemental features may include, among other things, log level (INFO, WARNING, ERROR, etc.), time stamp (hours, minutes, seconds, etc.), frequency of occurrence of events, etc. For discrete data, such as log level and thread information, a one-hot encoding (one-hot encoding) method may be used to convert the discrete data into numerical data. The single thermal coding can effectively represent the category information of discrete data, and meanwhile, misunderstanding of a model on the original numerical value is avoided.

In practical application, other target log features may be set according to requirements, and are not limited to the above examples, and are not described herein.

By extracting the log features, key information can be extracted from huge log data, and generalization capability and accuracy in the subsequent processing process are enhanced.

S130: determining an anomaly factor of the target log feature based on an anomaly log rule; the abnormal log rule is used for limiting log features corresponding to an abnormal system; the anomaly factors are used for representing the anomaly conditions reflected by the target system based on the log.

The abnormal log rule is used for limiting log features corresponding to the abnormal system. The log features corresponding to the target log data are analyzed through the abnormal log rules, so that the abnormal condition reflected by the target log can be rapidly and accurately determined, and whether the system is abnormal or not is judged.

In some implementations, the anomaly log rules can correspond to an anomaly identification model. The anomaly recognition model is a corresponding deep neural network learning model, and for example, a cyclic neural network (RNN), a long and short time memory network (LSTM) or a gating cyclic unit (GRU) can be selected as a main model structure. And taking the target log characteristics as the input of a model, processing the target log characteristics through a multi-layer RNN or LSTM structure, and finally outputting an abnormality detection result through a full connection layer and an activation function.

The anomaly identification model may specifically include an input layer, a hidden layer, and an output layer. The input layer receives the feature vector, the hidden layer is responsible for learning the internal rule of the data, and the output layer outputs an abnormality detection result.

Before the application anomaly is modeled, the model may be trained first, and corresponding sample log data may be obtained first. The sample log data can be marked in advance to complete model training in a supervision or semi-supervision mode. The data annotation can be accomplished manually or semi-automatically.

The preprocessed log data may then be divided into training log features, verification log features, and test log features. The training log features are used for model training, the verification log features are used for model tuning, and the test log features are used for evaluating the final performance of the model. Typically, 80% of the data may be used as a training set, 10% as a validation set, and 10% as a test set.

The verification log feature is mainly used for verifying the model to select a proper model. The super parameters such as the learning rate, the batch size, the hidden layer neuron number and the like are adjusted by the methods such as grid search, random search, bayesian optimization and the like so as to improve the model performance.

The training log features are mainly used for main body training of the model, and for the two classification problems of whether the log is abnormal or not, the cross entropy loss function is adopted as an optimization target. Model parameters are adjusted using optimization algorithms such as gradient descent to minimize the loss function. Correspondingly, in the training process, if the loss on the verification set is found not to be obviously reduced, training is stopped in advance, and overfitting is avoided.

Under the condition that the optimized abnormal recognition model accords with the model application condition, the abnormal recognition model can be tested by utilizing the test log characteristics to obtain a test result, and the abnormal recognition model is evaluated based on the test result to obtain an evaluation result. The evaluation result can reflect the performance of the model, so that the model can be determined to be put into practical application. Specifically, the evaluation parameters may include, for example, indexes such as accuracy, recall, and F1 score, so as to measure the performance of the model. The final evaluation result can also be visualized, and the model performance is visualized by using methods such as confusion matrix, ROC curve, PR curve and the like.

The neural network technology is adopted to detect the abnormality of the log, so that the log has strong model expression capability and can capture complex semantic information and potential association. And the model structure is adjusted according to the actual requirements, so that the performance of the model is improved. Through operations such as data set division and super parameter adjustment, generalization capability and accuracy of the model are improved. Meanwhile, the trained model can be rapidly deployed to a production environment, and real-time monitoring and analysis are realized.

S140: and judging whether the target system is abnormal or not based on the abnormal factors.

After determining the abnormality factors according to the abnormality log rule, whether the system is abnormal or not can be judged according to the abnormality factors. Different judgment standards can be set in advance for different abnormal factors, for example, part of abnormal factors can be directly ignored, and part of abnormal factors represent that the system is in fault currently and need to be fed back to operation and maintenance personnel for maintenance in time. Specific setting criteria may be set according to the needs of practical applications, which will not be described herein.

In some embodiments, in the case of determining that the target system is abnormal, system abnormality positioning can be performed based on the abnormality factors to determine a system failure cause, and the system failure cause is sent to a system manager for timely positioning. For example, when the abnormal log is detected, the abnormal information is sent to related personnel in time, such as notification through mail, short message or enterprise communication tool, so that the related personnel repair the system in time according to the detailed information of the abnormal log.

Further, during system operation, log data may change as the system is operated and updated. And periodically using new log data to carry out iterative updating on the model so as to ensure the application effect of the model. Specifically, training efficiency can be improved by methods such as transfer learning, incremental learning and the like, and training time is shortened.

Based on the description of the embodiment and the scene example, it can be seen that the method acquires the log data corresponding to the target system, extracts the log features in the log data, further determines the abnormal factors in the target log features according to the abnormal log rule, and finally determines whether the target system is abnormal by analyzing the abnormal factors. The method can automatically analyze the log to find out the abnormality of the system in time, reduce the risk of system faults and improve the stability of the system. In addition, the system abnormality which is newly generated can be identified through analysis, so that the actual application effect of the method is ensured, the application scene is expanded, and further, the method can be rapidly deployed and monitored in real time according to the requirements of the production environment. The method can be widely applied to software systems of various scales, and has important significance for improving the operation and maintenance efficiency of the system, reducing the operation and maintenance cost and ensuring the stable operation of the system.

Based on the system log-based system abnormality determination method corresponding to fig. 1, a system log-based system abnormality determination device according to an embodiment of the present disclosure is described. The system log-based system abnormality determination device may be provided on a corresponding electronic device. As shown in fig. 2, the system abnormality determination apparatus based on the system log includes the following modules.

The log data acquisition module 210 is configured to acquire target log data corresponding to a target system.

The log feature extraction module 220 is configured to extract a target log feature from the target log data.

An anomaly factor determination module 230 for determining anomaly factors of the target log feature based on anomaly log rules; the abnormal log rule is used for limiting log features corresponding to an abnormal system; the anomaly factors are used for representing the anomaly conditions reflected by the target system based on the log.

An anomaly determination module 240, configured to determine whether the target system is abnormal based on the anomaly factor.

Based on the system log-based system anomaly determination method corresponding to fig. 1, an embodiment of the present disclosure provides an electronic device. The electronic device may include a memory and a processor.

In this embodiment, the memory may be implemented in any suitable manner. For example, the memory may be a read-only memory, a mechanical hard disk, a solid state hard disk, or a usb disk. The memory may be used to store computer program instructions.

In this embodiment, the processor may be implemented in any suitable manner. For example, the processor may take the form of, for example, a microprocessor or processor, and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a programmable logic controller, and an embedded microcontroller, among others. The processor may execute the computer program instructions to implement the system log-based system anomaly determination method in the embodiment corresponding to fig. 1.

The present description provides a computer-readable storage medium having stored thereon computer programs/instructions. The computer readable storage medium may be read by a processor based on an internal bus of a device, and program instructions in the computer readable storage medium are implemented by the processor.

In this embodiment, the computer-readable storage medium may be implemented in any suitable manner. The computer readable storage medium includes, but is not limited to, random access Memory (Random Access Memory, RAM), read-Only Memory (ROM), cache (Cache), hard Disk (HDD), memory Card (Memory Card), and the like. The computer storage medium stores computer program instructions. Program instructions or modules implementing the system log-based system anomaly determination method of the embodiment corresponding to fig. 1 of the present specification when the computer program instructions are executed.

The present description also provides a computer program product comprising a computer program/instructions. The computer program product may be a program written in a corresponding computer program language, stored in a corresponding storage device in a program manner, and transmitted over a computer network. The computer program product may be executable by a processor. In the embodiment of the present disclosure, the computer program product implements the program instructions or modules of the system log-based system anomaly determination method according to the corresponding embodiment of fig. 1 when executed.

It should be noted that the system log-based system anomaly determination method, device and equipment can be applied to the technical field of artificial intelligence, and can also be applied to other technical fields besides the technical field of artificial intelligence, for example, can be fields related to finance, and are not limited thereto.

In addition, the implementation process of the embodiment relates to operations of acquiring, processing, using, storing and the like of the data, which all meet the requirements of relevant national laws and regulations.

While the process flows described above include a plurality of operations occurring in a particular order, it should be apparent that the processes may include more or fewer operations, which may be performed sequentially or in parallel (e.g., using a parallel processor or a multi-threaded environment).

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, read only optical disk read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic tape storage, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the present specification embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present description embodiments may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present embodiments may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The embodiments of the specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments. In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the embodiments of the present specification. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.

The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.

Claims (13)

1. The system exception judging method based on the system log is characterized by comprising the following steps of:

acquiring target log data corresponding to a target system;

extracting target log features from the target log data;

determining an anomaly factor of the target log feature based on an anomaly log rule; the abnormal log rule is used for limiting log features corresponding to an abnormal system; the abnormal factors are used for representing abnormal conditions reflected by the target system based on the log;

and judging whether the target system is abnormal or not based on the abnormal factors.

2. The method of claim 1, wherein prior to extracting target log features from the target log data, further comprising:

preprocessing the target log data; the preprocessing comprises at least one of irrelevant information cleaning, word segmentation and stem extraction.

3. The method of claim 1, wherein the target log features include text features and supplemental features; the extracting the target log features from the target log data includes:

vector conversion is carried out on the target log data to obtain text characteristics;

converting the target log data into numerical data;

extracting supplemental features from the numerical data; the supplemental features include at least one of log level, timestamp, event occurrence frequency, log level, thread information.

4. The method of claim 1, wherein the anomaly log rules correspond to an anomaly identification model; the anomaly identification model is obtained by the following steps:

acquiring sample log data;

extracting features of the sample log data to obtain sample log features;

dividing the sample log features into training log features and verification log features;

training the adjusted abnormal recognition model by utilizing the training log characteristics;

adjusting model types and super parameters of a pre-constructed abnormal recognition model based on verification log features;

and training the model by repeatedly utilizing the training log features, and adjusting the model type and the super-parameters based on the verification log features until the abnormal recognition model accords with the model application conditions.

5. The method of claim 4, wherein training the adjusted anomaly identification model using the training log feature until the anomaly identification model meets a model application condition comprises:

optimizing model parameters of the anomaly identification model by using training log features;

calculating a loss function corresponding to the optimized abnormal recognition model;

judging whether the abnormal recognition model accords with the model application condition or not based on the loss function; the model application condition includes that the decreasing amplitude of the loss function is smaller than a preset optimization threshold.

6. The method of claim 4, wherein the classifying the sample log features into training log features, verification log features, comprises:

dividing the sample log features into training log features, verification log features and test log features;

correspondingly, after optimizing the anomaly identification model based on the verification result, the method further comprises the following steps:

under the condition that the optimized abnormal recognition model accords with the model application condition, testing the abnormal recognition model by using the test log characteristics to obtain a test result;

and evaluating the abnormal recognition model based on the test result to obtain an evaluation result.

7. The method of claim 6, wherein testing the anomaly identification model using test log features to obtain test results comprises:

calculating an evaluation index of the abnormality recognition model; the evaluation index comprises at least one of an accuracy rate, a recall rate and an F1 score.

8. The method of claim 1, wherein after determining whether the target system is abnormal based on the abnormality factor, further comprising:

under the condition that the abnormality of the target system is determined, carrying out system abnormality positioning based on the abnormality factors so as to determine the cause of the system fault;

and sending the system fault reasons to a system manager.

9. The method of claim 1, wherein the target log data comprises real-time log data obtained from real-time detection of a target system.

10. A system abnormality determination apparatus based on a system log, comprising:

the log data acquisition module is used for acquiring target log data corresponding to a target system;

the log feature extraction module is used for extracting target log features from the target log data;

the abnormal factor determining module is used for determining abnormal factors of the target log characteristics based on an abnormal log rule; the abnormal log rule is used for limiting log features corresponding to an abnormal system; the abnormal factors are used for representing abnormal conditions reflected by the target system based on the log;

and the abnormality judging module is used for judging whether the target system is abnormal or not based on the abnormality factors.

11. An electronic device includes a memory and a processor; wherein the memory is for storing computer programs/instructions; the processor for executing the computer program/instructions to implement the steps of the method according to any of claims 1-9.

12. A computer storage medium having stored thereon a computer program/instruction which, when executed, is adapted to carry out the steps of the method according to any of claims 1-9.

13. A computer program product comprising computer programs/instructions which, when executed, are adapted to carry out the steps of the method according to any one of claims 1-9.