CN116185751A - Abnormality log recognition method, abnormality log recognition device, electronic equipment and storage medium - Google Patents
Abnormality log recognition method, abnormality log recognition device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116185751A CN116185751A CN202111420323.2A CN202111420323A CN116185751A CN 116185751 A CN116185751 A CN 116185751A CN 202111420323 A CN202111420323 A CN 202111420323A CN 116185751 A CN116185751 A CN 116185751A
- Authority
- CN
- China
- Prior art keywords
- log
- log data
- abnormal
- recognition
- identification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Test And Diagnosis Of Digital Computers (AREA)
Abstract
The method comprises the steps of obtaining log data corresponding to a service to be identified, preprocessing the log data according to a pre-stored preprocessing rule to obtain structured log data, inputting the structured log data into a pre-trained abnormal log identification model for identification, obtaining an abnormal log identification result, and sending the abnormal log identification result to terminal equipment for display.
Description
Technical Field
The present disclosure relates to the field of big data technologies, and in particular, to an anomaly log identification method, an anomaly log identification device, an electronic device, and a storage medium.
Background
With the continuous expansion of the application scenes of the big data clusters, more and more business scenes are deployed with the big data clusters. In the running process of the big data cluster, the abnormal running condition may occur, so that an abnormal log can be obtained from the log generated in the running process of the big data cluster, and then the abnormal information is positioned according to the abnormal log.
In the prior art, when the log is collected, a log collection request is usually sent to a log server through a log processing terminal, the log server searches the log to be collected according to the content of the log collection request, and the log is collected in a mode of sending the collected log file to the log processing terminal, after the log processing terminal receives the log, the log can be displayed according to the receiving sequence of the log, and then the abnormal log is determined in a manual mode.
However, the abnormal logs are determined in a manual mode, the operation and maintenance personnel are required to check the logs one by one, then the abnormal logs are screened out from the logs, the subjectivity of the abnormal log determination is strong too depending on the working experience of the operation and maintenance personnel, the accuracy of the abnormal log determination is reduced, meanwhile, the manpower consumption is increased, the efficiency of the abnormal log determination is reduced, and the normal operation of the service is further affected.
Disclosure of Invention
The application provides an anomaly log identification method, an anomaly log identification device, electronic equipment and a storage medium, so that the efficiency and the accuracy of anomaly log determination are improved.
In a first aspect, the present application provides an anomaly log identifying method, including:
acquiring log data corresponding to a service to be identified;
preprocessing the log data according to a pre-stored preprocessing rule to obtain structured log data;
inputting the structured log data into a pre-trained abnormal log recognition model for recognition to obtain an abnormal log recognition result;
and sending the abnormal log identification result to terminal equipment for display.
Optionally, after the structured log data is input into the pre-trained abnormal log recognition model to be recognized, the method further includes:
and inputting the abnormal log recognition result into a pre-trained emotion recognition model for recognition to obtain an emotion recognition result.
Optionally, the log data is audit log data and/or operation log data, and the step of inputting the abnormal log recognition result into a pre-trained emotion recognition model for recognition to obtain an emotion recognition result includes:
if the abnormal log recognition result is that the number of the abnormal log data exceeds a preset number threshold, inputting the abnormal log recognition result into a pre-trained emotion recognition model for recognition to obtain the abnormal operator emotion recognition result corresponding to the audit log data and/or the operation log data;
if the abnormal log identification result is that the number of the abnormal log data does not exceed the preset number threshold, inputting the abnormal log identification result into a pre-trained emotion identification model for identification, and obtaining the normal operator emotion identification result corresponding to the audit log data and/or the operation log data.
Optionally, the preprocessing the log data according to a pre-stored preprocessing rule to obtain structured log data includes:
and converting the log data of different types into structured log data containing table identifiers according to pre-stored preprocessing rules.
Optionally, before the obtaining the log data corresponding to the service to be identified, the method further includes:
acquiring training log data;
preprocessing the training log data according to the preprocessing rule to obtain structured training log data;
inputting the structured training log data into a first network model for training to obtain an initial abnormal log identification model;
acquiring test log data, and inputting the test log data into an initial abnormal log identification model for identification to obtain a test abnormal log identification result;
judging whether the accuracy rate of the abnormal log identification in the test abnormal log identification result exceeds a preset accuracy rate threshold value;
and if the accuracy rate of the abnormal log identification exceeds the accuracy rate threshold, determining the initial abnormal log identification model as an abnormal log identification model.
Optionally, the method further comprises:
if the accuracy rate of the abnormal log identification does not exceed the accuracy rate threshold, adjusting parameters of the initial abnormal log identification model;
and inputting the training log data into the adjusted initial abnormal log recognition model for training until the accuracy of the adjusted initial abnormal log recognition model exceeds the accuracy threshold.
Optionally, after preprocessing the training log data according to the preprocessing rule to obtain structured training log data, the method further includes:
carrying out emotion marking on the structured training log data according to the operation time, the connection address, the connection user, the operation type, the operation content and the operation frequency corresponding to the training log data to obtain marked structured training log data;
and inputting the marked structured training log data into a second network model for training to obtain an emotion recognition model.
In a second aspect, the present application provides an anomaly log identifying device, including:
the acquisition module is used for acquiring log data corresponding to the service to be identified;
the processing module is used for preprocessing the log data according to a pre-stored preprocessing rule to obtain structured log data;
the processing module is further used for inputting the structured log data into a pre-trained abnormal log recognition model for recognition to obtain an abnormal log recognition result;
the processing module is further used for sending the abnormal log identification result to a terminal device for display.
In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively coupled to the processor;
the memory stores computer-executable instructions;
the processor executes computer-executable instructions stored in the memory to implement the anomaly log identification method of any one of the first aspect.
In a fourth aspect, the present application provides a computer-readable storage medium having stored therein computer-executable instructions which, when executed by a processor, implement the anomaly log identification method of any one of the first aspects.
After the scheme is adopted, the log data corresponding to the service to be identified can be acquired first, then the log data is preprocessed according to the pre-stored preprocessing rule to obtain structured log data, after the structured log data is obtained, the structured log data can be input into a pre-trained abnormal log identification model to be identified, an abnormal log identification result is obtained, the abnormal log identification result is sent to terminal equipment to be displayed, the log data is automatically identified through the pre-trained abnormal log identification model, and the mode of obtaining the abnormal log data is improved, so that the objectivity and accuracy of abnormal log determination are improved, meanwhile, the abnormal log determination efficiency is improved, and further the normal operation of the service is guaranteed.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive faculty for a person skilled in the art.
Fig. 1 is a schematic architecture diagram of an application system of an anomaly log identification method provided in an embodiment of the present application;
fig. 2 is a flow chart of an anomaly log identification method provided in an embodiment of the present application;
FIG. 3 is a schematic diagram of an application of the model training process provided in the embodiments of the present application;
FIG. 4 is a flowchart illustrating an anomaly log identification method according to another embodiment of the present application;
fig. 5 is a schematic structural diagram of an abnormality log identifying device according to an embodiment of the present application;
fig. 6 is a schematic hardware structure of an electronic device according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
The terms "first," "second," "third," "fourth" and the like in the description and in the claims of this application and in the above-described figures, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the present application described herein may be capable of including other sequential examples in addition to those illustrated or described. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The existing big data cluster and Kubernetes fault checking and other functions are realized through log collection, when the logs are collected, the logs are simply collected together through some open source tools, for example, a log collection request is usually sent to a log server through a log processing terminal, the log server searches the logs needing to be collected according to the content of the log collection request and sends collected log files to the log processing terminal, the log processing terminal can display the logs according to the receiving sequence of the logs after receiving the logs, and then abnormal logs are determined through a manual mode. However, the abnormal logs are determined manually, the operation and maintenance personnel are required to screen the abnormal logs one by one from the collected logs, the same log possibly appears, some operation and maintenance personnel determine the abnormal logs, and some operation and maintenance personnel determine the normal logs, so that the subjectivity of the determination of the abnormal logs is strong due to too much dependence on the working experience of the operation and maintenance personnel, the accuracy of the determination of the abnormal logs is reduced, meanwhile, the manpower consumption is increased, the determination efficiency of the abnormal logs is reduced, and the normal operation of the business is further influenced.
Based on the technical problems, the method and the device automatically identify the log data through the pre-trained abnormal log identification model to obtain the abnormal log data, so that objectivity and accuracy of abnormal log determination are improved, meanwhile, efficiency of abnormal log determination is improved, and further, the technical effect of normal operation of a service is guaranteed.
Fig. 1 is a schematic architecture diagram of an application system of an abnormal log identifying method provided in an embodiment of the present application, as shown in fig. 1, in the application system, the application system may include a server 101, a terminal device 102 and a database 103, where the database 103 may store log data of different services, and the server 101 may obtain log data corresponding to a service to be processed from the database 103, then further process the log data corresponding to the service to be processed, and finally obtain an abnormal log identifying result, and send the abnormal log identifying result to the terminal device 102 for displaying.
In addition, log data corresponding to the service to be processed can also be generated in real time in the running process of the service to be processed.
The terminal device 102 may be a smart phone, a tablet, a personal computer, a wearable smart device, a display, or other devices capable of displaying the recognition result of the abnormal log.
The technical scheme of the present application is described in detail below with specific examples. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.
Fig. 2 is a flowchart of an anomaly log recognition method according to an embodiment of the present application, where the method of the present embodiment may be executed by the server 101. As shown in fig. 2, the method of the present embodiment may include:
s201: and acquiring log data corresponding to the service to be identified.
In this embodiment, logs of the big data peripheral component have certain normalization and relevance, that is, each line of logs is generated by output statements of source codes, during the running process of the big data component program, the logs printed by the bottom one source code are of the same type, and the codes are called as log keys. One log often depends on the previous log or the previous logs, the association degree of adjacent or similar logs is high, and when the logic is broken by a certain log, the log execution is abnormal, so that whether the operation of a business or the operation of equipment is abnormal or not can be determined through log data.
The log data may be audit log data, operation log data, etc.
S202: preprocessing the log data according to a pre-stored preprocessing rule to obtain structured log data.
In this embodiment, the log data may include parameters of different dimensions, and illustratively, the log data may include time information and level information, so that in order to improve accuracy of identification, the log data may be preprocessed according to a pre-stored preprocessing rule to obtain structured log data.
Further, preprocessing the log data according to a pre-stored preprocessing rule to obtain structured log data, which specifically may include:
and converting the log data of different types into structured log data containing table identifiers according to pre-stored preprocessing rules.
In particular, log data may be represented uniformly in the form of a data table, and thus structured log data may be log data that contains table identifications that represent specific parameters in the log data.
In addition, preprocessing of log data can be achieved in an existing manner. By way of example, preprocessing of log data may be implemented by logParser.
S203: and inputting the structured log data into a pre-trained abnormal log recognition model for recognition to obtain an abnormal log recognition result.
In this embodiment, after the structured log data is obtained, the structured log data may be input into an anomaly log recognition model trained in advance to recognize an anomaly log, so as to obtain an anomaly log recognition result.
The anomaly log recognition result may include anomaly log data, and the number of the anomaly log data may be 0 (i.e., no anomaly log data), or may be 1 or more.
S204: and sending the abnormal log identification result to the terminal equipment for display.
In this embodiment, after the abnormal log identification result is obtained, in order to facilitate the operation and maintenance personnel to know the abnormal log identification situation in time, the abnormal log identification result may be sent to the terminal device for display.
After the scheme is adopted, the log data corresponding to the service to be identified can be firstly obtained, then the log data is preprocessed according to the pre-stored preprocessing rule to obtain the structured log data, after the structured log data is obtained, the structured log data can be input into the pre-trained abnormal log identification model to be identified, the abnormal log identification result is obtained, then the abnormal log identification result is sent to the terminal equipment to be displayed, the log data is automatically identified on the basis of the pre-trained abnormal log identification model, the abnormal log data is obtained, the objectivity and the accuracy of abnormal log determination are improved, the abnormal log determination efficiency is improved, and the normal operation of the service is further ensured.
The examples of the present specification also provide some specific embodiments of the method based on the method of fig. 2, which is described below.
Furthermore, in another embodiment, after the structured log data is input into the pre-trained abnormal log recognition model to be recognized, and an abnormal log recognition result is obtained, the method may further include:
and inputting the abnormal log recognition result into a pre-trained emotion recognition model for recognition to obtain an emotion recognition result.
In this embodiment, in the cluster management process, for most of the problems, operation and processing of the clusters are required by operation and maintenance personnel, which usually have special rights. If the operation and maintenance personnel have cluster management authority and data operation authority. Therefore, the actions and operations of the operation and maintenance personnel can greatly influence the clusters and the services running in the clusters, the simple authority management cannot avoid the occurrence of some intentional or unintentional operations, and the simple formulation process is too complicated, so that some intelligent judgment is needed to judge which operation actions need interception and reminding. In addition, some operation behaviors of operators are sometimes related to character behavior factors of people, so that the character behaviors of related operators also need to be considered.
Therefore, after the abnormal log recognition result is obtained, the abnormal log recognition result is input into a pre-trained emotion recognition model for recognition, and the emotion recognition result is obtained.
Further, the log data is audit log data and/or operation log data, and the step of inputting the abnormal log recognition result into a pre-trained emotion recognition model for recognition to obtain an emotion recognition result may specifically include:
if the abnormal log recognition result is that the number of the abnormal log data exceeds a preset number threshold, inputting the abnormal log recognition result into a pre-trained emotion recognition model for recognition to obtain audit log data and/or operator emotion abnormal recognition results corresponding to the operation log data.
If the abnormal log recognition result is that the number of the abnormal log data does not exceed the preset number threshold, the abnormal log recognition result is input into a pre-trained emotion recognition model for recognition, and an operator emotion normal recognition result corresponding to the audit log data and/or the operation log data is obtained.
Specifically, the preset number threshold may be set according to the actual application scenario in a user-defined manner, and exemplary, the preset number threshold may be zero. If an abnormal log exists, audit log data and/or an operator emotion abnormal recognition result corresponding to the operation log data can be obtained, and the operator needs to pay additional attention to avoid that the operator executes some wrong operations to influence the normal operation of the service.
In addition, in another embodiment, before acquiring the log data corresponding to the service to be identified, the method may further include:
training log data is obtained.
And preprocessing the training log data according to the preprocessing rule to obtain structured training log data.
And inputting the structured training log data into a first network model for training to obtain an initial abnormal log identification model.
And acquiring test log data, and inputting the test log data into an initial abnormal log identification model for identification to obtain a test abnormal log identification result.
Judging whether the accuracy rate of the abnormal log identification in the test abnormal log identification result exceeds a preset accuracy rate threshold value.
And if the accuracy rate of the abnormal log identification exceeds the accuracy rate threshold, determining the initial abnormal log identification model as an abnormal log identification model.
In this embodiment, the first network model may be an LSTM (Long Short-Term Memory) model. In training based on structured training log data, specific may include: semantic recognition, namely, all possible log displays contained in the current log need to be numbered and classified, then encoded into template vectors, and a structured log data set is generated by extracting the template vectors and converting the template sequences into semantic vector sequences. And then designing template vectors, substituting the template vectors into a training set, generating sequence vectors by using a sliding window sampling method, counting the vectors and semantic vectors, and selecting a feature vector combination to be used for substituting the feature vector combination into the LSTM to train and obtain an initial abnormal log recognition model. And then testing the stored initial abnormal log recognition model by using a test set, namely sorting the output of the model according to the size of the probability value, taking out g log templates with the maximum probability value, if the newly generated log template m is in the g log templates, considering m as a normal log, otherwise considering m as an abnormal log, determining the accuracy of the initial abnormal log recognition model according to the test result, and if the accuracy of abnormal log recognition exceeds an accuracy threshold, determining the initial abnormal log recognition model as an abnormal log recognition model.
The accuracy threshold can be set according to the actual application scene customization, and specifically, can be any value between 85% and 95%.
Furthermore, the method may further comprise:
and if the accuracy rate of the abnormal log identification does not exceed the accuracy rate threshold, adjusting parameters of the initial abnormal log identification model.
And inputting the training log data into the adjusted initial abnormal log recognition model for training until the accuracy of the adjusted initial abnormal log recognition model exceeds the accuracy threshold.
Furthermore, in another embodiment, after preprocessing the training log data according to the preprocessing rule to obtain structured training log data, the method may further include:
and carrying out emotion marking on the structured training log data according to the operation time, the connection address, the connection user, the operation type, the operation content and the operation frequency corresponding to the training log data, and obtaining the marked structured training log data.
And inputting the marked structured training log data into a second network model for training to obtain an emotion recognition model.
In this embodiment, emotion marking training may be performed on the existing log data through the second network model, and an emotion recognition model is constructed. Then substituting the obtained abnormal log into the emotion recognition model to obtain emotion of the user at the moment or obtain character of the operator. And further, whether the character of the operator is suitable for the post or whether the operator is under a frightened, impatient, irritable or error-prone emotion at the moment can be obtained, so that the operator is reminded to manage the emotion of the operator in time. The second network model may be a BERT (Bidirectional Encoder Representations from Transformer, depth bi-directional representation) model, among others.
Fig. 3 is an application schematic diagram of a model training process provided in an embodiment of the present application, as shown in fig. 3, in this embodiment, log data may be obtained first, where the log data is audit log data and operation log data, and the log data is stored, and after the log data is stored to a certain magnitude, preparation may be made for training recognition of subsequent classification. The log data can be firstly analyzed and converted into a structured log data set, and character and emotion marking is carried out by using the BERT model. The emotion states of operators at the moment are considered from various aspects such as operation time, connection address, connection user, operation type, operation content, operation frequency and the like. The recognition problem of the log is also a multi-classification problem, namely the recognition problem can be converted into an algorithm model problem, and the generation of the model needs a large amount of data support, so that the log needs to be analyzed firstly and is converted into a structured log data set, and then the BERT model and the LSTM model are trained based on the structured log data set to obtain an abnormal log recognition model and an abnormal emotion recognition model.
In addition, fig. 4 is a flow chart of an abnormal log identifying method according to another embodiment of the present application, as shown in fig. 4, in this embodiment, log data may be collected first, then an abnormal log identifying model and an emotion identifying model obtained by training the log data may be used, and the two models may be applied respectively to obtain emotion output of an operation and maintenance person.
Based on the same idea, the embodiment of the present disclosure further provides a device corresponding to the method, and fig. 5 is a schematic structural diagram of an anomaly log identifying device provided in the embodiment of the present disclosure, where, as shown in fig. 5, the device provided in the embodiment may include:
the obtaining module 501 is configured to obtain log data corresponding to a service to be identified.
The processing module 502 is configured to preprocess the log data according to a pre-stored preprocessing rule, so as to obtain structured log data.
In this embodiment, the processing module 502 is further configured to:
and converting the log data of different types into structured log data containing table identifiers according to pre-stored preprocessing rules.
The processing module 502 is further configured to input the structured log data into a pre-trained abnormal log recognition model for recognition, so as to obtain an abnormal log recognition result.
The processing module 502 is further configured to send the abnormal log identification result to a terminal device for display.
Furthermore, in another embodiment, the processing module 502 is further configured to:
and inputting the abnormal log recognition result into a pre-trained emotion recognition model for recognition to obtain an emotion recognition result.
In this embodiment, the log data is audit log data and/or operation log data, and the processing module 502 is further configured to:
if the abnormal log recognition result is that the number of the abnormal log data exceeds a preset number threshold, inputting the abnormal log recognition result into a pre-trained emotion recognition model for recognition, and obtaining an operator emotion abnormal recognition result corresponding to the audit log data and/or the operation log data.
If the abnormal log identification result is that the number of the abnormal log data does not exceed the preset number threshold, inputting the abnormal log identification result into a pre-trained emotion identification model for identification, and obtaining the normal operator emotion identification result corresponding to the audit log data and/or the operation log data.
Furthermore, in another embodiment, the processing module 502 is further configured to:
training log data is obtained.
And preprocessing the training log data according to the preprocessing rule to obtain structured training log data.
And inputting the structured training log data into a first network model for training to obtain an initial abnormal log identification model.
And acquiring test log data, and inputting the test log data into an initial abnormal log identification model for identification to obtain a test abnormal log identification result.
Judging whether the accuracy rate of the abnormal log identification in the test abnormal log identification result exceeds a preset accuracy rate threshold value.
And if the accuracy rate of the abnormal log identification exceeds the accuracy rate threshold, determining the initial abnormal log identification model as an abnormal log identification model.
In addition, the processing module 502 is further configured to:
and if the accuracy rate of the abnormal log identification does not exceed the accuracy rate threshold, adjusting parameters of the initial abnormal log identification model.
And inputting the training log data into the adjusted initial abnormal log recognition model for training until the accuracy of the adjusted initial abnormal log recognition model exceeds the accuracy threshold.
In addition, the processing module 502 is further configured to:
and carrying out emotion marking on the structured training log data according to the operation time, the connection address, the connection user, the operation type, the operation content and the operation frequency corresponding to the training log data, and obtaining the marked structured training log data.
And inputting the marked structured training log data into a second network model for training to obtain an emotion recognition model.
The device provided in the embodiment of the present application may implement the method of the embodiment shown in fig. 2, and its implementation principle and technical effects are similar, and are not described herein again.
Fig. 6 is a schematic hardware structure of an electronic device provided in an embodiment of the present application, as shown in fig. 6, an apparatus 600 provided in the embodiment includes: a processor 601, and a memory communicatively coupled to the processor. The processor 601 and the memory 602 are connected by a bus 603.
In a specific implementation process, the processor 601 executes the computer-executed instructions stored in the memory 602, so that the processor 601 executes the method for identifying an abnormal log in the above method embodiment.
The specific implementation process of the processor 601 may refer to the above-mentioned method embodiment, and its implementation principle and technical effects are similar, and this embodiment will not be described herein again.
In the embodiment shown in fig. 6, it should be understood that the processor may be a central processing unit (english: central Processing Unit, abbreviated as CPU), or may be other general purpose processors, digital signal processors (english: digital Signal Processor, abbreviated as DSP), application specific integrated circuits (english: application Specific Integrated Circuit, abbreviated as ASIC), or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present invention may be embodied directly in a hardware processor for execution, or in a combination of hardware and software modules in a processor for execution.
The memory may comprise high speed RAM memory or may further comprise non-volatile storage NVM, such as at least one disk memory.
The bus may be an industry standard architecture (Industry Standard Architecture, ISA) bus, an external device interconnect (Peripheral Component Interconnect, PCI) bus, or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, among others. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, the buses in the drawings of the present application are not limited to only one bus or one type of bus.
The embodiment of the application also provides a computer readable storage medium, wherein computer execution instructions are stored in the computer readable storage medium, and when a processor executes the computer execution instructions, the method for identifying the abnormal log in the embodiment of the method is realized.
The embodiment of the application also provides a computer program product, which comprises a computer program, wherein the computer program realizes the abnormal log identification method when being executed by a processor.
The computer readable storage medium described above may be implemented by any type of volatile or non-volatile memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. A readable storage medium can be any available medium that can be accessed by a general purpose or special purpose computer.
An exemplary readable storage medium is coupled to the processor such the processor can read information from, and write information to, the readable storage medium. In the alternative, the readable storage medium may be integral to the processor. The processor and the readable storage medium may reside in an application specific integrated circuit (Application Specific Integrated Circuits, ASIC for short). The processor and the readable storage medium may reside as discrete components in a device.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the method embodiments described above may be performed by hardware associated with program instructions. The foregoing program may be stored in a computer readable storage medium. The program, when executed, performs steps including the method embodiments described above; and the aforementioned storage medium includes: various media that can store program code, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.
Claims (10)
1. An anomaly log recognition method, comprising:
acquiring log data corresponding to a service to be identified;
preprocessing the log data according to a pre-stored preprocessing rule to obtain structured log data;
inputting the structured log data into a pre-trained abnormal log recognition model for recognition to obtain an abnormal log recognition result;
and sending the abnormal log identification result to terminal equipment for display.
2. The method according to claim 1, further comprising, after the step of inputting the structured log data into a pre-trained abnormal log recognition model to perform recognition, obtaining an abnormal log recognition result:
and inputting the abnormal log recognition result into a pre-trained emotion recognition model for recognition to obtain an emotion recognition result.
3. The method according to claim 2, wherein the log data is audit log data and/or operation log data, the step of inputting the abnormal log recognition result into a pre-trained emotion recognition model for recognition to obtain an emotion recognition result includes:
if the abnormal log recognition result is that the number of the abnormal log data exceeds a preset number threshold, inputting the abnormal log recognition result into a pre-trained emotion recognition model for recognition to obtain the abnormal operator emotion recognition result corresponding to the audit log data and/or the operation log data;
if the abnormal log identification result is that the number of the abnormal log data does not exceed the preset number threshold, inputting the abnormal log identification result into a pre-trained emotion identification model for identification, and obtaining the normal operator emotion identification result corresponding to the audit log data and/or the operation log data.
4. A method according to any one of claims 1-3, wherein preprocessing the log data according to pre-stored preprocessing rules to obtain structured log data comprises:
and converting the log data of different types into structured log data containing table identifiers according to pre-stored preprocessing rules.
5. A method according to claim 2 or 3, further comprising, prior to said obtaining log data corresponding to the service to be identified:
acquiring training log data;
preprocessing the training log data according to the preprocessing rule to obtain structured training log data;
inputting the structured training log data into a first network model for training to obtain an initial abnormal log identification model;
acquiring test log data, and inputting the test log data into an initial abnormal log identification model for identification to obtain a test abnormal log identification result;
judging whether the accuracy rate of the abnormal log identification in the test abnormal log identification result exceeds a preset accuracy rate threshold value;
and if the accuracy rate of the abnormal log identification exceeds the accuracy rate threshold, determining the initial abnormal log identification model as an abnormal log identification model.
6. The method as recited in claim 5, further comprising:
if the accuracy rate of the abnormal log identification does not exceed the accuracy rate threshold, adjusting parameters of the initial abnormal log identification model;
and inputting the training log data into the adjusted initial abnormal log recognition model for training until the accuracy of the adjusted initial abnormal log recognition model exceeds the accuracy threshold.
7. The method of claim 5, further comprising, after the preprocessing the training log data according to the preprocessing rule to obtain structured training log data:
carrying out emotion marking on the structured training log data according to the operation time, the connection address, the connection user, the operation type, the operation content and the operation frequency corresponding to the training log data to obtain marked structured training log data;
and inputting the marked structured training log data into a second network model for training to obtain an emotion recognition model.
8. An abnormality log identifying apparatus, comprising:
the acquisition module is used for acquiring log data corresponding to the service to be identified;
the processing module is used for preprocessing the log data according to a pre-stored preprocessing rule to obtain structured log data;
the processing module is further used for inputting the structured log data into a pre-trained abnormal log recognition model for recognition to obtain an abnormal log recognition result;
the processing module is further used for sending the abnormal log identification result to a terminal device for display.
9. An electronic device, comprising: a processor, and a memory communicatively coupled to the processor;
the memory stores computer-executable instructions;
the processor executes computer-executable instructions stored in the memory to implement the anomaly log identification method of any one of claims 1 to 7.
10. A computer-readable storage medium having stored therein computer-executable instructions that, when executed by a processor, implement the anomaly log identification method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111420323.2A CN116185751A (en) | 2021-11-26 | 2021-11-26 | Abnormality log recognition method, abnormality log recognition device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111420323.2A CN116185751A (en) | 2021-11-26 | 2021-11-26 | Abnormality log recognition method, abnormality log recognition device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116185751A true CN116185751A (en) | 2023-05-30 |
Family
ID=86440782
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111420323.2A Pending CN116185751A (en) | 2021-11-26 | 2021-11-26 | Abnormality log recognition method, abnormality log recognition device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116185751A (en) |
-
2021
- 2021-11-26 CN CN202111420323.2A patent/CN116185751A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111460250B (en) | Image data cleaning method, image data cleaning device, image data cleaning medium, and electronic apparatus | |
| CN109872162B (en) | Wind control classification and identification method and system for processing user complaint information | |
| CN112560453B (en) | Voice information verification method and device, electronic equipment and medium | |
| CN113450147B (en) | Product matching method, device, equipment and storage medium based on decision tree | |
| CN112416778B (en) | Test case recommendation method and device and electronic equipment | |
| CN109684157A (en) | Alarm method, equipment, storage medium and device based on the log that reports an error | |
| CN111708938A (en) | Method, apparatus, electronic device, and storage medium for information processing | |
| CN111813593A (en) | Data processing method, equipment, server and storage medium | |
| CN109101487A (en) | Conversational character differentiating method, device, terminal device and storage medium | |
| CN115618415A (en) | Sensitive data identification method and device, electronic equipment and storage medium | |
| CN116383742A (en) | Rule chain setting processing method, system and medium based on feature classification | |
| CN112579781B (en) | Text classification method, device, electronic equipment and medium | |
| CN113962670A (en) | File approval method and equipment of file approval system and storage medium | |
| CN113628043A (en) | Complaint validity judgment method, device, equipment and medium based on data classification | |
| CN109491970B (en) | Bad picture detection method and device for cloud storage and storage medium | |
| CN116185751A (en) | Abnormality log recognition method, abnormality log recognition device, electronic equipment and storage medium | |
| CN115688107A (en) | Fraud-related APP detection system and method | |
| CN118277560A (en) | Text classification method and device for short messages | |
| CN115544566A (en) | Log desensitization method, device, equipment and storage medium | |
| CN113449506A (en) | Data detection method, device and equipment and readable storage medium | |
| CN112465149A (en) | Same-city part identification method and device, electronic equipment and storage medium | |
| CN114189585A (en) | Crank call abnormity detection method and device and computing equipment | |
| CN111027296A (en) | Report generation method and system based on knowledge base | |
| CN114707042B (en) | Application software classification method and device, electronic equipment and readable storage medium | |
| CN114548825B (en) | Complaint work order distortion detection method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |