CN117668541A

CN117668541A - Log abnormality detection method and device, electronic equipment and storage medium

Info

Publication number: CN117668541A
Application number: CN202311616797.3A
Authority: CN
Inventors: 吴得全
Original assignee: China Mobile Communications Group Co Ltd; China Mobile Suzhou Software Technology Co Ltd
Current assignee: China Mobile Communications Group Co Ltd; China Mobile Suzhou Software Technology Co Ltd
Priority date: 2023-11-29
Filing date: 2023-11-29
Publication date: 2024-03-08

Abstract

The invention discloses a detection method and device for log abnormality, electronic equipment and a storage medium, and relates to the technical field of data processing.

Description

Log abnormality detection method and device, electronic equipment and storage medium

Technical Field

The disclosure relates to the technical field of data processing, and in particular relates to a method and a device for detecting log abnormality, electronic equipment and a storage medium.

Background

In the era of cloud computing, system modules are gradually increased, inter-project service coupling and calling are increasingly complex, and a large-scale distributed system becomes a core technology of a cloud computing platform, so that generation of massive system-level logs is driven.

Most of existing log anomaly detection modes are detection modes of manually matching regular expressions and searching keywords by developers, and in the face of such massive log scales, the traditional log anomaly detection technology is more and more worry, and in the face of log data with low correlation, the detection efficiency is lower.

Disclosure of Invention

The disclosure provides a log abnormality detection method, a log abnormality detection device, electronic equipment and a storage medium. The method mainly aims to solve the problem of low detection efficiency of log anomaly detection.

According to a first aspect of the present disclosure, there is provided a method for detecting log anomalies, including:

dividing the log data to be detected according to a sliding window to obtain each information block;

determining the dispersion degree of the feature words in each information block according to a preset algorithm, screening according to the dispersion degree of each feature word, and reserving a preset number of target feature words;

inputting the reserved target feature words into a pre-training neural network model for clustering to obtain a clustering result of the log data to be detected;

and determining whether the clustering result of the log data to be detected is abnormal or not according to a preset clustering distance threshold value.

Optionally, the pre-training neural network model includes a self-coding network and a clustering algorithm, and the inputting the retained target feature words into the pre-training neural network model for clustering, to obtain the clustering result of the log data to be detected includes:

inputting the target feature words into the self-coding network for coding and decoding to obtain decoded target feature words;

and clustering the decoded target feature words according to the clustering algorithm, and determining a clustering result of the log to be detected.

Optionally, the determining the dispersion degree of the feature words in each information block according to a preset algorithm, and the screening according to the dispersion degree of each feature word further includes:

calculating the dispersion degree of each characteristic word in each information block based on a preset word frequency algorithm;

and sequencing according to the dispersion degree of each feature word, and determining a preset number of feature words as the target feature words according to the sequence.

Optionally, before the retained target feature words are input into a pre-training neural network model to be clustered, and a clustering result of the log data to be detected is obtained, the method further includes:

Inputting training log data into an encoder in the self-encoding network, and reconstructing the training log data according to the encoder and a decoder in the self-encoding network; the training log data are processed according to the preset algorithm;

calculating an error between the training log data after reconstruction and the training log data before reconstruction;

and adjusting network parameters of the self-encoder network according to the errors, and repeating the training step of the self-encoder network model until the error value between the reconstructed training log data and the training log data before reconstruction is smaller than a preset error threshold value, and stopping training the self-encoder network.

Optionally, the clustering the decoded target feature words according to the clustering algorithm, and determining the clustering result of the log to be detected includes:

according to the clustering algorithm, respectively calculating a clustering result of each target feature word; wherein the clustering algorithm comprises a preset number of clustering categories;

determining a clustering result of the log data to be detected according to the clustering result of each feature word and the application scene of the log data to be detected;

Or determining the clustering result of the log data to be detected according to the clustering result of each characteristic word and the importance degree of each characteristic word; wherein, the importance level of the feature words is determined according to the discrete level.

According to a second aspect of the present disclosure, there is provided a log abnormality detection apparatus including:

the segmentation unit is used for segmenting the log data to be detected according to a sliding window to obtain each information block;

the first determining unit is used for determining the dispersion degree of the feature words in each information block according to a preset algorithm, screening according to the dispersion degree of each feature word and reserving a preset number of target feature words;

the clustering unit is used for inputting the reserved target feature words into a pre-training neural network model for clustering to obtain a clustering result of the log data to be detected;

and the second determining unit is used for determining whether the clustering result of the log data to be detected is abnormal or not according to a preset clustering distance threshold value.

Optionally, the pre-training neural network model includes a self-coding network and a clustering algorithm, and the clustering unit is further configured to:

Optionally, the first determining unit is further configured to:

Optionally, the apparatus further includes:

the reconstruction unit is used for inputting training log data into an encoder in the self-coding network before the clustering unit inputs the reserved target feature words into a pre-training neural network model for clustering to obtain a clustering result of the log data to be detected, and reconstructing the training log data according to the encoder and the decoder in the self-coding network; the training log data are processed according to the preset algorithm;

a calculation unit configured to calculate an error between the training log data after reconstruction and the training log data before reconstruction;

and the adjusting unit is used for adjusting the network parameters of the self-encoder network according to the errors, and repeating the training steps of the self-encoder network model until the error value between the reconstructed training log data and the training log data before reconstruction is smaller than a preset error threshold value, and stopping training the self-encoder network.

Optionally, the second determining unit is further configured to:

According to a third aspect of the present disclosure, there is provided an electronic device comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of the first aspect.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of the preceding first aspect.

According to a fifth aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method of the first aspect described above.

The method, the device, the electronic equipment and the storage medium for detecting log abnormality provided by the disclosure have the main technical scheme that: dividing the log data to be detected according to a sliding window to obtain each information block; determining the dispersion degree of the feature words in each information block according to a preset algorithm, screening according to the dispersion degree of each feature word, and reserving a preset number of target feature words; inputting the reserved target feature words into a pre-training neural network model for clustering to obtain a clustering result of the log data to be detected; and determining whether the clustering result of the log data to be detected is abnormal or not according to a preset clustering distance threshold value. Compared with the related art, the method and the device have the advantages that by means of adding the data preprocessing mode, the sliding window is introduced in the log data feature extraction stage to calculate the dispersion degree of the feature words in the log data in combination with the preset algorithm, structured log information is output as the feature vector of the limited block, extraction of the feature words can be achieved when log data with low correlation are faced, meanwhile, the problem that the feature word dimension is extremely high due to the fact that the log feature value faces large data quantity is achieved, dimension reduction is conducted on the data dimension of the feature words, and the detection speed of log abnormality is improved.

It should be understood that the description of this section is not intended to identify key or critical features of the embodiments of the application or to delineate the scope of the application. Other features of the present application will become apparent from the description that follows.

Drawings

The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

fig. 1 is a flow chart of a method for detecting log anomalies according to an embodiment of the present disclosure;

fig. 2 is a flow chart of a method for detecting log anomalies according to an embodiment of the present disclosure;

fig. 3 is a schematic structural diagram of a log abnormality detection device according to an embodiment of the present disclosure;

fig. 4 is a schematic structural diagram of a log abnormality detection device according to an embodiment of the present disclosure;

fig. 5 is a schematic block diagram of an example electronic device provided by an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

The following describes a method, an apparatus, an electronic device, and a storage medium for detecting log abnormality of an embodiment of the present disclosure with reference to the accompanying drawings.

Fig. 1 is a flowchart of a method for detecting log anomalies according to an embodiment of the present disclosure.

As shown in fig. 1, the method comprises the steps of:

and step 101, dividing the log data to be detected according to a sliding window to obtain each information block.

The sliding window is a common data processing technology, and can be used for dividing log data to be detected, in one implementation manner of the embodiment of the application, before dividing the log data to be detected according to the sliding window, the sliding window and the sliding size are first determined, in practical application, the sliding window and the sliding step frequency size can be determined according to the characteristics and the requirements of the log data to be detected, and the embodiment of the application is not limited to this.

After the size and the step frequency of the sliding window are determined, starting from the initial position of log data to be detected, dividing according to the set size and step frequency of the sliding window to obtain each information block.

Step 102, determining the dispersion degree of the feature words in each information block according to a preset algorithm, screening according to the dispersion degree of each feature word, and reserving a preset number of target feature words.

In practical application, some common algorithms such as TF-IDF (word frequency-inverse document frequency), information entropy, text clustering and the like may be adopted to determine the degree of dispersion of feature words in the information block according to a preset algorithm, and in this embodiment of the present application, the preset algorithm is illustrated as TF-IDF, but it should be noted that the description manner is not limited to a specific preset algorithm.

After the dispersion degree of each feature word is obtained, screening is carried out according to the dispersion degree of each feature word, a word with higher dispersion degree can be selected as a target feature word according to the preset number of target feature words, and the feature words with representativeness and dispersion can be reserved, so that the follow-up data analysis and mining work can be facilitated.

And step 103, inputting the reserved target feature words into a pre-training neural network model for clustering, and obtaining a clustering result of the log data to be detected.

The reserved target feature words are input into the pre-training neural network model for clustering, so that further analysis and mining of log data to be detected can be facilitated. In this process, some common neural network models, such as models of a self encoder (Autoencoder), a convolutional neural network (Convolutional Neural Network, CNN), or a cyclic neural network (Recurrent Neural Network, RNN), may be used for performing cluster analysis, and the embodiment of the present application uses a pre-trained neural network model as a convolutional neural network for illustration, but it should be noted that this illustration method is not limited to a specific pre-trained neural network model, and the embodiment of the present application is not limited to this.

And 104, determining whether the clustering result of the log data to be detected is abnormal or not according to a preset clustering distance threshold value.

After the clustering result of the log data to be detected is obtained, whether the clustering result is abnormal or not needs to be further determined. A preset clustering distance threshold may be used to determine whether the clustering result is normal.

Firstly, determining a clustering distance threshold according to the characteristics of a data set and a preset threshold; and secondly, judging whether the clustering result is abnormal or not according to the clustering distance and a preset clustering distance threshold value. If the clustering distance is smaller than a preset clustering distance threshold value, the clustering result is considered to be normal; otherwise, the clustering result is considered to be abnormal.

The detection method for log abnormality provided by the disclosure mainly comprises the following steps: dividing the log data to be detected according to a sliding window to obtain each information block; determining the dispersion degree of the feature words in each information block according to a preset algorithm, screening according to the dispersion degree of each feature word, and reserving a preset number of target feature words; inputting the reserved target feature words into a pre-training neural network model for clustering to obtain a clustering result of the log data to be detected; and determining whether the clustering result of the log data to be detected is abnormal or not according to a preset clustering distance threshold value. Compared with the related art, the method and the device have the advantages that by means of adding the data preprocessing mode, the sliding window is introduced in the log data feature extraction stage to calculate the dispersion degree of the feature words in the log data in combination with the preset algorithm, structured log information is output as the feature vector of the limited block, extraction of the feature words can be achieved when log data with low correlation are faced, meanwhile, the problem that the feature word dimension is extremely high due to the fact that the log feature value faces large data quantity is achieved, dimension reduction is conducted on the data dimension of the feature words, and the detection speed of log abnormality is improved.

In one implementation manner of the embodiment of the present application, when clustering the target feature words in step 103, the following steps may be referred to for clustering, refer to fig. 2, and fig. 2 is a schematic flow chart of a log abnormality detection method provided in the embodiment of the present application; comprising the following steps:

step 201, inputting the target feature words into the self-coding network to perform coding and decoding processes, and obtaining decoded target feature words.

The self-encoder is an unsupervised learning algorithm, which can compress input data into low-dimensional encoded vectors, and then reconstruct the encoded vectors into original data through a decoder. And inputting the coded vector of the target feature word into a decoder for decoding processing to obtain the decoded target feature word.

By inputting the target feature words from the encoder for encoding and decoding, a low-dimensional feature vector can be obtained, which represents the position of each information block in the feature space. The feature vector can be used for tasks such as cluster analysis, anomaly detection and fault diagnosis, so that accuracy and efficiency of log data analysis are improved.

Step 202, clustering the decoded target feature words according to the clustering algorithm, and determining a clustering result of the log to be detected.

In one implementation manner of the embodiment of the present application, in practical application, a suitable clustering algorithm, such as K-means clustering, hierarchical clustering, DBSCAN, etc., may be selected according to characteristics and requirements of data, where the embodiment of the present application uses K-means clustering as an example for illustration, however, it should be noted that the description method is not a specific limitation of a specific clustering algorithm, and the embodiment of the present application does not limit this.

Inputting the decoded target feature words into a selected clustering algorithm to perform clustering treatment, and obtaining a clustering result.

In one implementation manner of the embodiment of the present application, the clustering result of the log to be detected may be determined according to the analysis of the clustering result and according to the indexes such as the clustering center and the clustering distance.

In an implementation manner of the embodiment of the present application, when determining the degree of dispersion of the feature words in each information block according to the preset algorithm, the method may include the following steps:

The log data is divided into a limited character sequence through a sliding window, and meanwhile, the log sequence needs to be converted into a characteristic vector which can be identified by a neural network for training of a subsequent deep clustering algorithm. The TF-IDF is an algorithm combining word frequency and inverse document frequency, so as to achieve the purpose of mining feature words in log data. The TF-IDF formula is as follows:

TF-idf=word frequency (TF) ×inverse document frequency (IDF) (1)

In short, IDF reflects the frequency of occurrence of a word in all text libraries and is calculated as follows:

where N represents the total number of texts in the text library, N (x) represents the total number of texts in the text library containing word x, and in order to prevent some words from not existing in the text library and resulting in denominator of 0, the IDF is smoothed as follows:

the feature vector input of the deep clustering algorithm is obtained after the logs are preprocessed through the TF-IDF algorithm, for processing massive logs, the number of the log feature words which are not repeated in log data is particularly large, so that the vector matrix input in the deep clustering neural network is extremely high in dimensionality and extremely sparse, and therefore the efficiency and accuracy in the neural network training are improved through a dimension reduction method. Reducing the mass log feature vector matrix from n dimensions to k dimensions, sequencing according to the dispersion degree of log features, and sequentially taking out the first k items, so as to obtain a covariance matrix vector after dimension reduction:

∑＝E[(X-E(X))(X-E(X) ^T )] (4)

It can be seen that the main feature information retained from the massive log data is the feature vector corresponding to the top k maximum feature value of the matrix.

In summary of one implementation manner of the embodiment of the present application, before clustering according to the pre-training neural network model in step 103, training is first performed on the pre-training neural network model, and for a specific training process, reference may be made to the following steps:

In one implementation manner of the embodiment of the present application, the training data may be log data in a database, or may be collected, for example, each service and component of a distributed system are deployed in a plurality of hosts, and log information is collected by deploying a lightweight agent on each host to log program logs, service logs, and the like, where the agent implements one-key batch deployment in a form issued by an operation platform. The Agent reports the collected log data to a high-performance, high-availability and real-time multi-node link cluster, the link cluster processes log information in real time in a stream processing mode and takes the kafka cluster as a cache area of a massive log to be responsible for temporarily storing and forwarding the log information after real-time processing; and data acquisition is realized.

The massive log feature vector matrix is input into a self-encoder network model, the log feature vector x is mapped to a feature space z through encoding of an encoder, then the feature space z is mapped back to input features by a encoder decoder to obtain reconstructed log information x ', and error of the reconstructed log information x' and standard input x is calculated. The method comprises the steps of dividing massive log data into training sets, inputting the training sets into a self-encoder network model for training after log information preprocessing and dimension reduction, and continuously adjusting model encoder network and encoder network weight parameters with error minimization as a target, wherein the target function is

Wherein θ, φ are network weight parameters of the encoder and the decoder, respectively.

In massive log anomaly detection training based on a deep clustering model, a clustering algorithm adopts a K-Means algorithm to cluster log features decoded by a self-encoder network. Selecting Euclidean distance as similarity index for n decoded two-dimensional log data matrixes X and categories K to be obtained, wherein the clustering target is to minimize the sum of squares of all kinds of clusters, namely, minimize:

U _k feature vector, X representing cluster center _i Characteristic vector representing data points, J representing total loss value

The meaning of the formula is to calculate the square of the euclidean distance between each data point and its cluster center, and take the sum of the distances of all data points as the total loss value. By minimizing the loss function, better clustering results can be obtained

In one implementation manner of the embodiment of the present application, when determining the clustering result of the log in step 103, the method may further be performed according to the following steps:

in one implementation manner of the embodiment of the present application, in the network security field, the logs may be classified according to the characteristics such as the network attack type and the malware behavior; in the field of system operation and maintenance, logs can be classified according to characteristics such as system abnormality, performance problems and the like; in the business intelligence field, logs can be classified according to characteristics such as user behaviors, transaction records and the like; according to different application scenes, different characteristic words can be selected for cluster analysis, different category rules are formulated, and different anomaly detection methods are adopted. Therefore, when determining the category of the log, screening and customizing are required according to a specific application scene, so as to ensure effective classification and anomaly detection of the log. Specifically, the embodiment of the present application is not limited thereto.

Analyzing each clustering result to find out representative feature words or phrases, which can be feature words with higher frequency or have important meaning on anomaly detection, and formulating rules of each category according to the clustering result and the representative feature words, wherein the rules comprise which feature words or phrases belong to the category and the anomaly conditions corresponding to the category; specifically, in actual use, the setting may be performed according to actual requirements, which is not limited in the embodiment of the present application.

It should be noted that, in the embodiments of the present disclosure, a plurality of steps may be included, and these steps are numbered for convenience of description, but these numbers are not limitations on the execution time slots and execution orders between the steps; the steps may be performed in any order, and embodiments of the present disclosure are not limited in this regard.

Corresponding to the detection method of log abnormality, the invention also provides a detection device of log abnormality. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, details not disclosed in the device embodiment may refer to the above-mentioned method embodiment, and details are not described in detail in the present invention.

Fig. 3 is a schematic structural diagram of a log abnormality detection device provided in an embodiment of the present disclosure, where, as shown in fig. 3, the log abnormality detection device includes:

a dividing unit 31, configured to divide the log data to be detected according to a sliding window, so as to obtain each information block;

a first determining unit 32, configured to determine a degree of dispersion of feature words in each of the information blocks according to a preset algorithm, and perform screening according to the degree of dispersion of each of the feature words, so as to retain a preset number of target feature words;

and the clustering unit 33 is used for inputting the reserved target feature words into a pre-training neural network model for clustering to obtain a clustering result of the log data to be detected.

And the second determining unit 34 is configured to determine whether the clustering result of the log data to be detected is abnormal according to a preset clustering distance threshold.

The present disclosure provides a detection apparatus for log abnormality, and the main technical scheme includes: dividing the log data to be detected according to a sliding window to obtain each information block; determining the dispersion degree of the feature words in each information block according to a preset algorithm, screening according to the dispersion degree of each feature word, and reserving a preset number of target feature words; inputting the reserved target feature words into a pre-training neural network model for clustering to obtain a clustering result of the log data to be detected; and determining whether the clustering result of the log data to be detected is abnormal or not according to a preset clustering distance threshold value. Compared with the related art, the method and the device have the advantages that by means of adding the data preprocessing mode, the sliding window is introduced in the log data feature extraction stage to calculate the dispersion degree of the feature words in the log data in combination with the preset algorithm, structured log information is output as the feature vector of the limited block, extraction of the feature words can be achieved when log data with low correlation are faced, meanwhile, the problem that the feature word dimension is extremely high due to the fact that the log feature value faces large data quantity is achieved, dimension reduction is conducted on the data dimension of the feature words, and the detection speed of log abnormality is improved.

Further, in a possible implementation manner of this embodiment, as shown in fig. 4, the pre-trained neural network model includes a self-coding network and a clustering algorithm, and the clustering unit 33 is further configured to:

Further, in a possible implementation manner of this embodiment, as shown in fig. 4, the first determining unit 32 is further configured to:

Further, in a possible implementation manner of this embodiment, as shown in fig. 4, the apparatus further includes:

a reconstruction unit 35, configured to input training log data into an encoder in the self-encoding network before the clustering unit 33 inputs the retained target feature words into a pre-training neural network model to perform clustering, and obtain a clustering result of the log data to be detected, and reconstruct the training log data according to the encoder and decoder in the self-encoding network; the training log data are processed according to the preset algorithm;

A calculation unit 36 for calculating an error between the training log data after reconstruction and the training log data before reconstruction;

and an adjusting unit 37, configured to adjust the network parameters of the self-encoder network according to the error, and repeat the training step of the self-encoder network model until the error value between the reconstructed training log data and the training log data before reconstruction is smaller than a preset error threshold, and then stop training the self-encoder network.

Further, in a possible implementation manner of this embodiment, as shown in fig. 4, the second determining unit 34 is further configured to:

The foregoing explanation of the method embodiment is also applicable to the apparatus of this embodiment, and the principle is the same, and this embodiment is not limited thereto.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

Fig. 5 shows a schematic block diagram of an example electronic device 400 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 5, the apparatus 400 includes a computing unit 401 that can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 402 or a computer program loaded from a storage unit 408 into a RAM (Random Access Memory ) 403. In RAM 403, various programs and data required for the operation of device 400 may also be stored. The computing unit 401, ROM 402, and RAM 403 are connected to each other by a bus 404. An I/O (Input/Output) interface 405 is also connected to bus 404.

Various components in device 400 are connected to I/O interface 405, including: an input unit 406 such as a keyboard, a mouse, etc.; an output unit 407 such as various types of displays, speakers, and the like; a storage unit 408, such as a magnetic disk, optical disk, etc.; and a communication unit 409 such as a network card, modem, wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

The computing unit 401 may be a variety of general purpose and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 401 include, but are not limited to, a CPU (Central Processing Unit ), a GPU (Graphic Processing Units, graphics processing unit), various dedicated AI (Artificial Intell igence ) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor ), and any suitable processor, controller, microcontroller, etc. The calculation unit 401 executes the respective methods and processes described above, for example, a log abnormality detection method. For example, in some embodiments, the method of detecting log anomalies may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 400 via the ROM 402 and/or the communication unit 409. When the computer program is loaded into RAM 403 and executed by computing unit 401, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the aforementioned method of detecting log anomalies in any other suitable way (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, FPGA (Field Programmable Gate Array ), ASIC (application-specific integrated circuit), ASSP (Appl ication Specific Standard Product, application-specific standard product), SOC (System On Chip ), CPLD (Complex Programmable Logic Device, complex programmable logic device), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory, erasable programmable read-Only Memory) or flash Memory, an optical fiber, a CD-ROM (Compact Di sc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display ) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network ), WAN (Wide Area Network, wide area network), internet and blockchain networks.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service ("Virtual Private Server" or simply "VPS") are overcome. The server may also be a server of a distributed system or a server that incorporates a blockchain.

It should be noted that, artificial intelligence is a subject of studying a certain thought process and intelligent behavior (such as learning, reasoning, thinking, planning, etc.) of a computer to simulate a person, and has a technology at both hardware and software level. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing, and the like; the artificial intelligence software technology mainly comprises a computer vision technology, a voice recognition technology, a natural language processing technology, a machine learning/deep learning technology, a big data processing technology, a knowledge graph technology and the like.

The various numbers of first, second, etc. referred to in this disclosure are merely for ease of description and are not intended to limit the scope of embodiments of this disclosure, nor to indicate sequencing.

At least one of the present disclosure may also be described as one or more, a plurality may be two, three, four or more, and the present disclosure is not limited. In the embodiment of the disclosure, for a technical feature, the technical features in the technical feature are distinguished by "first", "second", "third", "a", "B", "C", and "D", and the technical features described by "first", "second", "third", "a", "B", "C", and "D" are not in sequence or in order of magnitude.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially, or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.

The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.

Claims

1. A method for detecting log anomalies, comprising:

2. The method according to claim 1, wherein the pre-training neural network model includes a self-coding network and a clustering algorithm, and the inputting the retained target feature words into the pre-training neural network model for clustering, to obtain the clustering result of the log data to be detected includes:

3. The method of claim 1, wherein determining the degree of dispersion of the feature words in each of the information blocks according to a predetermined algorithm, and filtering according to the degree of dispersion of each of the feature words further comprises:

4. The method according to claim 2, wherein before the target feature words that remain are input into a pre-trained neural network model for clustering, the method further comprises:

5. The method according to claim 2, wherein the clustering the decoded target feature words according to the clustering algorithm, and determining the clustering result of the log to be detected includes:

6. A log abnormality detection device, comprising:

7. The apparatus of claim 6, wherein the pre-trained neural network model comprises a self-encoding network and a clustering algorithm, the clustering unit further to:

8. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

9. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-5.

10. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any of claims 1-5.