CN117851942A - Database system anomaly detection method and device based on reconstruction countermeasure training - Google Patents

Database system anomaly detection method and device based on reconstruction countermeasure training Download PDF

Info

Publication number
CN117851942A
CN117851942A CN202311416862.8A CN202311416862A CN117851942A CN 117851942 A CN117851942 A CN 117851942A CN 202311416862 A CN202311416862 A CN 202311416862A CN 117851942 A CN117851942 A CN 117851942A
Authority
CN
China
Prior art keywords
training
reconstruction
current
data
database system
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311416862.8A
Other languages
Chinese (zh)
Inventor
刘渝
魏如凯
周可
张永强
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huazhong University of Science and Technology
Original Assignee
Huazhong University of Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huazhong University of Science and Technology filed Critical Huazhong University of Science and Technology
Priority to CN202311416862.8A priority Critical patent/CN117851942A/en
Publication of CN117851942A publication Critical patent/CN117851942A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/243—Classification techniques relating to the number of classes
    • G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/094—Adversarial learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/62—Extraction of image or video features relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2123/00—Data types
    • G06F2123/02—Data types in the time domain, e.g. time-series data

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Biomedical Technology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a database system anomaly detection method and device based on reconstruction countermeasure training, which belong to the technical field of database system detection, and comprise the following steps: organizing monitoring data of a database system from one-dimensional representation to two-dimensional image mode representation so as to perform feature extraction, so as to construct a training sample set; the training samples may be learned using a neural network. Training the anomaly detection model by using the training sample set based on the reconstruction countermeasure training mode to fully learn the rule in the input characteristics, and completing the anomaly detection work by using the trained anomaly detection model with reconstruction capability and discrimination capability and a dynamic threshold value, wherein the anomaly detection accuracy is higher and the speed is faster; in addition, the abnormal threshold is dynamically adjusted, so that a complicated manual threshold adjusting process is avoided, and the applicability of the model is improved. Therefore, the technical problems of low detection accuracy, high time overhead and poor applicability of the existing anomaly detection method are solved.

Description

Database system anomaly detection method and device based on reconstruction countermeasure training
Technical Field
The invention belongs to the technical field of database system detection, and particularly relates to a database system anomaly detection method and device based on reconstruction countermeasure training.
Background
With the continuous development of the cloud service industry, a storage system in a data center scene bears various services of a large enterprise, and once performance problems occur, the storage system cannot provide a stable service for an upper layer, so that the enterprise suffers a huge loss. The abnormal diagnosis of the data center can help enterprises to quickly judge and remove system faults, so that the operation efficiency of the enterprises can be effectively improved, and the management cost can be reduced.
The existing learning model-based method is not specially used for extracting features aiming at the mutual influence relation among indexes, so that feature learning is insufficient. Then, the time sequence data collected by the data center storage system is difficult to label, and cannot judge which data are abnormal, so that the supervised abnormal detection method requiring a large amount of tag data is difficult to be actually deployed in a real scene, and the performance of the existing unsupervised abnormal detection method does not meet the requirements of the system. In addition, the current detection method model has long initialization time considering the timeliness of the abnormality detection problem. In addition, the abnormal threshold selection is not dynamic enough and is not suitable for the data center storage system scene with dynamic change of data distribution.
The problems cause that the existing anomaly detection method is low in detection accuracy, high in detection time cost and poor in detection method applicability.
Disclosure of Invention
Aiming at the above defects or improvement demands of the prior art, the invention provides a database system anomaly detection method and device based on reconstruction countermeasure training, which aims at organizing monitoring data of a database system from one-dimensional representation to two-dimensional image mode representation so as to perform feature extraction, so as to construct a training sample set; then training the anomaly detection model by using the training sample set based on a reconstruction countermeasure training mode; the trained abnormality detection model is used for identifying the monitoring data of the current period so as to perform abnormality detection, and the model is higher in detection accuracy and abnormality detection rate; in addition, the abnormal threshold is dynamically adjusted, so that a complicated manual threshold adjusting process is avoided, and the applicability of the model is improved; therefore, the technical problems of low detection accuracy, high time overhead and poor applicability of the existing anomaly detection method are solved.
To achieve the above object, according to one aspect of the present invention, there is provided a database system anomaly detection method based on reconstruction countermeasure training, including:
Training phase:
s1: converting the monitoring data corresponding to each time period of the database system into two-dimensional image mode data, and extracting the characteristics of the two-dimensional image mode data to obtain a training sample, thereby constructing a training sample set;
s2: reconstructing an anomaly detection model by using the training sample set for countermeasure training; wherein the anomaly detection model comprises: a generator network G, a discriminator network D, and an encoder network E; the generator network G performs feature extraction on the input training samples to obtain original features x, and the original features x are passed through an encoder G included in the original features x E And decoder G D Reconstructing to obtain a reconstructed feature x'; the discriminator network D compares the original characteristic x with the reconstructed characteristic x' to perform countermeasure training; the encoder network E performs downsampling on the reconstruction feature x 'to obtain a bottleneck vector z'; the bottleneck vector z' is used forAutomatically adjusting the corresponding abnormal threshold value in the training process;
the application stage comprises the following steps:
s3: converting the monitoring data corresponding to the current time period of the database system into two-dimensional image mode data, and extracting features to obtain current original features;
s4: inputting the current original characteristics into a trained abnormality detection model so as to enable the current original characteristics to output current reconstruction characteristics and a current abnormality threshold corresponding to the current original characteristics;
S5: generating a current anomaly score by utilizing the difference value of the current original characteristic and the corresponding current reconstruction characteristic, and comparing the current anomaly score with the current anomaly threshold value so as to detect anomalies.
In one embodiment, the S1 includes:
s11: converting monitoring data corresponding to a period of time of a database system into two-dimensional image mode data;
s12: performing feature extraction of time dimension and index dimension on the two-dimensional image modal data, and splicing the obtained two-dimensional features to obtain the training sample;
s13: and constructing the training sample set by utilizing training samples corresponding to a plurality of time periods.
In one embodiment, the step S11 includes:
monitor data w= { W for one period of time 1 ,…,W T Conversion to a set of image modality technique IMR data: x= { X 1 ,…,x 2 ,x T }，x i Is [0,255]Pixel values within a range, x i ∈R w×h The time period comprises T time points, i is the sequence number of the time point, and i is less than or equal to T;
wherein each pixel X in the IMR data X i Performance index value W corresponding to corresponding time point i The method comprises the steps of carrying out a first treatment on the surface of the w and h are the width and height of IMR data X, respectively.
In one embodiment, the S2 includes:
s21: inputting each training sample into the anomaly detection model;
S22: extracting features of the training samples by using the generator network G to obtain the original features x, and then using an encoder G in the generator network G E Compressing the original features x into bottleneck vectors z, reusing the decoder G in the generator network G D Downsampling the bottleneck vector z to obtain a reconstruction feature x'; comparing the original characteristic x and the reconstruction characteristic x ' by utilizing the discriminator network D so as to enable the original characteristic x and the reconstruction characteristic x ' to have the capability of judging whether the input data of the original characteristic x or the reconstruction characteristic x '; downsampling the reconstructed feature x 'with the encoder network E to obtain a bottleneck vector z';
s23: automatically adjusting a corresponding abnormal threshold value by utilizing the bottleneck vector z';
s24: repeating the steps S21-S23, and continuously changing model parameters in the training process until the abnormal detection model converges or reaches the preset maximum iteration times.
In one embodiment, the step S23 includes: and calculating a corresponding abnormal threshold value by using the peak value of the element in the bottleneck vector z' corresponding to each training sample based on the extreme value theory.
In one embodiment, the step S23 includes:
the bottleneck vector z' corresponding to each training sample is represented as a dataset (X 1 ,X 2 ,X 3 ,…,X n ) N is the number of the time points corresponding to the bottleneck vector z', and the peak value set is Y t The risk value is q;
by function SetInitial Threshold (X 1 ,X 2 ,X 3 ,…,X n ) Setting an initial threshold t; using the formula t≡ { X- i -t|X i >t, i.ltoreq.n } updates the initial threshold and uses equation N t ←Len(Y t ) Obtaining the number N of peak values t The method comprises the steps of carrying out a first treatment on the surface of the And then according to the formulaMaximum likelihood estimation to obtain extremum parameter estimated value +.>And estimation function->Using the formulaCalculating the current corresponding abnormal threshold z q 。
In one embodiment, the step S24 includes:
s241: initializing initial parameters theta corresponding to the generator network G, the discriminator network D and the encoder network E in the anomaly detection model network G ，θ E And theta D ；
S242: fixing the parameter θ G And the parameter theta E By means of a function L A ＝||f(x)-f(x′)|| 2 Calculate the challenge loss L A F (x) and f (x ') represent vector representations of the original feature x and the reconstructed feature x' after feature extraction of the identifier network D, respectively 2 Represents an L2 norm;
s243: by using the countering loss L A Corresponding gradient information update parameter θ D ；
S244: fixing the parameter θ D Using the formula l=θl A +βL C +γL E Calculating an overall optimization loss function L trained by the anomaly detection model; wherein L is C And L E The context loss and the coding loss, L, respectively C ＝||x-x′|| 1 ，L E ＝||z-z′|| 2 θ, β and γ are super-parameters, for adjusting the influence of a single loss term on the total loss function L 1 Represents an L1 norm;
s245: updating the parameter θ using gradient information of the overall optimization loss L G And parameter theta E The method comprises the steps of carrying out a first treatment on the surface of the S241-S244 are repeatedly performed until the anomaly detection model converges or a preset maximum number of iterations is reached.
In one embodiment, before the application stage, the anomaly detection method further includes: if the trained abnormality detection model is aged, updating the abnormality detection model trained in the training stage by using a model fine-tuning technology.
According to another aspect of the present invention, there is provided a database system abnormality detection apparatus based on reconstruction countermeasure training, including:
a data processing unit, configured to, in a training phase: converting the monitoring data corresponding to each time period of the database system into two-dimensional image mode data, and extracting the characteristics of the two-dimensional image mode data to obtain a training sample, thereby constructing a training sample set; also for use in the application phase: converting the monitoring data corresponding to the current time period of the database system into two-dimensional image mode data, and extracting features to obtain current original features;
the model execution unit is used for, in a training stage: reconstructing an anomaly detection model by using the training sample set for countermeasure training; wherein the anomaly detection model comprises: a generator network G, a discriminator network D, and an encoder network E; the generator network G performs feature extraction on the input training samples to obtain original features x, and the original features x are passed through an encoder G included in the original features x E And decoder G D Reconstructing to obtain a reconstructed feature x'; the discriminator network D compares the original characteristic x with the reconstructed characteristic x' to perform countermeasure training; the encoder network E performs downsampling on the reconstruction feature x 'to obtain a bottleneck vector z'; the bottleneck vector z' is used for automatically adjusting the corresponding abnormal threshold value in the training process; also for use in the application phase: inputting the current original characteristics into a trained abnormality detection model so as to enable the current original characteristics to output current reconstruction characteristics and a current abnormality threshold corresponding to the current original characteristics;
and the abnormality detection unit is used for generating a current abnormality score by utilizing the difference value of the current original characteristic and the corresponding current reconstruction characteristic, and comparing the current abnormality score with the current abnormality threshold value so as to perform abnormality detection.
According to another aspect of the present invention there is provided a detection system for a database system comprising a memory storing a computer program and a processor implementing the steps of the above method when the processor executes the computer program.
According to another aspect of the present invention there is provided a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the above method.
In general, the above technical solutions conceived by the present invention, compared with the prior art, enable the following beneficial effects to be obtained:
(1) According to the invention, the monitoring data of the database system is organized from one-dimensional representation to two-dimensional image mode representation so as to perform feature extraction, so that a training sample set is constructed; the training samples may be learned using a neural network. Training the anomaly detection model by using the training sample set based on the reconstruction countermeasure training mode to fully learn the rule in the input characteristics, and completing the anomaly detection work by using the trained anomaly detection model with reconstruction capability and discrimination capability and a dynamic threshold value, wherein the anomaly detection accuracy is higher and the speed is faster; in addition, the abnormal threshold is dynamically adjusted, so that a complicated manual threshold adjusting process is avoided, and the applicability of the model is improved. Therefore, the technical problems of low detection accuracy, high time overhead and poor applicability of the existing anomaly detection method are solved.
(2) According to the method, the one-dimensional state data corresponding to the monitoring data are characterized to be in a two-dimensional image mode, the two one-dimensional convolution kernels are used for feature extraction, and the data features are specifically extracted from the two aspects of the time dimension and the index dimension of the two-dimensional image mode, so that more data information can be obtained, and the features input to the anomaly detection model are more accurate.
(3) In the scheme, the monitoring data are converted into a group of Image modality technology (IMR) data according to the numerical value, specifically, the pixel value in the range of [0,255], the conversion process is simple, the calculation complexity is low, and the time cost of the whole algorithm is further improved.
(4) In the scheme, the anomaly detection model mainly comprises three self-networks: a generator network, a discriminator network and an encoder network. The aim of the arbiter network is to implement the process of discrimination in the countermeasure training, identifying the data sources for the real input x and the input x' reconstructed by the generator network. Considering that the training of the anomaly detection model by using reconstruction only easily leads to overfitting, the existence of the discriminator network refers to different learning mechanisms, and the partial reconstruction training of the model training is changed into the countermeasure training, so that the robustness of the training result is stronger. The discriminator network optimizes model parameters in the process of judging whether the data is generated by reconstruction of the generator or comes from a real sample, and enhances the discriminating capability of the network to the data, so that the generator is forced to reconstruct the input data better, and finally the trained anomaly detection model with stronger anomaly identification judging capability is obtained.
(5) In the scheme, an abnormal threshold value is automatically adjusted by adopting an extreme value theory. The extreme theory (Extreme Value Theory, EVT) is to infer the distribution of extreme events that might be observed without any distribution assumption based on the raw data. The primary goal of extreme value theory is to find rules for extreme events. According to existing theoretical conclusions, these extreme events are different from the overall data distribution, and they tend to follow their own data distribution. Furthermore, the extreme theory is not constrained by the data distribution. And selecting a proper abnormal threshold value as a reference value to perform subsequent abnormal reasoning according to the extreme value theory after the training process is finished, so that the abnormality detection accuracy is high, and a complicated manual threshold value adjusting process is avoided.
(6) In the scheme, an abnormal threshold is automatically adjusted based on a super-threshold model, wherein the first step of the super-threshold model is to select an initial threshold, and if the threshold is selected to be too large, available data is reduced, so that the data is wasted; while too small a threshold selection increases the difference between the super-threshold distribution and the generalized pareto distribution (Generalized Pareto Distribution, GPD), thereby increasing the bias. Selecting an appropriate initial threshold enables the super-threshold data samples to be well compliant with GPD. This allows maximum utilization of the data without affecting the final model. Relative to fitting an extreme value distribution to X i The super-threshold model may fit GPDs to X i Above-t. Then maximum likelihood estimation is used to obtain the extreme parameter estimated valueAnd estimation function->And finally, calculating to obtain the current abnormal threshold value.
(7) In the scheme, data is sent into a proposed anomaly detection model with reconstruction capability and discrimination capability, model training is completed according to a predefined loss function, wherein L2 norms between vector representations of original data and reconstructed data after network D feature extraction are calculated against loss, and a discriminator network D has the capability of judging whether the data come from a real sample or a generator reconstructed sample; the context loss function optimizes and constrains the reconstruction process of the generator network G, the generator is expected to realize better data reconstruction capability, the coding loss enables network parameters to be incapable of effectively reconstructing abnormal samples, the capability of detecting the abnormal detection of the model is improved, and the characteristic that only the label information of the normal samples is known in the training process of the model is fully utilized.
(8) In the scheme, the fine tuning of the model refers to training again by using new incoming data on the basis of the model which is already trained previously, and information is transferred from one data set to another data set, so that the model is better adapted to new data distribution, and finally the generalization capability and the abnormality detection precision of the model are improved. The purpose of using fine tuning in this solution is to adapt the model to the distribution of new data, and the task of anomaly detection is performed before and after fine tuning. It should be noted that the training rounds of the fine tuning process are less than those of the model initialization, and the fine tuning can be performed on part of the layer parameters in the model according to the need, without retraining all the layer parameters, so that the training cost is saved.
Drawings
Fig. 1 is a flowchart of a database system anomaly detection method based on reconstruction countermeasure training according to embodiment 1 of the present invention.
Fig. 2 is a schematic structural diagram of an abnormality detection model provided in embodiment 1 of the present invention.
Fig. 3 is a flowchart of a database system anomaly detection method based on reconstruction countermeasure training according to embodiment 2 of the present invention.
Fig. 4 is a schematic diagram of a loss function convergence curve of the anomaly detection method based on the reconstruction countermeasure training in embodiment 7 of the present invention.
FIG. 5 is a graph showing the comparison of the performance of the database system anomaly detection method based on the reconstructed challenge training in example 8 of the present invention, using a model update strategy, and without using it.
Fig. 6 is a schematic structural diagram of a database system anomaly detection device based on reconstruction countermeasure training according to embodiment 9 of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention. In addition, the technical features of the embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.
Example 1
The embodiment provides a database system anomaly detection method based on reconstruction countermeasure training, as shown in fig. 1, including:
training phase:
s1: and converting the monitoring data corresponding to each time period of the database system into two-dimensional image mode data, and extracting the characteristics of the two-dimensional image mode data to obtain training samples, thereby constructing a training sample set.
S2: and (3) utilizing the training sample set to reconstruct the anomaly detection model for countermeasure training. As shown in fig. 2, the anomaly detection model includes: a generator network G, a discriminator network D, and an encoder network E. The generator network G performs feature extraction on the input training samples to obtain original features x, and the original features x are obtained through an encoder G included in the original features x E And decoder G D To reconstruct to obtain a reconstructed feature x'. The discriminator network D compares the original characteristic x and the reconstructed characteristic x' for countermeasure training. The encoder network E downsamples the reconstructed feature x 'to obtain its bottleneck vector z'. The bottleneck vector z' is used for automatically adjusting the corresponding abnormal threshold value in the training process.
The application stage comprises the following steps:
s3: and converting the monitoring data corresponding to the current time period of the database system into two-dimensional image mode data, and extracting the characteristics to obtain the current original characteristics.
S4: and inputting the current original characteristics into a trained abnormality detection model so as to enable the current original characteristics to output the current reconstruction characteristics and the current abnormality threshold corresponding to the current original characteristics.
S5: generating a current anomaly score by utilizing the difference value of the current original characteristic and the corresponding current reconstruction characteristic, and comparing the current anomaly score with a current anomaly threshold value so as to detect anomalies.
Example 2
In this embodiment, S1 includes: s11: and converting the monitoring data corresponding to one time period of the database system into two-dimensional image mode data by using an image mode technology. S12: and performing feature extraction of time dimension and index dimension on the two-dimensional image modal data, and splicing the obtained two-dimensional features to obtain a training sample. S13: and constructing a training sample set by utilizing training samples corresponding to a plurality of time periods.
Specifically, as shown in FIG. 3, the raw monitoring data is processed and converted for subsequent analysis. Techniques used may include data normalization, sliding window, image modality techniques, and one-dimensional convolution operations.
Firstly, the data normalization means are used for normalizing the acquired data to the same dimension, and the calculation mode of the normalization process is as follows: Where p is the original value of the system performance index and p' is the normalized value. P is p max And p min Respectively representing the maximum and minimum values of the index.
Then, the time sequence information between the historical data and the current data in the system monitoring data is captured by using a sliding window mode.The data processing unit defines a sliding window W with a length K at the moment t t The following are provided: w (W) t ＝{s t-K+1 ,…,s t-1 ,s t And converting the normalized multivariate time sequence S into a window sequence to represent the number of window data. Each window corresponds to a number of observations over a range of time that will be input to the next stage.
The data is then processed as two-dimensional image modality data using image modality techniques.
Finally, the convolution kernel is used to extract features. After the image modality data is generated, the data is then feature extracted using a plurality of one-dimensional convolution operations. Two one-dimensional convolutions are used to extract features along the time dimension and the index dimension, respectively. Finally, a final feature vector is obtained by using a splicing operation, and the extracted features can be spliced through a torch.cat () function.
Example 3
In this embodiment, S11 includes:
monitor data w= { W for one period of time 1 ,…,W T Conversion to a set of image modality technique IMR data: x= { X 1 ,…,x 2 ,x T }，x i Is [0,255]Pixel values within a range, x i ∈R w×h The time period comprises T time points, i is the sequence number of the time point, and i is less than or equal to T. Wherein each pixel X in the IMR data X i Performance index value W corresponding to corresponding time point i . w and h are the width and height of IMR data X, respectively.
Specifically, the monitor data w= { W for one period of time 1 ,…,W T Conversion into: x= { X 1 ,…,x 2 ,x T },(x i ∈R w×h I is less than or equal to T); the process will convert the data value for each point within the window to [0,255 ]]Within the scope, i.e. organized in a picture format, each pixel in the IMR data corresponds to a performance indicator value at a certain point in time.
Example 4
In this embodiment, S2 includes:
s21: each training sample is input into an anomaly detection model.
S22: the training sample is subjected to feature extraction by using a generator network G to obtain an original feature x, and then an encoder G in the generator network G is utilized E Compressing the original features x into bottleneck vectors z, reusing the decoder G in the generator network G D And downsampling the bottleneck vector z to obtain a reconstruction feature x'. The original feature x and the reconstructed feature x 'are compared by the arbiter network D to enable it to determine whether its input data is the original feature x or the reconstructed feature x'. The reconstructed feature x 'is downsampled by the encoder network E to obtain its bottleneck vector z'.
S23: and automatically adjusting the corresponding abnormal threshold value by using the bottleneck vector z'.
S24: repeating the steps S21-S23, and continuously changing model parameters in the training process until the anomaly detection model converges or reaches the preset maximum iteration times.
Specifically, the data is sent to an anomaly detection model to complete model training according to a predefined loss function, and as shown in fig. 2, the anomaly detection model mainly comprises three self-networks: a generator network G, a discriminator network D and an encoder network E.
Wherein the generator network extracts characteristics of the input samples and then passes through an encoder network G E And a decoder network G D To reconstruct the input. The anomaly detection model first inputs a feature vector x corresponding to an input sample into the generator network G, and then passes x to the encoder network G E . Here G E Will compress the input data into a bottleneck vector z, where z ε R d . Followed by a decoder network G D A section that uses deconvolution layer to reconstruct vector z to yield output x '(x' =g (x)).
Wherein the encoder network E compresses an output x' resulting from the reconstruction of the input x by the generator G. The encoder network E includes G in a generator network E Have the same network structure but their network parameters are not consistent. The encoder network E downsamples x ' to obtain its bottleneck vector z ' =e (x ').
The aim of the discriminator network is to realize the discriminating process in the countermeasure training, and the identification of the data sources is carried out on the real input x and the input x' reconstructed by the generator network. Considering that the training of the anomaly detection model by using reconstruction only easily leads to overfitting, the existence of the discriminator network refers to different learning mechanisms, and the partial reconstruction training of the model training is changed into the countermeasure training, so that the robustness of the training result is stronger. The discriminator network optimizes model parameters in the process of judging whether the data is reconstructed by the generator or comes from a real sample, and enhances the discriminating capability of the network on the data, thereby forcing the generator to reconstruct the input data better.
Example 5
In this embodiment, S23 includes: and calculating a corresponding abnormal threshold value by using the peak value of the element in the bottleneck vector z' corresponding to each training sample based on the extreme value theory.
Extreme value theory is a statistical theory whose purpose is to find the law of extreme values. In the scheme, an abnormal threshold value is automatically adjusted by adopting an extreme value theory. The extreme theory (Extreme Value Theory, EVT) is to infer the distribution of extreme events that might be observed without any distribution assumption based on the raw data. The primary goal of extreme value theory is to find rules for extreme events. According to existing theoretical conclusions, these extreme events are different from the overall data distribution, and they tend to follow their own data distribution. Furthermore, the extreme theory is not constrained by the data distribution. And selecting a proper abnormal threshold value as a reference value to perform subsequent abnormal reasoning according to the extreme value theory after the training process is finished, so that the abnormality detection accuracy is high, and a complicated manual threshold value adjusting process is avoided.
Example 6
In this embodiment, S23 includes:
the bottleneck vector z' corresponding to each training sample is represented as a dataset (X 1 ,X 2 ,X 3 ,…,X n ) N is the number of time points corresponding to the bottleneck vector z', and the peak value set is Y t The risk value is q.
Pass boxNumber SetInitial Threshold (X) 1 ,X 2 ,X 3 ,…,X n ) An initial threshold t is set. Using the formula t≡ { X- i -t|X i >t, i.ltoreq.n } updates the initial threshold and uses equation N t ←Len(Y t ) Obtaining the number N of peak values t . And then according to the formulaMaximum likelihood estimation to obtain extremum parameter estimated value +.>And estimation function->Using the formulaCalculating the current corresponding abnormal threshold z q 。
Example 7
In this embodiment, S24 includes:
s241: initializing initial parameters theta corresponding to a generator network G, a discriminator network D and an encoder network E in an anomaly detection model network G ，θ E And theta D 。
S242: fixed parameter theta G And parameter theta E By means of a function L A ＝||f(x)-f(x′)|| 2 Calculate the challenge loss L A F (x) and f (x ') respectively represent vector representations of the original feature x and the reconstructed feature x' after feature extraction of the identifier network D, i 2 Representing the L2 norm.
S243: by countering losses L A Corresponding gradient information update parameter θ D 。
S244: fixed parameter theta D Using the formula l=θl A +βL C +γL E Calculating an overall optimization loss function L trained by an anomaly detection model; wherein L is C And L E The context loss and the coding loss, L, respectively C ＝||x-x′|| 1 ，L E ＝||z-z′|| 2 θ, β and γ are super parameters for adjusting the orderThe effect of individual loss terms on the overall loss function L, I.I. | 1 Representing the L1 norm.
S245: updating the parameter θ using gradient information for the overall optimization loss L G And parameter theta E . S241-S244 are repeatedly performed until the anomaly detection model converges or a preset maximum number of iterations is reached.
Specifically, the specific training process of the model is as follows:
first, training data X is prepared train ＝{x 1 ,x 2 ,…,x N Training data is a normal sample collected under the data center storage system. And prepares the model as shown in fig. 2. The model includes a generator G, an encoder E, and a discriminator D. Initializing parameters theta of modules of a model network G ，θ E And theta D . And initializing a learning rate.
Then, the network parameter θ is fixed G And theta E Calculate the countermeasures loss L A . The purpose of the countermeasure training is to give the arbiter network D the ability to determine whether the data is coming from a real sample or a sample reconstructed by the generator. Conventional GANs typically employ a decision of a discriminator on their input data to update the network, where the model updates the parameters of network D in a manner that does not directly employ the output result, but rather a vector representation of the previous layer of network that the discriminator network D classifies for the output result. Specifically, countering the loss would calculate the L2 norm between the vector representations of the original data and the reconstructed data after the network D feature extraction. Thus, L A The loss is defined as: l (L) A ＝||f(x)-f(x′)|| 2 。
Then, updating θ based on the gradient information obtained by the loss D 。
Second, fix parameter θ D The overall optimization loss function L trained by the anomaly detection model is calculated as follows: l=θl A +βL C +γL E The method comprises the steps of carrying out a first treatment on the surface of the Wherein L is C And L E The context loss and the coding loss, respectively, θ, β and γ are hyper-parameters for adjusting the effect of a single loss term on the total loss function L. Context loss to optimize the reconstruction process of the generator network GBy means of the method, the generator is expected to achieve good data reconstruction capability, network parameters cannot be effectively reconstructed for abnormal samples due to coding loss, the capability of detecting the abnormality of the model is improved, and the characteristic that only the label information of the normal samples is known in the training process of the model is fully utilized. The two are defined as follows: l (L) C ＝||x-x′|| 1 ，L E ＝||z-z′|| 2 。
Finally, updating the parameter θ according to the gradient information of the loss L G And parameter theta E . And the above steps are alternately performed until the model converges or a preset maximum number of iterations is reached. Fig. 4 is a loss function convergence curve of the anomaly detection method based on the reconstruction countermeasure training in the present embodiment.
Example 8
In this embodiment, before the application stage, the anomaly detection method further includes: and updating the anomaly detection model trained in the training stage by using a model fine tuning technology.
As the database system continues to operate, the distribution of the collected data will also change gradually. If no further operations are performed on the deployed model, model aging problems can occur. Thus, the present embodiment uses a model Fine tuning (Fine Tune) technique to update the model. The fine tuning of the model means that new incoming data is used for training again on the basis of a model which is already trained previously, information is transferred from one data set to another data set, the model is enabled to be better suitable for new data distribution, and generalization capability and anomaly detection accuracy of the model are finally improved. The data center database system can judge whether the system is in a normal running state according to the rule formulated by the operation and maintenance expert, so that the data only comprising the sample in the normal running state can be collected, and the fine tuning process only uses the data sample in the normal running state of the system. The traditional fine tuning technology firstly trains a model in an unsupervised mode in an upstream task, converts data such as images, characters and the like into vectors, and then selects a proper network for downstream tasks such as classification or regression. The purpose of the fine tuning is to adapt the model to the new data distribution, and the task of anomaly detection is performed before and after the fine tuning. It should be noted that, the training rounds of the fine tuning process in this embodiment are less than the training rounds of the model initialization, and the fine tuning can be performed on part of the layer parameters in the model according to the need, without retraining all the layer parameters, so as to save the training overhead. The training time and resource consumption of the fine-tuning model is less than training a new model from scratch. The fine tuning technique belongs to the cumulative update method, i.e. re-using the existing model and using the newly arrived data for training. Compared with an alternative strategy scheme, the fine tuning strategy adopted by the embodiment can save training time.
The following describes the experimental procedure of the abnormality detection method provided by the present invention:
the experimental environment of the invention: the experimental codes are all completed by using a PyTorch5 deep learning software framework, the used experimental equipment is a single server, and relevant hardware configuration of the server comprises a CPU (Central processing Unit) with the model number of Intel (R) Xeon (R) Silver 4114CPU@2.20GHz, four GPUs with the model number of NVIDIATeslaV100, DDR4 memory with the space of 64GB, a storage hard disk with the space of 100TB and an operating system with the version number of CentOS Linux release 7.9.2009 (Core).
In the model learning process, the shape of input data is divided according to a sliding window. Throughout the experiment, batch size, i.e., batch size, of 128 was used. The dimension of the bottleneck vector z set in the method is 100, and on the last loss function learning formula, the model super parameters set in the method are θ=1, β=50 and γ=1. The method adopts a random gradient descent (Stochastic Gradient Descent, SGD) optimizer, and the learning rate is set to be 2 multiplied by 10 -3 。
To illustrate the superiority of the present invention in anomaly detection, the present invention was tested on three data sets SWaT, WADI, SMD. Experimental results show that the proposed method can obtain a higher F1 score value compared with the existing method. The image mode technology used in the method is described, the countermeasure model is reconstructed, and the abnormality detection precision can be effectively improved by the dynamic threshold.
TABLE 1
Meanwhile, the experimental results on the effectiveness of the model update strategy are shown in fig. 5. It can be seen that when an update strategy is used, it performs significantly better over time than if no update strategy is used.
Example 9
The embodiment provides a database system anomaly detection device based on reconstruction countermeasure training, as shown in fig. 6, including:
a data processing unit, configured to, in a training phase: and converting the monitoring data corresponding to each time period of the database system into two-dimensional image mode data, and extracting the characteristics of the two-dimensional image mode data to obtain training samples, thereby constructing a training sample set. Also for use in the application phase: and converting the monitoring data corresponding to the current time period of the database system into two-dimensional image mode data, and extracting the characteristics to obtain the current original characteristics.
Wherein the data processing unit may process and convert raw data collected from the data center storage system for subsequent analysis. Techniques used by the data processing unit include data normalization, sliding window, image modality techniques, and one-dimensional convolution operations.
The model execution unit is used for, in a training stage: and (3) utilizing the training sample set to reconstruct the anomaly detection model for countermeasure training. Wherein the abnormality detection model includes: a generator network G, a discriminator network D, and an encoder network E. The generator network G performs feature extraction on the input training samples to obtain original features x, and the original features x are obtained through an encoder G included in the original features x E And decoder G D To reconstruct to obtain a reconstructed feature x'. The discriminator network D compares the original feature x with the reconstructed feature x' for countermeasure training. The encoder network downsamples the reconstructed feature x 'to obtain its bottleneck vector z'. The bottleneck vector z' is used for automatically adjusting the corresponding abnormal threshold value in the training process. Also for use in the application phase: inputting the current original characteristics into a trained abnormality detection model so as to outputAnd the current reconstruction feature and the current abnormality threshold value corresponding to the current original feature.
Wherein the model execution unit uses the model execution unit to complete the training and updating process of the model, and selects a proper abnormal threshold value as a reference value to perform subsequent abnormal reasoning after the training process is finished according to the extreme value theory
The abnormality detection unit is used for generating a current abnormality score by utilizing the difference value of the current original characteristic and the corresponding current reconstruction characteristic, and comparing the current abnormality score with a current abnormality threshold value so as to perform abnormality detection.
The main function of the abnormality detection unit is to determine whether the data center storage system component node has abnormality. The abnormality detection unit consists of a score calculation module and an abnormality judgment module. By S T Representing the data to be detected, where T represents the length of the data. These data are first sent to a data processing unit to be converted into two-dimensional IMR data, expressed as Wherein->Representing one IMR data. The score calculation module (model execution unit) then calculates corresponding anomaly scores for all IMR data using the trained model in the model execution unit, the anomaly scores being expressed as +.>Is->Is a anomaly score of (2). If the input data follows the normal mode of system operation, the input data is well reconstructed by the generator network and therefore has a low anomaly score. On the other hand, if the anomaly score is higher, the worse the reconstruction of the input data is explained, which means that the probability of the input being anomaly is higherLarge. Based on this knowledge, the anomaly determination module will compare these anomaly scores to EVT thresholds z selected in the model execution units q A comparison is made. Specifically, if->Then S T Then it will be marked as abnormal and otherwise normal data. Once S T Marked as abnormal, the abnormality detection unit triggers an abnormality alarm and notifies the data center operation staff for subsequent processing.
The above-mentioned division of each unit in the database system abnormality detection device is only for illustration, and in other embodiments, the database system abnormality detection device may be divided into different units as needed to complete all or part of the functions of the database system abnormality detection device.
The specific limitation of the database system abnormality detection device may be referred to as limitation of the database system abnormality detection hereinabove, and will not be described herein. The respective units in the database system abnormality detection apparatus described above may be realized in whole or in part by software, hardware, or a combination thereof. The units can be embedded in hardware or independent of a processor in the computer equipment, and can also be stored in a memory in the computer equipment in a software mode, so that the processor can call and execute the operations corresponding to the units.
Example 10
According to another aspect of the present invention there is provided a detection system for a database system comprising a memory and a processor, the memory storing a computer program, the processor implementing the steps of the above method when executing the computer program.
The detection system of the database system comprises a processor and a memory which are connected through a system bus. Wherein the processor is configured to provide computing and control capabilities to support operation of the entire electronic device. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program may be executed by a processor to implement a database system anomaly detection method provided by the above embodiments. The internal memory provides a cached operating environment for operating system computer programs in the non-volatile storage medium.
Example 11
According to another aspect of the present invention there is provided a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the above method.
Any reference to memory, storage, database, or other medium used in the present embodiments may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It will be readily appreciated by those skilled in the art that the foregoing description is merely a preferred embodiment of the invention and is not intended to limit the invention, but any modifications, equivalents, improvements or alternatives falling within the spirit and principles of the invention are intended to be included within the scope of the invention.

Claims (10)

1. A database system anomaly detection method based on reconstruction countermeasure training, comprising:
training phase:
s1: converting the monitoring data corresponding to each time period of the database system into two-dimensional image mode data, and extracting the characteristics of the two-dimensional image mode data to obtain a training sample, thereby constructing a training sample set;
s2: reconstructing an anomaly detection model by using the training sample set for countermeasure training; wherein the abnormality detectionThe model comprises: a generator network G, a discriminator network D, and an encoder network E; the generator network G performs feature extraction on the input training samples to obtain original features x, and the original features x are passed through an encoder G included in the original features x E And decoder G D Reconstructing to obtain a reconstructed feature x'; the discriminator network D compares the original characteristic x with the reconstructed characteristic x' to perform countermeasure training; the encoder network E downsamples the reconstructed feature x' to obtain a bottleneck vector z thereof ′ The method comprises the steps of carrying out a first treatment on the surface of the The bottleneck vector z ′ The method is used for automatically adjusting the corresponding abnormal threshold value in the training process;
the application stage comprises the following steps:
s3: converting the monitoring data corresponding to the current time period of the database system into two-dimensional image mode data, and extracting features to obtain current original features;
S4: inputting the current original characteristics into a trained abnormality detection model so as to enable the current original characteristics to output current reconstruction characteristics and a current abnormality threshold corresponding to the current original characteristics;
s5: generating a current anomaly score by utilizing the difference value of the current original characteristic and the corresponding current reconstruction characteristic, and comparing the current anomaly score with the current anomaly threshold value so as to detect anomalies.
2. The method for detecting anomalies in a database system based on training for reconstruction countermeasures as recited in claim 1, wherein said S1 includes:
s11: converting monitoring data corresponding to a period of time of a database system into two-dimensional image mode data;
s12: performing feature extraction of time dimension and index dimension on the two-dimensional image modal data, and splicing the obtained two-dimensional features to obtain the training sample;
s13: and constructing the training sample set by utilizing training samples corresponding to a plurality of time periods.
3. The method for detecting anomalies in a database system based on training for reconstruction countermeasure according to claim 2, characterized in that said S11 includes:
monitor data w= { W for one period of time 1 ,…,W T Conversion to a set of image modality technique IMR data: x= { X 1 ,…,x 2 ,x T }，x i Is [0,255]Pixel values within a range, x i ∈R w×h The time period comprises T time points, i is the sequence number of the time point, and i is less than or equal to T;
wherein each pixel X in the IMR data X i Performance index value W corresponding to corresponding time point i The method comprises the steps of carrying out a first treatment on the surface of the w and h are the width and height of IMR data X, respectively.
4. The method for detecting anomalies in a database system based on training for reconstruction countermeasures as recited in claim 1, wherein said S2 includes:
s21: inputting each training sample into the anomaly detection model;
s22: extracting features of the training samples by using the generator network G to obtain the original features x, and then using an encoder G in the generator network G E Compressing the original features x into bottleneck vectors z, reusing the decoder G in the generator network G D Downsampling the bottleneck vector z to obtain a reconstruction feature x'; comparing the original characteristic x and the reconstruction characteristic x ' by utilizing the discriminator network D so as to enable the original characteristic x and the reconstruction characteristic x ' to have the capability of judging whether the input data of the original characteristic x or the reconstruction characteristic x '; downsampling the reconstructed feature x' with the encoder network E to obtain its bottleneck vector z ′ ；
S23: using the bottleneck vector z ′ Automatically adjusting the corresponding abnormal threshold value;
s24: repeating the steps S21-S23, and continuously changing model parameters in the training process until the abnormal detection model converges or reaches the preset maximum iteration times.
5. The method for detecting anomalies in a database system based on training for reconstruction countermeasure according to claim 4, wherein S23 includes:
utilizing a bottleneck vector z corresponding to each training sample based on extreme value theory ′ The peak value of each element in the list calculates the corresponding abnormal threshold value.
6. The method for detecting anomalies in a database system based on training for reconstruction with respect to claim 5, wherein S23 comprises:
the bottleneck vector z' corresponding to each training sample is represented as a dataset (X 1 ,X 2 ,X 3 ,…,X n ) N is the number of the time points corresponding to the bottleneck vector z', and the peak value set is Y t The risk value is q;
by function SetInitial Threshold (X 1 ,X 2 ,X 3 ,…,X n ) Setting an initial threshold t; using the formula t≡ { X- i -t|X i >t, i.ltoreq.n } updates the initial threshold and uses equation N t ←Len(Y t ) Obtaining the number N of peak values t The method comprises the steps of carrying out a first treatment on the surface of the And then according to the formulaMaximum likelihood estimation to obtain extremum parameter estimated value +.>And estimation function->Using the formulaCalculating the current corresponding abnormal threshold z q 。
7. The method for detecting anomalies in a database system based on training for reconstruction countermeasure of claim 4, wherein S24 includes:
S241: initializing the generator network G, the arbiter network D and the anomaly detection model networkInitial parameter θ corresponding to encoder network E G ，θ E And theta D ；
S242: fixing the parameter θ G And the parameter theta E By means of a function L A ＝||f(x)-f(x′)|| 2 Calculate the challenge loss L A F (x) and f (x') denote the original feature x and the reconstructed feature x, respectively ′ Vector representation after feature extraction of the arbiter network D 2 Represents an L2 norm;
s243: by using the countering loss L A Corresponding gradient information update parameter θ D ；
S244: fixing the parameter θ D Using the formula l=θl A +βL C +γL E Calculating an overall optimization loss function L trained by the anomaly detection model; wherein L is C And L E The context loss and the coding loss, L, respectively C ＝||x-x′|| 1 ，L E ＝||z-z′|| 2 θ, β and γ are super-parameters, for adjusting the influence of a single loss term on the total loss function L 1 Represents an L1 norm;
s245: updating the parameter θ using gradient information of the overall optimization loss L G And parameter theta E The method comprises the steps of carrying out a first treatment on the surface of the S241-S244 are repeatedly performed until the anomaly detection model converges or a preset maximum number of iterations is reached.
8. A method of anomaly detection for a database system based on training for reconstruction countermeasure as claimed in any one of claims 1 to 7, wherein prior to the application stage, the anomaly detection method further comprises: if the trained abnormality detection model is aged, updating the abnormality detection model trained in the training stage by using a model fine-tuning technology.
9. A database system anomaly detection device based on reconstruction countermeasure training, characterized by comprising:
a data processing unit, configured to, in a training phase: converting the monitoring data corresponding to each time period of the database system into two-dimensional image mode data, and extracting the characteristics of the two-dimensional image mode data to obtain a training sample, thereby constructing a training sample set; also for use in the application phase: converting the monitoring data corresponding to the current time period of the database system into two-dimensional image mode data, and extracting features to obtain current original features;
the model execution unit is used for, in a training stage: reconstructing an anomaly detection model by using the training sample set for countermeasure training; wherein the anomaly detection model comprises: a generator network G, a discriminator network D, and an encoder network E; the generator network G performs feature extraction on the input training samples to obtain original features x, and the original features x are passed through an encoder G included in the original features x E And decoder G D Reconstructing to obtain a reconstructed feature x'; the discriminator network D compares the original characteristic x with the reconstructed characteristic x' to perform countermeasure training; the encoder network E downsamples the reconstructed feature x' to obtain a bottleneck vector z thereof ′ The method comprises the steps of carrying out a first treatment on the surface of the The bottleneck vector z ′ The method is used for automatically adjusting the corresponding abnormal threshold value in the training process; also for use in the application phase: inputting the current original characteristics into a trained abnormality detection model so as to enable the current original characteristics to output current reconstruction characteristics and a current abnormality threshold corresponding to the current original characteristics;
and the abnormality detection unit is used for generating a current abnormality score by utilizing the difference value of the current original characteristic and the corresponding current reconstruction characteristic, and comparing the current abnormality score with the current abnormality threshold value so as to perform abnormality detection.
10. A detection system for a database system, comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 8 when the computer program is executed.
CN202311416862.8A 2023-10-27 2023-10-27 Database system anomaly detection method and device based on reconstruction countermeasure training Pending CN117851942A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311416862.8A CN117851942A (en) 2023-10-27 2023-10-27 Database system anomaly detection method and device based on reconstruction countermeasure training

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311416862.8A CN117851942A (en) 2023-10-27 2023-10-27 Database system anomaly detection method and device based on reconstruction countermeasure training

Publications (1)

Publication Number Publication Date
CN117851942A true CN117851942A (en) 2024-04-09

Family

ID=90543854

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311416862.8A Pending CN117851942A (en) 2023-10-27 2023-10-27 Database system anomaly detection method and device based on reconstruction countermeasure training

Country Status (1)

Country Link
CN (1) CN117851942A (en)

Similar Documents

Publication Publication Date Title
US10877863B2 (en) Automatic prediction system for server failure and method of automatically predicting server failure
CN110163261B (en) Unbalanced data classification model training method, device, equipment and storage medium
Pamina et al. An effective classifier for predicting churn in telecommunication
US11544558B2 (en) Continual learning of artificial intelligence systems based on bi-level optimization
CN111738520A (en) System load prediction method fusing isolated forest and long-short term memory network
CN113326177B (en) Index anomaly detection method, device, equipment and storage medium
CN113283368B (en) Model training method, face attribute analysis method, device and medium
CN117110748A (en) Transformer substation main equipment operation state abnormality detection method based on fusion terminal
CN114500004A (en) Anomaly detection method based on conditional diffusion probability generation model
Bi et al. Large-scale network traffic prediction with LSTM and temporal convolutional networks
US20230252060A1 (en) Artificial Intelligence (AI)-based Engine for Processing Service Requests
Moon et al. Anomaly detection using a model-agnostic meta-learning-based variational auto-encoder for facility management
Assuncao et al. Automatic evolution of autoencoders for compressed representations
Dalle Pezze et al. A multi-label continual learning framework to scale deep learning approaches for packaging equipment monitoring
Baradie et al. Managing the Fifth Generation (5G) Wireless Mobile Communication: A Machine Learning Approach for Network Traffic Prediction
US20210319269A1 (en) Apparatus for determining a classifier for identifying objects in an image, an apparatus for identifying objects in an image and corresponding methods
CN117575685A (en) Data analysis early warning system and method
CN109190505A (en) The image-recognizing method that view-based access control model understands
CN117079017A (en) Credible small sample image identification and classification method
CN116932355A (en) Information processing method and system based on big data
WO2024039508A1 (en) Trigger point detection for online root cause analysis and system fault diagnosis
CN109165587B (en) Intelligent image information extraction method
CN109165586B (en) Intelligent image processing method for AI chip
CN110717577A (en) Time series prediction model construction method for noting regional information similarity
CN117851942A (en) Database system anomaly detection method and device based on reconstruction countermeasure training

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination