CN115098345B

CN115098345B - Data anomaly detection method, electronic device and readable storage medium

Info

Publication number: CN115098345B
Application number: CN202211022426.8A
Authority: CN
Inventors: 李梦婷; 杨奉山; 李光辉
Original assignee: Guangzhou Jianyue Information Technology Co ltd
Current assignee: Guangzhou Jianyue Information Technology Co ltd
Priority date: 2022-08-25
Filing date: 2022-08-25
Publication date: 2022-12-23
Anticipated expiration: 2042-08-25
Also published as: CN115098345A

Abstract

The disclosure relates to a data anomaly detection method, an electronic device and a readable storage medium, wherein the method comprises the following steps: acquiring first data of a detection object corresponding to a first timestamp and a first sequence corresponding to a first time window; wherein the first time window precedes the first timestamp; splicing the first sequence and the initial self-adjusting data, and inputting the spliced first sequence and the initial self-adjusting data into a preset first converter model to obtain first reconstruction data of the first data; updating the self-adjusting data through a first residual error between the first data and the first reconstruction data, splicing the first array and the updated self-adjusting data, and inputting the spliced first array and the updated self-adjusting data into a preset second converter model to obtain second reconstruction data of the first data; and detecting whether a second residual error between the first data and the second reconstruction data exceeds a set residual error range, if so, determining that the first data is abnormal data.

Description

Data anomaly detection method, electronic device and readable storage medium

Technical Field

The disclosed embodiments relate to the field of computer technologies, and in particular, to a data anomaly detection method, an electronic device, and a computer-readable storage medium.

Background

The service platform collects and counts platform data every day, and arranges the data according to the time stamp and the time sequence to form a dynamic sequence, which can also be called as a time sequence. Because the dynamic sequence reflects the operation condition of the service platform, operation and maintenance personnel can perform operation and maintenance management on the service platform based on the dynamic sequence. In operation and maintenance management based on platform data, quality, efficiency and cost are three major core problems to be solved, and data anomaly detection focuses on solving the most basic quality stability problem.

The existing data anomaly detection methods mainly comprise a statistical learning method, a supervised learning method, an outlier detection method and the like, and the methods are only suitable for performing data anomaly detection on a dynamic sequence with a single anomaly form, but cannot accurately identify the abnormal data in the dynamic sequence with various anomaly forms, so that the quality stability of operation and maintenance management is influenced.

Disclosure of Invention

It is an object of the disclosed embodiments to provide a data anomaly detection scheme that can capture anomaly data in dynamic series of anomalies that are diverse in their form.

According to a first aspect of the present disclosure, there is provided a data anomaly detection method, including:

acquiring first data of a detection object corresponding to a first timestamp and a first sequence corresponding to a first time window; wherein the first time window precedes the first timestamp, and the first sequence comprises a plurality of data arranged in chronological order;

splicing the first sequence and initial self-adjusting data, and inputting the spliced first sequence and initial self-adjusting data into a preset first converter model to obtain first reconstruction data of the first data;

updating the self-adjusting data through a first residual error between the first data and the first reconstruction data, splicing the first array and the updated self-adjusting data, and inputting the spliced first array and the updated self-adjusting data into a preset second converter model to obtain second reconstruction data of the first data;

detecting whether a second residual error between the first data and the second reconstruction data exceeds a set residual error range;

and determining that the first data is abnormal data when the second residual exceeds the residual range.

Optionally, the first converter model and the second converter model have the same encoder.

Optionally, before the acquiring the first data corresponding to the first timestamp and the first sequence corresponding to the first time window of the detection object, the method further includes:

acquiring first original data corresponding to the first timestamp and a first original sequence corresponding to the first time window in the detection object;

normalizing the first original data and the first original sequence through set normalization index data to obtain the first data corresponding to the first original data and the first sequence corresponding to the first original sequence; the normalized index data is determined according to a second original sequence corresponding to a second time window in the detection object, wherein the second time window is before the first timestamp, and the data quantity of the second original sequence is greater than that of the first original sequence;

the determining that the first data is abnormal data is: and determining the first original data as abnormal data.

Optionally, after acquiring the first raw data corresponding to the first timestamp in the detection object, the method further includes:

detecting whether the first original data is in a set abnormal data range; wherein the abnormal data range is determined from the second raw sequence;

executing the step of normalizing the first original data and the first original sequence by the set normalization index data when the first original data is not in the abnormal data range;

determining that the first raw data is abnormal data when the first raw data is within the abnormal data range.

Optionally, before detecting whether a second residual between the first data and the second reconstructed data exceeds a set residual range, the method further includes:

acquiring a first time tag corresponding to the first timestamp;

acquiring a first residual error parameter value corresponding to the first time label; the first time label is a time label in a time label set, and different time labels in the time label set correspond to different residual parameter values;

and obtaining the set residual error range according to the first residual error parameter value.

obtaining a sample set; wherein, the jth sample in the sample set comprises jth sample data of a corresponding jth time stamp of the detection object and ith sample sequence of a corresponding jth time window, the jth time window is before the jth time stamp, different samples in the sample set correspond to different time stamps and different time windows, and the jth time stamp is before the first time stamp;

training the first converter model and the second converter model through the sample set and the set loss function to obtain a first model parameter of the first converter model and a second model parameter of the second converter model;

updating the first converter model according to the first model parameters and the second converter model according to the second model parameters.

Optionally, the loss function comprises a weighted sum of a first loss function corresponding to the first converter model and a second loss function corresponding to the second converter model, a weight of the first loss function decreases as the number of training iterations increases, and a weight of the second loss function increases as the number of training iterations increases;

the first loss function represents a first loss of first reconstruction data of sample data relative to the sample data, the second loss function represents a second loss of second reconstruction data of the sample data relative to the sample data, the first reconstruction data of the sample data is obtained by reconstructing the sample data through a corresponding sample number sequence by the first converter model, and the second reconstruction data of the sample data is obtained by reconstructing the sample data through the corresponding sample number sequence by the second converter model.

Optionally, the obtaining the sample set comprises:

acquiring a second original sequence of the detected object corresponding to a second time window; wherein the second time window precedes the first time stamp, and the second original sequence has a greater number of data than the first sequence;

acquiring and storing an abnormal data range and normalized index data of the detection object according to the second original sequence;

according to the abnormal data range, filtering out the data with obvious abnormality from the second original sequence to obtain a filtered second original sequence;

normalizing the filtered second original number series through the normalization index data to obtain a second number series;

and cutting the second array according to the set window length to obtain the sample set.

Optionally, after training the first converter model and the second converter model by the sample set and the set loss function, and obtaining a first model parameter of the first converter model and a second model parameter of the second converter model, the method includes:

acquiring at least part of sample data as calibration data and second reconstruction data of the calibration data obtained in training the first converter model and the second converter model;

obtaining a residual error of the calibration data according to the second reconstruction data of the calibration data and the calibration data;

aiming at each time label in a set time label set, obtaining a calibration data set corresponding to the time label;

obtaining and storing residual parameter values corresponding to the time labels according to the residual errors of the calibration data in the calibration data set;

before the detecting whether the second residual between the first data and the second reconstructed data exceeds the set residual range, the method further includes:

acquiring a first time tag corresponding to the first timestamp; wherein the first time tag is one time tag in the time tag set;

acquiring a first residual error parameter value corresponding to the first time tag;

According to a second aspect of the present disclosure, there is also provided an electronic device comprising a memory for storing a computer program and a processor for executing the data anomaly detection method according to the first aspect of the present disclosure under the control of the computer program.

According to a third aspect of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the data anomaly detection method according to the first aspect of the present disclosure.

In the abnormal data detection method provided by the embodiment of the present disclosure, when performing abnormal detection on first data serving as a real-time data point, a first converter (Transformer) model and a second converter (Transformer) model are invoked to reconstruct the first data through a first sequence to obtain second reconstructed data of the first data, and then whether the first data is abnormal data is determined according to whether a second residual between the first data and the second reconstructed data of the first data is within a set residual range. According to the abnormal data detection method, the capture capability of the Transformer model on long-term dependence and mutual interaction is outstanding, and time sequence characteristics can be deeply represented, so that various abnormal data points in a dynamic sequence can be efficiently and accurately captured. Meanwhile, when the Transformer model is called to reconstruct the first data through the first sequence, the self-adjusting data is introduced into the model input besides the first sequence, and the data reconstruction is completed in two stages by combining the updating of the self-adjusting data, so that the data reconstruction capability of the model on various types of dynamic sequence is effectively improved, the accurate capture of abnormal data with various forms is further realized, and the robustness of data abnormality detection is improved.

Features of embodiments of the present specification and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the specification and together with the description, serve to explain the principles of the embodiments of the specification.

Fig. 1 is a schematic diagram illustrating a structure of a business system to which a data anomaly detection method according to an embodiment of the present disclosure may be applied;

FIG. 2 illustrates a flow diagram of a data anomaly detection method according to some embodiments;

FIG. 3 illustrates a model structure diagram of a time series data reconstruction model in accordance with some embodiments;

FIG. 4 illustrates a flow diagram of a data anomaly detection method according to further embodiments;

FIG. 5 illustrates a schematic diagram of a segmentation process to obtain model training samples, according to some embodiments;

FIG. 6 illustrates a flow diagram for online real-time detection and pre-training updates, according to some embodiments;

fig. 7 illustrates a hardware architecture diagram of an electronic device according to some embodiments.

Detailed Description

Various exemplary embodiments of the present specification will now be described in detail with reference to the accompanying drawings.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the embodiments, their application, or uses.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.

It should be noted that all actions of acquiring signals, information or data in the present disclosure are performed under the premise of complying with the corresponding data protection regulation policy of the country of the location and obtaining the authorization given by the owner of the corresponding equipment.

The embodiment of the present disclosure relates to a data anomaly detection method for capturing anomalous data in a dynamic sequence, where the dynamic sequence is also referred to as a time sequence, and refers to a sequence in which data (i.e., numerical values) of the same statistical index are arranged according to the time sequence of occurrence of the data, that is, all data in the dynamic sequence correspond to a uniform statistical index, and each data has a different timestamp. For example, if the dynamic number list is a number list of statistical indexes of the number of online people of a certain service platform, each piece of data in the dynamic number list reflects the number of online people of a corresponding timestamp. The statistical index may be a one-dimensional index or a multidimensional index, for example, the statistical index is a two-dimensional index related to the number of people on line and the flow rate, and correspondingly, the data of the dynamic series may be one-dimensional data or multidimensional data, which is not limited herein.

Taking the application scenario shown in fig. 1 as an example, the data source device 2000 sequentially sends dynamic sequence data to the electronic device 1000 for data anomaly detection according to the time stamp sequence of the data

Data, data

Data, data

Data, data

Data, data

\8230A, a method for preparing the same and a method for preparing the same

Representing data with time stamp ti in the dynamic sequence, wherein i is an integer greater than or equal to 1, and ti represents the data

The corresponding time stamp also represents the data

Ordering in a dynamic series. The electronic device 1000 is receiving data

In time, data may be detected based on a set data anomaly detection method

Whether it is abnormal data or not, and storing the data

In the case of abnormal data, a predetermined abnormal process is performed, for example, an abnormal result is returned to the data source device 2000. For example, the electronic device 1000 is receiving data

Then, data is detected based on a set data anomaly detection method

If the data is abnormal, the data is processed

Marking as anomalous data and targeting data

And performing set exception processing.

At present, the method for detecting data abnormality of dynamic sequence mainly includes: statistical learning methods such as exponential Weighted Moving-Average (EWMA), supervised learning methods based on Gradient Boosting Decision Trees (GBDT), outlier detection methods based on isolated forests (isolation _ forest), deep learning methods based on Long Short-Term Memory artificial neural networks (LSTM), and the like. For the above data anomaly detection methods, the statistical learning method is generally greatly influenced by the curve form, and the complexity of the problem capable of being represented is limited; the supervised learning method is usually difficult to implement in a real scene, because abnormal labeling information is often difficult to obtain in the real scene, complicated artificial feature engineering is required, and the labeling cost is very high; the outlier detection method can only detect remarkable boundary abnormal points, cannot consider the periodic mode of a time sequence, and has low accuracy; the deep learning method has the problems of low calculation speed, high cost, insufficient capability of capturing long-term sequence information and the like. It can be seen that the above data anomaly detection methods cannot be implemented on the ground due to the difficulty in obtaining an anomalous data sample, or are only suitable for performing data anomaly detection on a dynamic sequence with a single anomaly form, whereas a large number of false reports and false reports occur on a dynamic sequence with multiple anomaly forms, so that an accurate detection result is difficult to obtain, and the quality stability of operation and maintenance management cannot be guaranteed.

In order to deal with dynamic arrays with various abnormal forms, the embodiment of the disclosure provides a data anomaly detection method for reconstructing data based on a Transformer model, which realizes accurate capture of abnormal data with various forms in the dynamic arrays, enables the method to be applicable to data anomaly detection of various types of dynamic arrays, and is beneficial to improving quality stability of operation and maintenance management. The method of the embodiment of the present disclosure may be applied to an intelligent operation and maintenance platform, a service data large disk, and the like, and is not limited herein.

The data anomaly detection method according to the embodiment of the present disclosure may be applied to the scenario shown in fig. 1 and implemented by the electronic device 1000, and the electronic device 1000 performs anomaly detection on data in the dynamic sequence provided by the data source device 2000 by using the data anomaly detection method according to the embodiment of the present disclosure.

The electronic device 1000 may be a server or a server cluster, and the like, which is not limited herein. As shown in fig. 1, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, and the like. The hardware configuration shown in fig. 1 is merely illustrative and is in no way intended to limit the present disclosure, its application, or uses.

Processor 1100 is used to execute computer programs, which may be written in an instruction set of architectures such as x86, arm, RISC, MIPS, SSE, and the like. The memory 1200 includes, for example, a ROM (read only memory), a RAM (random access memory), a nonvolatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a network interface and the like. The communication device 1400 is capable of wired or wireless communication, for example, the communication device 1400 may include at least one short-range communication module, such as any module for performing short-range wireless communication based on short-range wireless communication protocols, such as the Hilink protocol, wiFi (IEEE 802.11 protocol), mesh, bluetooth, zigBee, thread, Z-Wave, NFC, UWB, liFi, and the like, and the communication device 1400 may also include a long-range communication module, such as any module for performing WLAN, GPRS, 2G/3G/4G/5G long-range communication.

In this embodiment, the memory 1200 of the electronic device 1000 is used for storing a computer program, and the computer program is used for controlling the processor 1100 to operate so as to control the electronic device 1000 to execute the data anomaly detection method according to the embodiment of the present disclosure.

The data source device 2000 may be a server or a terminal device, and is not limited herein. As shown in fig. 1, the data source apparatus 2000 may include a processor 2100, a memory 2200, an interface device 2300, a communication device 2400, an output device 2500, an input device 2600, and the like. The hardware structure of the data source device 2000 can be understood with reference to the electronic device 1000, and will not be described herein.

The data source device 2000 may obtain data of the dynamic sequence by counting the service data, and send the data of the dynamic sequence to the electronic device 1000 for anomaly detection; the data source device 2000 may also send the service data to the electronic device 1000, and the electronic device 1000 performs statistics to obtain data of dynamic data for anomaly detection, which is not limited herein.

Next, a data anomaly detection method according to some embodiments, which can be applied to the application scenario as in fig. 1, is explained, and fig. 2 shows a flow diagram of the data anomaly detection method according to some embodiments. As shown in fig. 2, the data anomaly detection method of the present embodiment may include the following steps S210 to S250:

step S210, acquiring first data corresponding to a first timestamp of the detection object and a first sequence corresponding to a first time window.

In this embodiment, the detection object is a detected dynamic sequence, the detection object may be a dynamic sequence of any statistical index related to any service platform, and each data of the dynamic sequence is an index value related to the statistical index. The statistical index may be a one-dimensional index or a multidimensional index having more than one dimension, and correspondingly, the data of the detection object may be one-dimensional data or multidimensional data, which is not limited herein.

In this embodiment, the first data corresponds to a data point of the detection object, and the first data is a numerical value when the data of the detection object is one-dimensional data, and the first data is an array when the data of the detection object is multi-dimensional data. In this embodiment, the first time stamp is denoted by ti

Representing the first data.

In this embodiment, the first sequence corresponds to a plurality of data points of the detection object, and includes n data arranged in time sequence in the first time window w, and is recorded as

. The first time window precedes the first time stamp, i.e. the first time stamp follows the end time stamp of the first time window, a first sequence

The time stamp of the n data is before the first time stamp, and the embodiment will pass through the first sequence

For the first data

The reconstruction is performed, i.e. based on the first sequence

The characteristic of the data changing along with the time obtains the normal data which should appear at the first time stamp under the condition that the detected object has no abnormity, therefore, the reconstruction can be understood as based on the first sequence

The normal data of the detected object at the first time stamp are predicted, and the reconstructed data and the first data are compared

Determining the first data

Whether it is abnormal data. First array of numbers

The number n of data or the window length of the first time window may be set by comprehensively considering reconstruction accuracy, calculation amount, calculation speed, and the like, for example, 5 to 15 data with a timestamp before the first timestamp may be selected to reconstruct the first data, and the like, which is not limited herein.

As time goes forward, the detection object will continuously add new data, which may be understood as real-time data, and here, the steps S210 to S250 may be performed with the real-time data as the first data to perform anomaly detection on the real-time data.

The electronic device 1000 may obtain the first data from the data source device 2000, or may obtain the first data according to statistics of a set statistical indicator based on the service data provided by the data source device 2000. Due to the first array

Time of middle n dataThe timestamp precedes the first timestamp, and thus, is prior to the first data

Before anomaly detection, the first sequence

Wherein n data have completed abnormal detection and are stored locally, wherein the first array

The data in (1) may be data that all pass the anomaly detection, that is, all normal data, and therefore, in step S210, the electronic device 1000 may directly obtain the first sequence from the local

。

The above first data

May be the raw data of the detected object corresponding to the first timestamp ti, correspondingly, the first sequence

Is the original sequence of numbers of the test object corresponding to the first time window w. The above first data

The data may be data obtained by preprocessing the raw data, and the first sequence may be the first sequence, correspondingly

Is the sequence after the same pre-processing is performed on the original sequence. The preprocessing is, for example, a normalization processing, and is not limited herein.

The first time window w may be a time window adjacent to the first time stamp ti to select the first data in time

Adjacent n data pairs as first data

Performing the reconstruction to improve the accuracy of the reconstruction, e.g. the first sequence comprises

。

In the case where the data to be detected is one-dimensional data, the first number sequence may be regarded as a 1 × n matrix, and in the case where the data to be detected is a-dimensional data, where a is an integer greater than 1, the first number sequence may be regarded as an a × n matrix.

Step S220, concatenating the first sequence and the initial self-adjusting data, and inputting the concatenated data into a preset first Transformer (Transformer) model to obtain first reconstructed data of the first data.

In this embodiment, the first data is reconstructed through the first sequence based on a temporal data reconstruction model of a Transformer. The time series data reconstruction model comprises 2-stage Transformer models, namely a first Transformer model and a second Transformer model.

Referring to fig. 3, each of the first and second fransformer models includes an Encoder (Encoder) and a Decoder (Decoder). The Encoder (Encoder) includes a plurality of sub-encoders that the network structure is the same, and the quantity of sub-Encoder can set up as required, and wherein, each sub-Encoder is two-layer structure, is bull Attention mechanism layer (Multi-Head Attention) and feedforward (Feed Forward) full link layer from input to output direction in proper order, and a plurality of sub-encoders series connection are between the input and the output of Encoder. The Decoder (Decoder) comprises a plurality of sub-decoders with the same network structure, the number of the sub-decoders of the encoder is the same as that of the sub-encoders of the encoder, each sub-Decoder is of a three-layer structure, mask Multi-Head Attention mechanism layers (mask Multi-Head Attention mechanism), multi-Head Attention mechanism layers (Multi-Head Attention mechanism) and Feed-Forward (Feed Forward) full connection layers are sequentially arranged from the input direction to the output direction, the plurality of sub-decoders are connected in series, the input of the first-stage sub-Decoder is the output of the encoder, the input of other sub-decoders at each stage comprises the output of the previous-stage sub-Decoder and the output of the encoder, and the output of the last-stage sub-Decoder is the output of the Decoder.

Referring to fig. 3, the transform model further includes a Linear transformation layer (Linear) and an activation function layer, the Linear transformation layer may be a fully-connected neural network, and the activation function used by the activation function layer may be, for example, sigmoid, softmax, or the like, and is not limited herein.

For the transform model, since the encoder has no built-in mechanism to capture the relative position (i.e. time sequence relationship) between each data in the input sequence, referring to fig. 3, the transform model further includes a position Encoding module (Positional Encoding) to position-encode each data to be input to the encoder by the position Encoding model.

In step S220, the first sequence is used as an input sequence of the time-series data reconstruction model, the electronic device 1000 splices (concat) the first sequence and the initial Self-adjusting data (Self-adjusting score) together to obtain a first spliced sequence, inputs the first spliced sequence into a preset first transform model, and reconstructs the first data

Performing reconstruction of the first stage to obtain first reconstructed data

Wherein the first reconstructed data

The data obtained by performing a first stage of prediction on normal data that should appear at the first time stamp of the test object is obtained based on the first sequence and the initial self-adjustment data.

The initial self-adjustment data is a set value, which may be set to 0, for example.

Referring to fig. 3, inputting the first splice sequence to a preset first transform model includes: calling a position coding module to perform position coding on the first splicing sequence to obtain a first position sequence, wherein the first position sequence reflects the sequence of each data in the first splicing sequence; adding the first position sequence and the first splicing sequence to obtain a first fusion sequence; and inputting the first fusion sequence into an encoder of a first Transformer model, and reconstructing the first data at a first stage to obtain first reconstructed data

。

Step S230, updating the self-adjusting data according to a first residual error between the first data and the first reconstruction data, and splicing the first sequence and the updated self-adjusting data and inputting the spliced first sequence and updated self-adjusting data to a preset second transform model to obtain second reconstruction data of the first data.

First reconstructed data is obtained in step S220

Thereafter, the first data can be calculated

And the first reconstructed data

First residual error between. The first residual reflects the first data

And the first reconstructed data

Can be obtained based on various distance calculation manners, e.g. by calculating the first data

And the first reconstructed data

The Euclidean distance (also called L2 distance) between the first and second residuals to obtain a first residual error

。

After obtaining the first residual, the self-adjusting data is updated by the first residual, which may be updating the self-adjusting data from an initial value to the first residual. Then, the first sequence and the updated Self-adjusting data (Self-adjusting score) are spliced (concat) together to obtain a second spliced sequence, and the second spliced sequence is input into a preset second transform model to perform data processing on the first data

Performing reconstruction of the second stage to obtain second reconstruction data

. Here, since the first residual error is relative to the initial self-alignment data, the first data and the normal data corresponding to the first time stamp can be more accurately reflected

Therefore, after the self-adjusting data is updated through the first residual error, the reconstruction of the first data in the second stage is carried out by combining the first sequence, the reconstruction precision can be effectively improved, and the second reconstructed data can be obtained

Normal data corresponding to the first time stamp of the detected object can be represented, and the second reconstructed data can be further reconstructed

As detecting the first data

Reference data of whether or not there is an abnormality.

Referring to fig. 3, inputting the second splice sequence to a preset second transform model comprises: calling a position coding module to perform position coding on the second splicing sequence to obtain a second position sequence, wherein the second position sequence reflects the sequence of each data in the second splicing sequence; adding the second position sequence and the second splicing sequence to obtain a second fusion sequence; and inputting the second fusion sequence into an encoder of a second Transformer model, and reconstructing the first data at a second stage to obtain second reconstructed data

。

In the present embodiment, the second reconstruction data

Reconstructing the model for the time-series data based on the first sequence as a final reconstruction result of reconstructing the first data, i.e. the second reconstructed data

The output of the model is reconstructed for the time series data.

In some embodiments, the first Transformer model and the second Transformer model may have the same encoder, so that they may share one encoder. In these embodiments, referring to fig. 3, the first Transformer model includes an encoder and a first decoder, and the second Transformer model includes the encoder and a second decoder, wherein the first decoder and the second decoder may have the same network structure but different model parameters. In these embodiments, the same encoder is used for encoding in the two reconstruction stages, which is equivalent to performing feature extraction on the first fused sequence and the second fused sequence based on the same encoder, which is beneficial to improving the reconstruction accuracy of reconstructing the first data.

In step S240, it is detected whether a second residual between the first data and the second reconstructed data exceeds a set residual range.

Second reconstruction data is obtained through step S230

Then, the first data can be calculated

With second reconstructed data

Second residual error between. The second residual reflects the first data

With second reconstructed data

Can be obtained based on various distance calculation means, e.g. by calculating the first data

With second reconstructed data

The Euclidean distance (also called L2 distance) between the first and second residuals to obtain a second residual error

。

Due to the second reconstructed data

Represents the normal data of the detected object corresponding to the first timestamp ti, and thus the second residual reflects the first data

Proximity to normal data. If the second residual error is larger, the set residual error is exceededRange, then the first data is accounted for

Largely deviated from the normal data, the first data

The exception should occur, which is exception data; if the second residual error is smaller and is within the set residual error range, the first data is described

Relatively close to normal data, the first data

And is normal data according with the data distribution rule of the detection object.

The residual range is represented by a residual upper limit value and a residual lower limit value. The second residual error is smaller than the residual error lower limit value or larger than the residual error upper limit value, namely the second residual error exceeds the residual error range; the second residual is greater than or equal to the residual lower limit value and less than or equal to the residual upper line value, which means that the second residual does not exceed the residual range.

In some embodiments, in order to improve the accuracy of determining whether the first data is abnormal based on the set residual error range, a plurality of time labels may be set according to a distribution condition of time stamps of data in the detection object, that is, according to a time sequence periodicity characteristic of the detection object, the time labels form a time label set, and residual error parameter values corresponding to the time labels in the time label set are respectively set for the time labels in the time label set, where the residual error parameter values are used for calculating a residual error range, that is, for calculating a residual error upper limit value and a residual error lower limit value.

Each timestamp in the set of timestamps represents a common characteristic of the timestamps, and the common characteristic may be the same time point corresponding to 24 hours, the same time period corresponding to 24 hours, or the like. For example, when time tags are set according to time points within 24 hours, different time tags correspond to different time points within 24 hours, and the time sequence periodicity is characterized by, for example, an 8-point time tag, a 9-point time tag, a 10-point time tag, and the like, and accordingly, a time stamp at an arbitrary date of 8 points in the detection object corresponds to the 8-point time tag, and a time stamp at an arbitrary date of 9 points in the detection object corresponds to the 9-point time tag. For example, time stamps may be set according to time periods within 24 hours, each time stamp may correspond to one time period within 24 hours, and different time stamps may correspond to different time periods.

In these embodiments, before detecting whether the second residual between the first data and the second reconstructed data exceeds the set residual range, the method may further include the following steps: acquiring a first time tag corresponding to the first time stamp; acquiring a first residual error parameter value corresponding to a first time tag; and obtaining the set residual error range used in step S240 according to the first residual error parameter value.

The first time tag is a time tag in the time tag set. The time label set comprises a plurality of time labels, and different time labels in the time label set correspond to different residual parameter values. The residual parameter values may include the residual mean μ and the residual variance σ of the corresponding time tag. Residual upper limit of residual range

And residual lower limit value

Can be expressed as:

in the formula (1) and the formula (2),

and

to setAnd (4) super-parameter. For the first time stamp corresponding to the first time stamp, after obtaining the first residual parameter value corresponding to the first time stamp, according to the formula (1) and the formula (2), the residual range corresponding to the first time stamp can be obtained, so that the electronic device 1000 can use the residual range in step S240.

The electronic device 1000 may save mapping data reflecting the mapping relationship between the time stamp and the residual parameter value in the residual anomaly detection model for the electronic device 1000 to call when executing step S240.

Step S250, determining that the first data is abnormal data when the second residual exceeds the range of the residual.

In step S250, when the second residual exceeds the set residual range, the first data may be determined to be abnormal data. When the first data is data obtained by preprocessing original data of the detection object corresponding to the first timestamp, it may be determined that the original data is abnormal data. And in case the second residual does not exceed, i.e. is within, the range of residuals, the first data may be determined to be normal data.

From the above steps S210 to S250, the electronic device 1000 compares the first data as the real-time data point

When abnormality detection is carried out, a time sequence data reconstruction model based on a Transformer is called to pass through a first sequence

For the first data

And reconstructing to obtain second reconstructed data of the first data, and further judging whether the first data is abnormal data according to whether a second residual between the first data and the second reconstructed data is within a set residual range. Here, because the Transformer model has outstanding capturing ability for long-term dependence and interaction with each other, the time sequence characteristics can be deeply characterized, and the Transformer model has excellent performanceAnd due to the parallel computing capability, various abnormal data points in the dynamic sequence can be efficiently and accurately captured. In addition, when the electronic device 1000 invokes the transform model to reconstruct the first data through the first sequence based on the method of this embodiment, in addition to the first sequence, self-adjusting data is introduced into the model input, and the data reconstruction is completed in two stages in combination with the updating of the self-adjusting data, which effectively improves the data reconstruction capability of the model for various types of dynamic sequences, thereby realizing the accurate capture of abnormal data with various forms and improving the robustness of data abnormality detection.

On the other hand, the time series data reconstruction model based on the Transformer can perform self-supervision learning based on the dynamic sequence serving as the detection object, does not need to label data, is easy to implement on the ground in an actual application scene, and greatly reduces the training cost.

In some embodiments, the above first data

And a first array

The plurality of data may be normalized to data between 0 and 1, and the difference between the data can be amplified through appropriate normalization processing, which is beneficial to more accurate abnormality capture, and further improves the accuracy of abnormal data detection through the above steps S210 to S250. In these embodiments, the acquiring the first data corresponding to the first timestamp and the first sequence corresponding to the first time window of the detection object in step S210 may include the following steps: acquiring first original data corresponding to a first timestamp in a detection object and a first original sequence corresponding to a first time window; and normalizing the first original data and the first original sequence through the set normalization index data to respectively obtain first data corresponding to the first original data and a first sequence corresponding to the first original sequence.

In these embodiments, the first data is corresponded to

May be represented as

Corresponding to the first sequence

Can be expressed as

。

The above normalized index data may include a normalized index value of the detection object in each data dimension, for example, if the data of the detection object is two-dimensional data, the first dimension represents the number of persons on line, and the second dimension represents the flow rate, the normalized index data may include a normalized index value in the first dimension and a normalized index value in the second dimension.

The above normalization index data may be determined according to a normalization method to be adopted. For example, in the maximum-minimum normalization, the normalized index value for each dimension includes a maximum value and a minimum value of the detection object in the corresponding dimension. For another example, in the standard normalization, the normalized index value for each dimension includes a mean and a standard deviation of the detection object in the corresponding dimension.

The normalized index data may be based on a second raw sequence of numbers in the test object corresponding to a second time window

And (4) determining. The second time window is also earlier than the first time stamp, and the second original sequence of numbers

Is greater than the first original sequence

N, in the data. By detecting a second origin of the objectArray of numbers

The normalization index data is predetermined, which is beneficial to improving the first original sequence

The processing speed of the normalization processing.

To obtain suitable normalized index data to more effectively amplify the difference between the data, the second original sequence may be first obtained

Filtering the obvious abnormal data to obtain a second original sequence after filtering

And based on the filtered second original sequence

Determining normalized index data, e.g. determining a second original sequence

The maximum value and the minimum value in each dimension are used as normalized index data and the like.

When filtering the significant anomaly data, the filtering can be performed based on a set anomaly data range, which is represented by a significant anomaly upper limit value and a significant anomaly lower limit value, in the second original sequence

That is, the data is greater than the significant anomaly upper limit value or less than the significant anomaly lower limit value, which means that the data is significantly abnormal data, and the data is not within the anomalous data range, that is, the data is greater than or equal to the significant anomaly lower limit value and less than or equal to the significant anomaly upper limit value, which means that the data is significantly abnormal dataAre not considered significantly anomalous data.

In some embodiments, the anomalous data range can be based on the second raw sequence of numbers

Determining, for example, may be for a second original series of numbers

And calculating to obtain an upper limit value and a lower limit value of the significant anomaly by adopting statistical criteria such as K-sigma and Boxplot (box diagram), and further obtaining an abnormal data range indicating whether the data of the detection object is significantly abnormal.

The electronic device 1000 may sequence the above according to the second original number

The obtained abnormal data range and the normalization index range are stored in the significant abnormality judgment model, so that the electronic device 1000 can be used for performing abnormality detection on the first data serving as the real-time data point, and the detection efficiency and the detection accuracy are further improved.

In these embodiments, the first data is determined in step S250 above

The exception data is also: determining to correspond to the first data

First original data of

Is the exception data.

Since the above abnormal data range can be used to capture the significant abnormal data in the detection object, in some embodiments, for the real-time data point of the detection object, it can be determined whether it is abnormal data based on the above abnormal data range, and if the real-time data point is not significantly abnormal, then more detailed abnormal detection is performed through the above steps S210 to S250, which can effectively improve the detection efficiency. In these embodiments, as shown in fig. 4, the corresponding data anomaly detection method may include the following steps:

step S410, acquiring first original data of the detection object corresponding to the first timestamp and a first original sequence corresponding to the first time window.

Since the data in the first original sequence is the data that has undergone the abnormal detection before the first time stamp in the first time window, the electronic device 1000 may select the normal data that has undergone the abnormal detection to form the first original sequence, so as to improve the effectiveness of the reconstruction, that is, the data in the first original sequence is the normal data that has been previously determined through the steps shown in fig. 4.

Step S420, detecting whether the first raw data is within a set abnormal data range, and if the first raw data is not within the abnormal data range, performing step S4310; and under the condition that the first original data is in the abnormal data range, directly determining that the original data is abnormal data.

Step S4310, normalizing the first original data and the first original sequence by the set normalization index data to obtain first data corresponding to the first original data and a first sequence corresponding to the first original sequence, respectively, and then performing step S4320.

Step S4320, splicing the first sequence with the initial adjustment data, inputting the spliced first sequence into a preset first transform model to obtain first reconstruction data of the first data, and then executing step S4330.

Step S4330, updating the self-adjusting data according to a first residual between the first data and the first reconstruction data, splicing the first sequence and the updated self-adjusting data, and inputting the spliced first sequence and updated self-adjusting data to a preset second transform model to obtain second reconstruction data of the first data, and then executing step S4340.

Step S4340, detecting whether a second residual between the first data and the second reconstructed data exceeds a set residual range, and determining the first data as abnormal data when the second residual exceeds the set residual range.

Correspondingly, in the case that the second residual is within the range of the residual, the first data is determined to be normal data.

In these embodiments, in the case that the first raw data is within the abnormal data range, the detection of the first raw data may be ended without performing more elaborate abnormal detection on the first raw data through the above steps S4310 to S4340. In these embodiments, steps S4310 to S4340 correspond to steps S210 to S240, and steps S4310 to S4340 can be understood by referring to steps S210 to S240, which is not described herein again.

In these embodiments, the anomaly detection performed according to the anomaly data range corresponds to a coarse detection for a significant anomaly, while the anomaly detection performed by the time-series data reconstruction model corresponds to a fine detection for a non-significant anomaly, and since the calculation amount and time consumption of the fine detection are higher than those of the coarse detection, the detection efficiency for the anomaly data can be effectively improved by the combination of the coarse detection and the fine detection.

The time sequence data reconstruction model can be obtained by training based on data of a detection object, so that the data reconstruction capability of the time sequence data reconstruction model is continuously improved through the self-supervision learning without labeled data. Therefore, in some embodiments, before the step S210 of acquiring the first data corresponding to the first timestamp and the first sequence corresponding to the first time window of the detection object, model parameters of the updated time series data reconstruction model, which include the first model parameters of the first transform model and the second model parameters of the second transform model, may be trained through the sample set to perform anomaly detection on the first data through the updated time series data reconstruction model. In these embodiments, the method further includes the following steps S2011 to S2013:

in step S2011, a sample set is acquired.

The sample set comprises a plurality of samples, wherein the jth sample comprises jth sample data of a corresponding jth time stamp of the detected object

And a j-th sample sequence corresponding to a j-th time window,and the jth timestamp is behind the jth time window, different samples in the sample set correspond to different timestamps and different time windows, and the jth timestamp is before the first timestamp ti, wherein j takes a value in a range which is greater than or equal to 1 and less than or equal to N, and N is the number of samples in the sample set. The number of data of the j-th sample number series may be the same as the number of data n of the first number series.

In some embodiments, the sample set may be based on the filtered second raw sequence of numbers described above

And (4) obtaining. This may be based on the filtered second original sequence

After the normalized index data is obtained, the filtered second original number sequence is compared based on the normalized index data

Performing normalization processing to obtain a second series

Then sliding window to second array

Carrying out segmentation to obtain a sample set; wherein the sliding window has a fixed window length, and the sliding window is arranged from the second array

Starting with a set step size towards the second array

The sliding direction may be a direction from the earliest timestamp to the latest timestamp, or a direction from the latest timestamp to the earliest timestamp, which is not limited herein, and then the second sequence is completed

The cutting of (2). For example, as shown in FIG. 5, when the sliding window is at the first position (shown by the solid line) from the first end, the latest data in the time stamp in the range of the sliding window

As the 1 st sample data

Other data at the sliding window form a 1 st sample sequence, and then the sequence is arranged to a second sequence according to a set step length

The set step length may be 1 data bit or multiple data bits, and is not limited herein, so that the sliding window slides to the second position reached by the dotted line, and the data with the latest timestamp in the range of the sliding window

As the 2 nd sample data

And other data in the sliding window range form a 2 nd sample sequence, and by analogy, a plurality of samples can be obtained.

Step S2012 trains the first Transformer model and the second Transformer model through the sample set and the set loss function, and determines a first model parameter of the first Transformer model and a second model parameter of the second Transformer model.

Since the time series reconstruction model performs a two-stage reconstruction with two transform models, the loss function may include a weighted sum of a first loss function corresponding to the first transform model and a second loss function corresponding to the second transform model. The first loss function represents a first loss of first reconstruction data of the sample data relative to the sample data, the second loss function represents a second loss of second reconstruction data of the sample data relative to the sample data, the first reconstruction data of the sample data is obtained by reconstructing the sample data through a corresponding sample number sequence by the first transform model, and the second reconstruction data of the sample data is obtained by reconstructing the sample data through a corresponding sample number sequence by the second transform model.

In each iterative training, only one sample may be involved in the training, for example, the jth sample may be involved in the jth iterative training, or a plurality of samples may be involved in the training. In the case where only one sample is involved in the training at a time, the above first loss and second loss are the losses incurred for that sample. And in the case where a plurality of samples are involved in the training at a time, the above first loss and second loss may be an average loss of the plurality of samples. In this embodiment, the number of samples for each iterative training and the number of samples involved in the loss calculation may be set as needed, and are not limited herein.

On the basis, in order to enhance the robustness of the model, the weight of the first loss function is set to be reduced along with the increase of the training times, and the weight of the second loss function is set to be increased along with the increase of the training times. For example, the loss function

Can be expressed as:

；

in formula (3), λ represents the number of training iterations,

is a first loss function

The weight of (a) is calculated,

as a second loss function

The weight of (2).

Taking as an example that N samples in the sample set all participate in training and loss calculation in each iterative training, the first loss of the jth sample can be calculated according to the jth sample data

And the first reconstructed data

To determine the distance between them, e.g. first loss function

Can be expressed as:

；

similarly, taking the example that in each iterative training, N samples in the sample set all participate in training and loss calculation, the second loss of the jth sample can be calculated according to the jth sample data

With second reconstructed data

Determining the distance between them, e.g. second loss function

Can be expressed as:

。

in some embodiments, to improve the model training efficiency, the training may be iteratively performed once per pass through one sample of the sample set until the training is complete. This may be: performing first iterative training through a 1 st sample, then obtaining the loss of the first iterative training according to a loss function, adjusting model parameters according to the loss of the first iterative training and strategies such as backstepping Propagation (backstepping Propagation) and the like, and performing second iterative training according to the adjusted model; and then, performing second iterative training through a 2 nd sample, obtaining the loss of the second iterative training according to a loss function, continuously adjusting model parameters according to the loss of the second iterative training and strategies such as backstepping Propagation and the like, performing third training according to the adjusted model, and the like until the set convergence condition is met, and stopping training.

In these embodiments, the training of the first fransformer model and the second fransformer model by the sample set and the set loss function in step S2012, and the determining of the first model parameters of the first fransformer model and the second model parameters of the second fransformer model may include the following steps S2012-1 to S2012-4:

step S2012-1, in the lambda-th iteration training, training a first Transformer model and the second Transformer model through a sample set sample, and obtaining the loss of the lambda-th iteration training according to a loss function; wherein λ is an integer greater than or equal to 1.

In the lambda-th iterative training, all samples or part of samples in the sample set can be selected for training.

Taking the jth sample in the sample set participating in the lambda-th training as an example, referring to fig. 3, training the first Transformer model and the second Transformer model by the jth sample, and obtaining the loss of the jth sample according to the loss function may include: splicing the jth sample sequence with the initial adjustment data and inputting the spliced jth sample sequence into a first transform model to obtain jth sample data

First reconstructed data of

(ii) a Sample data through jth

And first reconstructed data

Updating self-adjusting data by residual errors, splicing the jth sample sequence and the updated self-adjusting data, and inputting the spliced jth sample sequence and the updated self-adjusting data into a second transform model to obtain jth sample data

Second reconstructed data of

(ii) a Sampling the jth sample data

First reconstructed data

Second reconstructed data

And inputting the current training iteration times lambda into the loss function to obtain the loss of the jth sample.

And obtaining the corresponding loss of each sample participating in the lambada iteration training according to the steps, and further obtaining the loss of the lambada iteration training according to the loss of each sample and referring to the formula (3) to the formula (5).

Step S2012-2 is to adjust the model parameters of the first transform model and the model parameters of the second transform model according to the loss of the lambda-th iterative training.

In step S2012-2, the model parameters may be adjusted according to a policy such as back Propagation (Backward Propagation).

Step S2012-3 stops training when the λ -th iterative training satisfies the set convergence condition, and determines the first model parameter and the second model parameter as the model parameters adjusted by the λ -th iterative training.

The convergence condition may be that the loss value is less than or equal to a set value, the number of times of training reaches a set number of times, or a rate of decrease in the loss value is less than or equal to a set rate, and the like, and is not limited herein.

And step S2012-4, when the lambda-th iteration training does not meet the convergence condition, performing the next iteration training through the sample in the sample set, namely returning to step S2012-1 for the next training.

Step S2013, the first Transformer model is updated according to the first model parameters, and the second Transformer model is updated according to the second model parameters.

After the first and second fransformer models are updated according to step S2013, steps S220-S230 may be performed based on the updated first and second fransformer models to reconstruct the first data more accurately through the first sequence.

In the embodiments, the time sequence data reconstruction model performs self-supervision learning without labeled data based on the data of the detection object, so that the data reconstruction capability of the time sequence data reconstruction model can be continuously improved in practical application, labeled data is not needed, implementation is easier, and training cost is greatly reduced. On the other hand, a 2-stage training process is introduced into model training, so that the robustness of the model can be enhanced, the model can process different types of dynamic data, and abnormal data with various forms can be captured.

In some embodiments, the filtered second raw number series is targeted based on a K-sigma criterion

The residual parameter values corresponding to the time labels may also be predetermined according to the time sequence periodicity characteristics of the detection object, so that the electronic device 1000 may call the residual parameter values when executing step S240. In these embodiments, after the step S2012 of training the first and second fransformer models by the sample set and determining the first model parameters of the first fransformer model and the second model parameters of the second fransformer model, the method may further include the following steps S2021 to S2024:

step S2021, acquiring at least a part of the sample data as calibration data, and acquiring second reconstruction data of the calibration data obtained in training the first transform model and the second transform model.

In the training process of the time series data reconstruction model, because the reconstruction precision in the later training stage is higher than that in the earlier training stage, the sample data participating in model training for the last times (the specific times can be set as required) can be selected as calibration data. Thus, in this step S2021, a plurality of calibration data will be acquired.

Step S2022, obtaining a residual error of the calibration data according to the second reconstruction data of the calibration data and the calibration data.

The second reconstruction data of the calibration data is the second reconstruction data obtained in the model training.

The residual of the calibration data reflects the proximity between the second reconstruction data of the reference data and the reference data, and the residual of the reference data may be represented by an L1 distance or an L2 distance, which is not limited herein.

Step S2023, obtaining a calibration data set corresponding to each time tag in the set of time tags.

Through the step S2023, a plurality of calibration data sets are obtained, and the plurality of calibration data sets correspond to the plurality of time tags in the time tag set one to one.

Step S2024, obtaining and storing a residual parameter value corresponding to the time stamp according to the residual of the calibration data in the calibration data set.

According to the residual errors of the calibration data in the calibration data set, the residual error mean value and the residual error variance of the calibration data can be calculated and obtained to be used as the residual error parameter values of the corresponding time labels.

The electronic device 1000 may store the mapping data reflecting the mapping relationship between the residual parameter value and the time stamp obtained in steps S2021 to S2024 in the residual anomaly detection model, so that the electronic device 1000 may perform anomaly detection on the first data, which is the real-time data point, in steps S210 to S250.

In some embodiments, referring to fig. 4 and 6, the electronic device 1000 includes the following steps S610 to S630 when performing anomaly detection on the first raw data as a real-time data point:

step S610, invoking an abnormal data range in the significant anomaly detection model MS, detecting whether the first raw data is significantly abnormal data, if so, outputting a detection result indicating that the first raw data is abnormal data, otherwise, performing normalization processing on the first raw data and the first raw sequence through normalization index data in the significant anomaly detection model to obtain first data and a first sequence, and then executing step S620.

Step S620, invoking the time series data reconstruction model MT to reconstruct the first data through the first sequence to obtain second reconstructed data of the first data, and then executing step S630.

Step S630, obtaining a residual parameter value corresponding to the first timestamp from the residual anomaly detection model MP, obtaining a corresponding residual range, and detecting whether a second residual between the first data and the second reconstructed data is within the residual range, if so, determining that the first original data is normal data, otherwise, outputting a detection result indicating that the first original data is anomalous data.

The significant anomaly detection model MS, the time-series data reconstruction model MT and the residual anomaly detection model MP used in the above steps S610 to S630 may pass through the second original sequence corresponding to the second time window

The updating is trained in advance, and the updating operation can be executed according to the set updating period so as to improve the robustness of the model. Pretrained updates through a second raw sequence

Then, as shown in FIG. 6, the second original sequence can be first obtained

Determining an anomaly RangeIndexes (including a significant anomaly upper limit value and a significant anomaly lower limit value) and normalized index data, and updating a significant anomaly detection model MS; then, referring to steps S2011-S2013, a second raw sequence is filtered based on passing the abnormal range index

The second filtered original sequence obtained from the significant anomaly data in (1)

Training the time sequence data reconstruction model MT, and updating the model parameters of the time sequence data reconstruction model MT according to the training result; finally, referring to steps S2021 to S2024, a residual parameter value of the residual anomaly detection model MP is obtained and updated according to the second residual of the training sample obtained by training, which is not described herein again.

In other embodiments, an electronic device for implementing the data anomaly detection method according to any embodiment of the present disclosure is further provided, and fig. 7 shows a schematic hardware structure diagram of the electronic device 700. As shown in fig. 7, the electronic device 700 may include a memory 720 and a processor 710, the memory 720 being used for storing a computer program, and the processor 710 being used for executing a data anomaly detection method according to any embodiment of the present disclosure under the control of the computer program. The electronic device may be a server, a terminal device, or the like, and is not limited herein.

Additionally, in further embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements a data anomaly detection method according to any of the embodiments of the present disclosure.

All the embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, as for the device and apparatus embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference may be made to some descriptions of the method embodiments for relevant points.

The foregoing description of specific embodiments has been presented for purposes of illustration and description. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

Embodiments of the present description may be an apparatus, method, and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to implement aspects of embodiments of the specification.

The computer-readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as a punch card or an in-groove protruding structure with instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.

The computer program instructions for carrying out operations for embodiments of the present description may be assembly instructions, instruction Set Architecture (ISA) instructions, machine related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, the electronic circuitry can execute computer-readable program instructions to implement various aspects of embodiments of the specification by personalizing, with state information of the computer-readable program instructions, a custom electronic circuit, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA).

Aspects of embodiments of the present specification are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices) and computer program products according to embodiments of the specification. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present description. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are equivalent.

The foregoing description of the embodiments of the present specification has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A data anomaly detection method is characterized by comprising the following steps:

splicing the first sequence and initial self-adjusting data, and inputting the spliced first sequence and initial self-adjusting data into a preset first converter model to obtain first reconstruction data of the first data, wherein the initial self-adjusting data is a set value;

determining the first data as abnormal data if the second residual is outside the range of residuals,

wherein the updating the self-alignment data by a first residual between the first data and the first reconstructed data comprises: and updating the self-adjusting data from the set value to the first residual error.

2. The method of claim 1, wherein the first converter model and the second converter model have the same encoder.

3. The method of claim 1, wherein prior to acquiring the first data corresponding to the first timestamp and the first sequence corresponding to the first time window of the test object, the method further comprises:

4. The method of claim 3, wherein after obtaining the first raw data corresponding to the first timestamp in the detected object, the method further comprises:

5. The method of claim 1, wherein before detecting whether a second residual between the first data and the second reconstructed data exceeds a predetermined residual range, the method further comprises:

acquiring a first time tag corresponding to the first timestamp;

acquiring a first residual error parameter value corresponding to the first time tag; the first time label is one time label in a time label set, and different time labels in the time label set correspond to different residual parameter values;

6. The method of any of claims 1 to 5, wherein prior to acquiring the first data corresponding to the first timestamp and the first sequence corresponding to the first time window of the test object, the method further comprises:

obtaining a sample set; wherein, the jth sample in the sample set comprises jth sample data corresponding to a jth timestamp of the detection object and a jth sample sequence corresponding to a jth time window, the jth time window is before the jth timestamp, different samples in the sample set correspond to different timestamps and different time windows, and the jth timestamp is before the first timestamp;

training the first converter model and the second converter model through the sample set and a set loss function to obtain a first model parameter of the first converter model and a second model parameter of the second converter model;

7. The method of claim 6, wherein the loss function comprises a weighted sum of a first loss function corresponding to the first converter model and a second loss function corresponding to the second converter model, wherein a weight of the first loss function decreases as the number of training iterations increases, and wherein a weight of the second loss function increases as the number of training iterations increases;

8. The method of claim 6, wherein the obtaining the sample set comprises:

filtering the data with remarkable abnormality from the second original sequence according to the abnormal data range to obtain a filtered second original sequence;

9. The method of claim 6, wherein after training the first and second converter models through the sample set and the set loss function to obtain first model parameters of the first converter model and second model parameters of the second converter model, the method comprises:

acquiring at least part of sample data as calibration data, and acquiring second reconstruction data of the calibration data obtained in training the first converter model and the second converter model;

obtaining and storing a residual parameter value of the corresponding time label according to the residual of the calibration data in the calibration data set;

10. An electronic device, comprising a memory for storing a computer program and a processor for executing the data anomaly detection method according to any one of claims 1 to 9 under the control of the computer program.

11. A computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements a data anomaly detection method according to any one of claims 1 to 9.