CN117950912A - High-robustness distributed data recovery method based on compressed sensing - Google Patents

Abstract

The present invention relates to a highly robust distributed data recovery method based on compressed sensing. The method utilizes the inherent characteristics of the original data that may have multiple defects, converts the data recovery process into an optimization problem, decomposes the data matrix, noise matrix, abnormal matrix and error matrix from the original data by designing the goal and constraints of the problem, and designs a set of distributed algorithms to greatly reduce the demand for computing power, thereby achieving highly robust and high-precision rapid recovery of data; in addition, a method for balancing recovery effect and recovery time is also provided, which directly affects and predicts the final recovery performance by adjusting the division scheme of the decomposition stage. In general, the data recovery method can be divided into three parts: distributed data decomposition, highly robust data recovery, and iterative data enhancement. Compared with the prior art, the present invention has the advantages of high robustness, strong adjustability, and easy use.

Description

A highly robust distributed data recovery method based on compressed sensing

Technical Field

The present invention relates to the technical field of data recovery, and in particular to a highly robust distributed data recovery method based on compressed sensing.

Background Art

Data is the bridge between the Internet of Things and the real world. Different from the traditional network model where data directly provides services to people, in the Internet of Things, devices are the main producers and consumers of data. In this M2M model, data is used as a communication medium between devices to provide ubiquitous intelligent services. However, due to the characteristics of the Internet of Things itself, such as the limitations of energy and volume, IoT devices cannot provide strong computing power; due to the limitations of scenarios and sensor performance, the Internet of Things often uses a simpler network protocol stack, and its wireless transmission process is more susceptible to electromagnetic interference, and there is a higher risk of intermittent disconnection in network connections. If the impact of these IoT characteristics on data quality is not fully considered, then subsequent operations and services based on IoT data will be extremely unreliable, which may cause immeasurable losses to upper-layer applications and businesses. Therefore, special attention must be paid to data quality.

In view of the shortcomings of insufficient computing power in the Internet of Things, the perception data collected by sensors are often processed using edge networks. However, the data collected by sensors often have multiple flaws and defects at the same time, and most algorithms do not allow data gaps or outliers in the input. In order to avoid trouble in subsequent analysis and processing, a common practice is to perform preprocessing of data recovery, but the existing data recovery algorithms have many limitations.

First, existing data recovery algorithms generally do not fully consider the possible defects in the data, including anomalies, missing data, and noise. If these defects in the data are not handled, the performance and accuracy of subsequent data analysis will be seriously reduced.

Secondly, the operation time of existing data recovery algorithms is too long. There are many large-scale data sets in the network field. This is because the cost of data collection is low and the speed is fast. It is relatively easy to obtain data with storage capacity of GB level. However, for data recovery algorithms, such large-scale data will pose a serious challenge to node computing power. But when computing power becomes a bottleneck, data recovery will take a lot of time, resulting in extremely high latency, which will deteriorate the real-time nature of data analysis.

Summary of the invention

The purpose of the present invention is to provide a highly robust distributed data recovery method based on compressed sensing. Combined with relevant scenario characteristics, it can complete the detection and recovery of various defects in the data on the Internet of Things side, so as to achieve the purpose of enhancing data quality and minimize the impact of Internet of Things data defects on higher-level services.

The purpose of the present invention can be achieved by the following technical solutions:

A highly robust distributed data recovery method based on compressed sensing, the method converts the data recovery process into an optimization problem, through the design of the problem objectives and constraints, decomposes the original data into a data matrix, a noise matrix, an abnormal matrix and an error matrix, and obtains the recovered data according to the data matrix.

The method is implemented based on a distributed algorithm. The master node executes a data decomposition process and a data enhancement process. The data decomposition process splits the original data into multiple sub-matrices. The data enhancement process optimizes the decomposition results through iteration. The slave node executes a data recovery process on the sub-matrices split by the master node.

The method comprises the following steps:

S1, distributed data decomposition:

S101, the master node obtains the original data to be restored and checks the low rank of the original matrix. If it meets the low rank requirement, the next step is executed, otherwise an error is reported;

S102, according to pre-configured parameters, the master node clusters the original data to split the matrix, and the data in the same cluster is spliced into a sub-matrix, thereby obtaining multiple sub-matrices;

S103, the master node sends the split sub-matrices to multiple slave nodes evenly with the matrix size as the weight for data recovery;

S2, highly robust data recovery:

S201, each slave node extracts all vacant positions from the corresponding sub-matrix to construct an error matrix;

S202, constructing the goal and optimization constraints of the optimization problem according to the defect characteristics in the data;

S203, bringing in the constraint term to obtain the augmented Lagrangian formula, minimizing the corresponding augmented Lagrangian formula for each slave node using the ADM method, solving the optimization problem, and obtaining the decomposed data matrix, noise matrix, and abnormality matrix;

S3, iterative data augmentation:

S301, after the slave nodes complete the matrix decomposition, the master node summarizes the decomposition results of all sub-matrices on each slave node, and reassembles them according to the split positions to obtain 4 matrices consistent with the input original data scale, including the restored data matrix;

S302, clustering the restored data matrix again to obtain the split sub-matrices, and the master node compares the difference between the current split result and the previous split result. If the difference is less than the threshold, the recovery is completed, and the four matrices spliced in step 301 are used as the recovery result, otherwise proceed to the next step;

S303, based on the pre-configured parameters, the master node determines whether the calculation time and the number of iterations have reached the maximum value. If so, the recovery is completed and the four matrices obtained by splicing in step 301 are used as the recovery result. Otherwise, the sub-matrices obtained by clustering again in step S302 are returned to step S103 to continue the next round of recovery.

The step S101 is specifically as follows: the master node obtains the original data to be restored, and temporarily fills all missing positions of the original data with the value 0, performs singular value decomposition on the filled original data matrix to calculate the rank of the data components therein, and determines the low-rank index; if the calculated low-rank index is less than the preset threshold, execute the next step, otherwise return an error value to report an error.

In step S102, the master node clusters the original data to split the matrix using the density-based improved clustering algorithm OPTICS. On the basis of the OPTICS algorithm, the determination of the end point of the region is added, and the characteristic of the minimum heap structure itself maintaining the pioneer node in the process of algorithm generation sequence is used to determine whether the end point should be removed from the cluster; after clustering is completed, all clusters are arranged in descending order according to size and marked with id starting from 1, that is, the cluster with id=1 contains the most data points, and all marked outliers are aggregated into an additional cluster, and id=0 is recorded.

In step S103, the master node sends each sub-matrix to the subordinate slave nodes using the network connection, and uses the square of the matrix size as the weight and the load factor to distribute tasks among the slave nodes. The process is as follows:

Pre-configure the weight w _j for the jth node according to its computing power. The default value is 1. The initial load of the node is L _j = 1. Define the load factor

For the submatrix D _i corresponding to cluster C _i , the task volume T _i is defined as the square of the scale, that is,

T _i =(M _i· col) ²

Select the node that currently has the largest load factor and update the load factor, that is,

L′ _Node = L _Node + _Ti

Repeat the above steps until the assignment is complete.

The assumption is that a total of N sub-matrices are divided, where the sub-matrix M _i corresponding to the cluster with id=i is of size m _i ×n _i . Step S2 decomposes it into a low-rank data matrix A _i , a sparse anomaly matrix B _i , a small noise matrix C _i and an error matrix E _i . The optimization problem is expressed as:

minimize:

subject to:

Di _· *E _i = _Di

Where α, β, σ are the coefficients of the three corresponding matrix components to meet the needs of linear transformation and dimensional conversion; X, Y, Z are global coefficient matrices; _Di is an intermediate variable of the same size as _Ei that is used to convert _Ei from a binary matrix to a numerical matrix; and It is an additional auxiliary variable used to decouple the _Ai and _Bi variables in the objective and constraints; is the penalty term, where γ is the corresponding coefficient, K is the number of penalty terms, and U ^k , V ^k and R ^k are three global matrix parameters set according to domain knowledge.

In S302, the calculation method used to compare the difference between the current split result and the previous split result is:

Wherein, Lab_Diff represents the obtained partition difference result, N is the number of sub-matrices, Labt(i) represents the split result of the t-th iteration, and Labt-1(i) represents the split result of the previous iteration before the t-th iteration.

The method predicts data recovery performance by adjusting the partitioning scheme in the data decomposition stage, and the data recovery performance includes recovery effect and recovery time, wherein the recovery effect is predicted according to the number of sub-matrices, and the recovery time is predicted according to the sum of squares of the sub-matrix sizes.

The prediction method of the recovery time is:

Where N is the number of sub-matrices to be split, k′ is the time coefficient, m×n is the input matrix size, Square_Sum is the sum of squares of cluster sizes,

The recovery effect is measured using the normalized mean absolute error (NMAE), defined as

Where A represents the ideal data matrix, represents the restored data matrix outputted in step S3, and M represents the original matrix with defects.

Compared with the prior art, the present invention has the following beneficial effects:

(1) Compared with algorithms at the optimization level such as ADMM, the present invention mainly focuses on the splitting of the input data matrix rather than the optimization objectives and constraints. This splitting is more direct and has a more obvious effect on saving computing time. In addition, the present invention can also use this type of optimization algorithm to solve the problem during the iterative step.

(2) Compared with general distributed scheduling algorithms, the present invention can more fully consider the internal structure of the data matrix, avoid arbitrary splitting and destruction of the internal structure, and better play the synergy between nodes.

(3) Compared with edge computing, the present invention can better adapt to edge scenarios, and can choose to transmit part of the original sub-matrix to the edge node or the cloud for solution to make full use of computing power. However, in this scenario, additional optimization solutions may be required based on the influence of transmission overhead and multi-level node priority to better play the advantages of edge computing.

(4) High robustness: Unlike many recovery algorithms that only target specific defects, the present invention allows the original matrix to simultaneously contain multiple defects such as noise, missing values, and outliers, greatly improving the robustness of the recovery.

(5) Strong adjustability: The present invention can predict the recovery time and effect at the first partitioning, and users can flexibly adjust clustering parameters to balance the contradiction between data recovery effect and running time. In addition, the present invention allows users to limit the maximum running time and number of iterations, providing more options for balancing recovery performance and computing resources.

(6) Easy to use: On the one hand, some parameters in the present invention can be adaptively adjusted, reducing the complexity of configuration; on the other hand, users can customize the clustering method when dividing according to the specific data source, or use a general division scheme, which can meet the needs of most scenarios.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a flow chart of the present invention;

Fig. 2 is a diagram of specific implementation steps of the present invention;

FIG3 is a pseudo code of a clustering part in an embodiment;

FIG. 4 is a graph comparing recovery performance under different error rates in an embodiment.

DETAILED DESCRIPTION

The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

In view of the defects of the prior art, the present invention makes full use of the structure of the edge network and develops a set of distributed data recovery algorithms in combination with the existing data recovery algorithm LENS. On the edge server, the perception data collected by the edge node will be collected, rearranged, and distributed according to certain rules, and the original data matrix will be divided into multiple original sub-matrices and passed to the edge nodes connected to it. At each edge node, each original sub-matrix will be accurately split into a low-rank data matrix, a sparse anomaly matrix, a small noise matrix, and an error matrix. Such a split naturally reflects the inherent structure of data in the real world, clearly reflects and expresses the three defects that may exist in the data, and can improve data quality more intuitively and clearly than the existing compressed sensing technology. Afterwards, this embodiment shows that this problem is a convex optimization problem, and uses an alternating descent method ADM to solve it.

The present invention is a highly robust distributed data recovery method based on compressed sensing, named LADEN. This method utilizes the inherent characteristics of the original data that may have multiple defects, converts the data recovery process into an optimization problem, and decomposes the data matrix, noise matrix, abnormal matrix and error matrix from the original data through the clever design of the problem's objectives and constraints, and designs a set of distributed algorithms to greatly reduce the demand for computing power, thereby achieving high robustness and high accuracy of data rapid recovery. In addition, LADEN also provides a method for balancing recovery effect and recovery time for users to choose: It has been proved through theory and experiments that the two sets of data, the recovery effect and the number of sub-matrices, and the recovery time and the sum of the squares of the sub-matrix sizes, are highly linearly related. Therefore, the final recovery performance of LADEN can be directly affected and predicted by adjusting the division scheme in the decomposition stage. Figure 1 is a flow chart of the present invention.

As can be seen from the schematic diagram, the present invention is mainly aimed at distributed systems. The "sensor" represents the data source, and the main logic and steps of LADEN are in the master node and the slave node. There is only one master node, which is responsible for executing step S1 data decomposition and S3 data enhancement, and there may be multiple slave nodes, which are responsible for executing step S2 data recovery. It should be noted that the nodes here are only used as a distinction for logical execution. A node can be either a thread or a server, which does not affect the correctness of the algorithm.

The name LADEN of the present invention is taken from the first letters of five words, including: Low-rank, Anomaly, Distributed, Error, and Noise, which are also the keywords of the present invention.

Specifically, as shown in FIG2 , the LADEN method comprises the following steps:

S1, distributed data decomposition

S101, after the master node obtains the original data M0 to be restored, it first temporarily fills the empty positions with 0, and then checks whether the low rank of the original data meets the requirements through SVD decomposition. If it meets the low rank requirements, it executes the next step, otherwise it reports an error and issues a warning to remind the user.

Specifically, in the initialization phase, the master node will be responsible for receiving the original data that needs to be restored, which is often full of flaws. In order to perform the recovery operation more accurately later, the master node will first temporarily fill all missing positions with the value 0, and then perform singular value decomposition (SVD) on the filled original data matrix M0. Singular value decomposition is a matrix decomposition method widely used in linear algebra and numerical analysis. By retaining some singular values and corresponding singular vectors, the original matrix can be approximately reconstructed to achieve feature extraction of the data and calculate its rank. The decomposition form is as follows:

M＝UΣV ^T

Among them, U and V are both unitary matrices, U is a column orthogonal matrix, V is a row orthogonal matrix, the elements on the main diagonal of the matrix Σ are singular values, and the rest are 0. Due to the various defects in the original data, especially the influence of noise and outliers, the rank of the matrix M calculated by the traditional method will be significantly larger and distorted. Therefore, the singular values can be sorted from large to small, and the number of singular values occupying the top σ intensity is defined as the rank _σ (M) of the matrix M, and σ is a ratio close to 1. Without loss of generality, this embodiment selects σ=90%. Afterwards, the Low_Rank value is defined as the ratio of the matrix rank to the number of matrix columns, which is used to measure the strength of the low rank of the data matrix, that is, the low rank index.

S102, according to the pre-configured parameters, the master node clusters the original data M0 to split the matrix, and the data in the same cluster is spliced into a sub-matrix, thereby obtaining multiple sub-matrices, and the division result is recorded as Lab0.

Assume that the input is a matrix M of size m×n. In the Internet of Things, such data often comes from the collection of n sensors at m moments. This data can also be regarded as a collection of m n-dimensional data points. Clustering, as a typical unsupervised learning method, is very suitable for dividing point sets according to similar features, which helps to discover hidden structures and understand patterns in data, and can ensure the low-rank characteristics of the sub-matrices after division.

Although the current mainstream clustering algorithms can be used for splitting, LADEN provides a more general and efficient clustering algorithm. The partitioning scheme in LADEN mainly refers to the density-based clustering algorithm OPTICS. The main idea is to construct a reachability graph by calculation to identify the clustering structure, which is particularly suitable for processing clusters with different densities and sizes. Its feature is that it does not need to pre-specify the number of clusters, but automatically determines the clusters based on the distribution of the data. In addition, it also has the advantages of being relatively insensitive to noise and outliers, and adaptive to the size and shape of the cluster.

In the LADEN clustering algorithm, it is necessary to specify the minimum number of neighbors MinPts and the maximum neighborhood radius MaxEps. Based on these two parameters, the points in the point set can be divided into three categories: core points, boundary points, and outliers. If a point has at least MinPts points within a neighborhood radius not exceeding MaxEps, then the point is considered to be a core point. If a point is not a core point, but there are core points in the neighborhood, then the point is considered to be a boundary point. The remaining points are considered to be outliers that do not belong to any cluster. Afterwards, it is necessary to quantify the reachability between points, which is defined as the larger value of the distance between the two points and the "core distance" of the starting point o, that is,

Reachability(o→p)=max{Distance(o,p),Core(o)}

The core distance is only valid for core points, which refers to the minimum neighborhood radius that makes the number of neighbors of the core point equal to MinPts, otherwise it is infinite ∞, that is,

From the above two formulas, we can see that the core distance is a property of the point itself under the distance parameters MinPts and MaxEps, while the reachability is a distance measure from one point to another, which is asymmetric. The smaller the reachability, the greater the density of the points.

After defining these two distances, this embodiment uses a minimum queue, which can approximate the density clustering structure of the data set by the reachability measurement. It finds the point with the lowest current reachability from the dense area according to the greedy principle, that is, the point with the highest expected density. This point is transferred from the minimum queue to the end of the current cluster sequence, and then the reachability of its neighbors is updated in the minimum queue. Iterate until the minimum queue is empty, and you can get an augmented cluster sequence sorted by reachability.

In the augmented cluster sequence, clusters can be found in order of density from high to low, and the corresponding neighborhood radius always increases gradually from small to large. The main method is to use the quantile of global reachability as a threshold to split the sequence. A few points above the threshold are judged as outliers, and the rest below the threshold will form multiple discontinuous areas, each of which represents a "valley" of reachability. The low reachability in the valley represents a high-density cluster. Here, this embodiment optimizes the determination of the end point of the region, making full use of the characteristic of the minimum heap structure itself to maintain the pioneer node in the process of generating the sequence to determine whether the end point should be removed from the cluster. This optimization can correct incorrect clustering on boundary conditions.

It should be noted that the clustering order of the augmented cluster sequence naturally makes it easy to distinguish clusters of different densities, making the adaptability of the neighborhood radius possible. Therefore, there is no need to explicitly specify the neighborhood radius, but only to limit the maximum neighborhood radius MaxEps. Note that the number of clusters increases monotonically with MaxEps. Under the guarantee of monotonicity, if the purpose is to find enough clusters, MaxEps can be set to positive infinity, and then MinPts can be adjusted according to the lower limit of the required cluster size. But without loss of generality, Minpts=2 is set by default in LADEN, and the division result is changed by changing the distance parameter Epsilon.

The pseudo code of LADEN clustering is shown in Figure 3.

After clustering, all clusters are sorted in descending order of size and labeled with ids starting from 1, i.e., the cluster with id=1 contains the most data points. All labeled outliers are also aggregated into an additional cluster with id=0. Afterwards, the division of all points in M0 can be recorded by vector Lab0, where Lab0(i)=id _j , indicating that the i-th point is assigned to cluster j.

S103, the master node sends the split sub-matrices to multiple slave nodes evenly with the matrix size as the weight for data recovery.

After the master node has completed the splitting of the original data matrix, it can use the network connection to send each sub-matrix to the subordinate slave nodes; the network connection includes but is not limited to wide area wireless network, wired network, etc. In order to ensure fairness among slave nodes and improve efficiency, the square of the scale (i.e. the number of rows) will be used as the weight, and the load factor will be used to distribute tasks among slave nodes. The main process is as follows:

In this embodiment, the weight w _j is pre-configured for the jth node according to the computing capacity, and the default value is 1. The initial load of the node L _j = 1, and the load factor is defined as

For the submatrix D _i corresponding to cluster C _i , the task volume T _i is defined as the square of the scale, that is,

T _i =(M _i· col) ²

Select the node that currently has the largest load factor and update the load factor, that is,

L′ _Node = L _Node + _Ti

Repeat the above steps until the allocation is completed. Since all clusters have been sorted in descending order above, the result obtained by this allocation strategy is approximately optimal, ensuring that the computing resources of the slave nodes can be utilized to the highest extent.

S2, highly robust data recovery

S201, each slave node extracts all vacant positions from the corresponding sub-matrix to construct an error matrix.

After receiving the sub-matrix, the slave node needs to consider three possible defects: errors, anomalies, and noise.

First of all, since the internal structure of the matrix is fully considered in the process of cluster splitting, under the premise that the input original matrix is approximately low-rank, the sub-matrix received in this step will also be approximately low-rank, which indicates the target direction for the recovery process. According to the premise assumptions and the verification of existing data sets, the data matrices collected by sensors from the physical world often show low-rank characteristics. In essence, such low-rank is a reflection of the redundancy of the data itself. The lower the rank of the matrix, the more redundancy it has, and the more beneficial it is for recovery. In terms of measurement, the intuitive method to calculate the rank is to use elementary transformations of the matrix. However, in order to reduce defects in the data and errors that may be caused by floating-point calculations, this embodiment still chooses to determine the rank of the matrix by calculating singular values through SVD decomposition, and the Low_Rank value can be used as an evaluation indicator for the data components in the sub-matrix. For sub-matrices with id greater than 0, their low rank will not be inferior to the original data matrix, that is

Low_Rank(M _i )≥Low_Rank(M)

For the convenience of expression, the sum of singular values is subsequently expressed in the form of a norm. Therefore, in this embodiment, the low-rank data matrix component in the i-th sub-matrix is recorded as ‖A _i ‖ _S .

After that, the characteristics of the three defects are analyzed. As mentioned above, outliers often come from emergencies. Compared with normal data under normal circumstances, such abnormal data must be special and rare, so the abnormal matrix composed of outliers must be a sparse matrix. Therefore, the sum of the absolute values of all elements in the matrix is used as the optimization target of the sparse component of the submatrix, and the sparse abnormal matrix is recorded in the form of the Manhattan norm of the matrix, that is, ‖B _i ‖ _M ＝∑|b|.

Another common defect is noise. The noise in IoT data mainly comes from random disturbances during the sensor acquisition process, including: electromagnetic interference, mechanical vibration, temperature and humidity and light changes, power supply noise, etc. During the data transmission process, more noise may also be introduced due to the transmission medium and data compression. Although the damage to the low-rank judgment can be reduced by discarding smaller singular values when calculating the Low_Rank value, the impact of noise must be fully considered in the recovery process to improve the accuracy of the data. Here, this embodiment uses a Gaussian noise model, and it is assumed that the sum of the above noises obeys a normal distribution. Then the overall amplitude of the noise in the submatrix must be small and ubiquitous, and there is a clear characteristic difference from the outliers. Therefore, the sum of the squares of all elements in the matrix can be used as the optimization target of the noise component of the submatrix, and the small noise matrix is recorded in the form of the square of the F norm (Frobenius Norm), that is,

Due to occasional sensor failures and packet loss during communication transmission, errors may occur in the data and are marked as NaN (Not a Number). Unlike anomalies and noise, the error position in the submatrix _Mi is clear, and its error component can be regarded as a known 0-1 binary matrix _Ei . In order to better represent the error matrix components in the constraint terms, it is necessary to introduce _Di, which is equal to _Ei, as an intermediate variable for converting from a binary matrix to a numerical matrix, and constrain it in the form of dot product, that is,

Di _· *E _i = _Di

Among them, if and only if a certain position in E _i is equal to 1, the corresponding position in D _i is not 0, which means that an error has occurred at this position. Therefore, the corresponding position in M _i can be set to 0 to remove the error component from the sub-matrix.

S202, constructing the goal and optimization constraints of the optimization problem according to the defect characteristics in the data.

According to the decomposition structure of step S201, suppose a total of N sub-matrices are divided, where the sub-matrix M _i corresponding to the cluster with id=i is of size m _i ×n _i , and it needs to be decomposed into a low-rank data matrix A _i , a sparse anomaly matrix B _i , a small noise matrix C _i and an error matrix E _i , then the following convex optimization problem can be preliminarily obtained:

minimize:

subject to:A _i +B _i +C _i +D _i =M _i

Among them, α, β, and σ are the coefficients of three corresponding matrix components to meet the needs of linear transformation and dimensional conversion.

Their default values are set to in Represents the standard deviation of the matrix _Mi.

In order to better meet and adapt to the changing needs of data recovery in different fields in reality, the present invention also makes two modifications. In the first part, considering that the low-rank, sparse and small-amplitude characteristics of the submatrix components are not necessarily explicit, it is necessary to allow the user to configure the corresponding linear change process separately according to existing experience to better capture the characteristics of the three components themselves. Formally, three global coefficient matrices are added, denoted as X, Y and Z. It should be noted that these three matrices do not have the subscript i to indicate that they are globally shared, but in actual calculations, adjustments will be made according to the size of the specific submatrix to meet the requirements of matrix multiplication. By default, X, Y and Z are all diagonal matrices.

In the second part, we need to add a penalty item to the target In order to give full play to the important guiding role of prior knowledge in specific fields in testing the underlying data structure, γ is the corresponding coefficient, K is the number of penalty terms, and U ^k , V ^k and R ^k are three global matrix parameters that can be set according to domain knowledge. By default, in order to better capture the local invariance of the data, K = 1 is set, U ¹ is a diagonal matrix, R ¹ is a zero vector, V ¹ is a constant diagonal matrix (Toeplitz Matrix) with the main diagonal of 1, the upper diagonal of -1, and the rest of the positions are 0. This default setting can reduce the numerical difference between adjacent positions in the data matrix _Ai , reflecting the local stability in the time domain. Similar to the first part, these parameters are also globally shared and can be adjusted according to the scale of _Ai .

In order to decouple the _Ai and _Bi variables in the objective and constraints, auxiliary variables are added and The following form is obtained:

minimize:

subject to:

Di _· *E _i = _Di

S203, the constraints are brought in to obtain the augmented Lagrangian formula, and each slave node uses the alternating direction method ADM to minimize the corresponding augmented Lagrangian formula to solve the optimization problem and obtain the decomposed data matrix, noise matrix and anomaly matrix.

Substituting the optimization objective in step S202 into the constraint term, the following augmented Lagrangian formula is obtained:

Where <P,A> represents the sum of the dot product of matrix P and A, _Pi , _Qi are all Lagrangian operators. In this way, the original optimization problem with equality constraints can be converted into an unconstrained optimization problem. When the augmented Lagrangian function takes the minimum value, its partial derivative must be 0, which must satisfy the original equality constraint.

For optimization problems There are many convex optimization solving tools available, such as CVX. This embodiment uses the alternating direction method ADM to solve the optimization problem. After convergence, three components in the submatrix _Mi can be obtained: low-rank data matrix _Ai , sparse anomaly matrix _Bi and small noise matrix _Ci .

After the solution is completed, the slave node returns the decomposition result to the master node.

S3, iterative data augmentation

S301, after the slave nodes complete the matrix decomposition, the master node summarizes the decomposition results of all sub-matrices on each slave node, and reassembles them according to the split positions to obtain 4 matrices consistent with the scale of the input original data, including the recovered data matrix Mt, Mt represents the recovered data matrix obtained at the tth iteration.

Specifically, when this step is executed for the first time, after the slave node completes the decomposition, the master node can splice in reverse order according to the subscript record Lab0 of the decomposition step in step S102. Let the input matrix in step S101 be M0, then all the decomposition results can be spliced to obtain 4 matrices of the same size as M0, representing the low-rank data matrix, sparse anomaly matrix, small noise matrix and error matrix respectively. The present invention mainly focuses on the low-rank data matrix, denoted as M1.

Similarly, when multiple iterations are performed, the master node can record Labt-1 according to the subscript of the decomposition step of the previous iteration, and splice them in reverse order. All decomposition results can still be spliced to obtain 4 matrices of the same size as M0, and the low-rank data matrix among them is recorded as Mt.

S302, cluster the recovered data matrix Mt again to obtain the split sub-matrices, record the split result as Labt, and compare the difference between the current split result Labt and the previous split result Labt-1 by the master node. If the difference is less than the threshold, the recovery is completed and the four matrices spliced in step 301 are used as the recovery result, otherwise proceed to the next step.

This embodiment takes the first iteration as an example for detailed description, and clusters M1 again according to the same configuration parameters in step S102. Similar to the subscript record in step S102, the result of this division is recorded as Lab1.

When the number of clusters is N, the difference between Lab0 and Lab1 is calculated according to the following formula and recorded as Lab_Diff value.

If it is the tth iteration, the above formula can be expressed as:

The obtained partition difference Lab_Diff will be compared with the configured threshold ∈, which can be set to 0.05. If Lab_Diff<∈, it means that the results of the two partitions are basically the same, which can be considered converged. LADEN will complete the decomposition and return the decomposition results including Mt, that is, the four matrices spliced in step S301; otherwise, it is necessary to continue to execute step S303.

S303, based on the pre-configured parameters, the master node determines whether the calculation time and the number of iterations have reached the maximum value. If so, the recovery is completed and the four matrices obtained by splicing in step 301 are used as the recovery result. Otherwise, the sub-matrices obtained by clustering again according to step S302 are returned to step S103 to continue the next round of recovery.

That is, the master node will record the current calculation time and the current number of iterations. According to the user configuration, it is determined whether the current calculation time has exceeded the maximum calculation time, or whether the current number of iterations has reached the maximum number of iterations. If any indicator has reached the limit, the decomposition is completed and the result is returned; otherwise, according to Mt and Labt, the division is performed according to the records in Labt, and then the process returns to step S103 to continue the next round of iterations. This multi-round recovery scheme allows users to limit the maximum value according to demand and resources, and can also avoid the impact of accidental deviations during clustering of high-dimensional data on the results, thereby better achieving the purpose of enhancing data quality.

Finally, this embodiment proves the linear relationship between the two groups of variables, namely, the recovery effect and the number of sub-matrices, and the recovery time and the sum of the squares of the sub-matrix sizes, from both theoretical and experimental aspects.

Assume that the size of the input matrix M is m×n. It is known that the worst time complexity of the optimized clustering step in the case of m points is And the constant term is very small, and the overhead in actual calculation can be basically ignored. The main operation of the data recovery step is the multiplication of a matrix of size m×m and a matrix of size m×n, and its time complexity is If we consider that the number of updates per iteration and the maximum number of iterations in the recovery are constant, and N sub-matrices are obtained by splitting, then the sum of the squares of the cluster sizes is

Assuming the time coefficient is k′, the running time can be estimated as

It is obvious from the formula that the running time is linearly related to the square of the cluster size.

Afterwards, the normalized mean absolute error (NMAE) is used to measure the recovery effect, which is defined as

Where A represents the ideal data matrix, represents the data matrix obtained by LADEN, and M represents the original matrix with defects. The smaller the NMAE, the better the recovery effect. As the number of clusters increases, the structure within each submatrix will be gradually destroyed, and the amount of information available for recovery will decrease. The worse the recovery effect of LADEN, the greater the deviation between the data matrix and the original data, and the larger the NMAE.

The strength of the linear relationship between variables can be measured by the correlation coefficient, and the range of values is usually between -1 and 1. The closer its absolute value is to 1, the stronger the correlation between the variables. It is generally believed that the absolute value of the correlation coefficient exceeds 0.7, which can illustrate that the relationship between the variables is very close. Since the present embodiment assumes that the sample data is normally distributed, only the Spearman's rank correlation coefficient can be used to test the correlation. It sacrifices a part of the accuracy of the linear relationship and the sensitivity to outliers, and gets rid of the dependence on the overall distribution parameters. In addition to the correlation coefficient, correlation analysis also needs to calculate significance. Significance can be quantified by the probability of obtaining the same result when the null hypothesis is established, which is called the P value. It is generally believed that when P<0.05, it can be considered that the null hypothesis is rejected and the conclusion is significant.

In order to more intuitively prove the two sets of linear relationships, the Abilene data set is selected, which includes the data flow size of 12 routers in the North American backbone network Abilene every five minutes in 24 weeks, representing the traffic data characteristics commonly used in traditional network analysis. This embodiment selects data with a scale of 1000×100, and the Low_Rank value is 17.0%, which is relatively ideal for low rank.

The data in the data set has been cleaned and does not contain defects. Therefore, it is necessary to artificially add various defects to verify the recovery effect of LADEN, set the anomaly rate to 0.1%, the anomaly scale to 1, the noise scale to 0.01, and do not limit the maximum number of recovery iterations. In order to reduce the impact of chance, this embodiment repeated the measurement 4 times on each set of parameters, and accumulated the actual running time on each slave node when calculating the running time, which is equivalent to a concurrency of 1. Under these premises, this embodiment added errors of varying proportions to the data, changed the clustering parameters to obtain a variety of partitioning schemes, and recorded the recovery performance separately. The results are shown in Figure 4.

It can be clearly seen from the left figure of Figure 4 that the number of clusters is linearly correlated with NMAE. At different error rates, four clear parallel lines are formed between NMAE and the number of cluster matrices. When the error rate is 0.3, the correlation coefficient r is as low as 0.943, corresponding to a P value of 1.0e-20, indicating a very high degree of linearity. It can be seen from the right figure of Figure 4 that the sum of the squares of the cluster sizes and the recovery time also show a clear strong linear relationship. The average correlation coefficient still reached 0.864, and the lowest was 0.838, corresponding to a P value of 4.3e-12, so it can be considered to have a strong linear correlation.

According to the experiments, it can be seen that in a fixed scenario, a linear conversion relationship between recovery time and recovery effect can be fitted through a small number of tests, allowing users to adjust the recovery performance of LADEN according to their needs, significantly enhancing the adjustability of the algorithm and its adaptability to complex needs, greatly improving its practical value.

The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims (10)

1. A highly robust distributed data recovery method based on compressed sensing, characterized in that the method converts the data recovery process into an optimization problem, decomposes the data matrix, noise matrix, anomaly matrix and error matrix from the original data by designing the goal and constraints of the problem, and obtains the recovered data based on the data matrix. 2. According to claim 1, a highly robust distributed data recovery method based on compressed sensing is characterized in that the method is implemented based on a distributed algorithm, and the master node performs a data decomposition process and a data enhancement process, wherein the data decomposition process splits the original data into multiple sub-matrices, and the data enhancement process optimizes the decomposition results through iteration, and the slave node performs a data recovery process on the sub-matrices split by the master node. 3. The method for highly robust distributed data recovery based on compressed sensing according to claim 1, characterized in that the method comprises the following steps: S1, distributed data decomposition: S101, the master node obtains the original data to be restored and checks the low rank of the original matrix. If it meets the low rank requirement, the next step is executed, otherwise an error is reported; S102, according to pre-configured parameters, the master node clusters the original data to split the matrix, and the data in the same cluster is spliced into a sub-matrix, thereby obtaining multiple sub-matrices; S103, the master node sends the split sub-matrices to multiple slave nodes evenly with the matrix size as the weight for data recovery; S2, highly robust data recovery: S201, each slave node extracts all vacant positions from the corresponding sub-matrix to construct an error matrix; S202, constructing the goal and optimization constraints of the optimization problem according to the defect characteristics in the data; S203, bringing in the constraint term to obtain the augmented Lagrangian formula, minimizing the corresponding augmented Lagrangian formula for each slave node using the ADM method, solving the optimization problem, and obtaining the decomposed data matrix, noise matrix, and abnormality matrix; S3, iterative data augmentation: S301, after the slave nodes complete the matrix decomposition, the master node summarizes the decomposition results of all sub-matrices on each slave node, and reassembles them according to the split positions to obtain 4 matrices consistent with the input original data scale, including the restored data matrix; S302, clustering the restored data matrix again to obtain the split sub-matrices, and the master node compares the difference between the current split result and the previous split result. If the difference is less than the threshold, the recovery is completed, and the four matrices spliced in step 301 are used as the recovery result, otherwise proceed to the next step; S303, based on the pre-configured parameters, the master node determines whether the calculation time and the number of iterations have reached the maximum value. If so, the recovery is completed and the four matrices obtained by splicing in step 301 are used as the recovery result. Otherwise, the sub-matrices obtained by clustering again in step S302 are returned to step S103 to continue the next round of recovery. 4. According to a highly robust distributed data recovery method based on compressed sensing in claim 3, it is characterized in that the step S101 is specifically as follows: the master node obtains the original data to be recovered, and temporarily fills the value 0 in all missing positions of the original data, and performs singular value decomposition on the filled original data matrix to calculate the rank of the data components therein and determine the low-rank index; if the calculated low-rank index is less than a preset threshold, execute the next step, otherwise return an error value to report an error. 5. According to the highly robust distributed data recovery method based on compressed sensing described in claim 3, it is characterized in that in the step S102, the master node clusters the original data to split the matrix using the density-based improved clustering algorithm OPTICS. On the basis of the OPTICS algorithm, the determination of the end point of the region is added, and the characteristic of the minimum heap structure itself maintaining the pioneer node in the process of algorithm generation sequence is used to determine whether the end point should be removed from the cluster; after clustering is completed, all clusters are arranged in descending order according to size and then labeled with id starting from 1, that is, the cluster with id=1 contains the most data points, and all marked outliers are aggregated into an additional cluster, and id=0 is recorded. 6. According to claim 3, a highly robust distributed data recovery method based on compressed sensing is characterized in that in step S103, the master node sends each sub-matrix to the subordinate slave nodes using the network connection, and uses the square of the matrix size as the weight and the load factor to distribute tasks among the slave nodes, and the process is as follows: Pre-configure the weight w _j for the jth node according to its computing power. The default value is 1. The initial load of the node is L _j = 1. Define the load factor For the submatrix D _i corresponding to cluster C _i , the task volume T _i is defined as the square of the scale, that is, T _i =(M _i .col) ² Select the node that currently has the largest load factor and update the load factor, that is, L′ _Node = L _Node + _Ti Repeat the above steps until the assignment is complete. 7. A highly robust distributed data recovery method based on compressed sensing according to claim 3, characterized in that, the assumption is divided into N sub-matrices, wherein the sub-matrix M _i corresponding to the cluster with id=i is of size m _i ×n _i , and step S2 decomposes it into a low-rank data matrix A _i , a sparse anomaly matrix B _i , a small noise matrix C _i and an error matrix E _i , then the optimization problem is expressed as: D _i. *E _i = D _i Where α, β, σ are the coefficients of the three corresponding matrix components to meet the needs of linear transformation and dimensional conversion; X, Y, Z are global coefficient matrices; _Di is an intermediate variable of the same size as _Ei that is used to convert _Ei from a binary matrix to a numerical matrix; and It is an additional auxiliary variable used to decouple the _Ai and _Bi variables in the objective and constraints; is the penalty term, where γ is the corresponding coefficient, K is the number of penalty terms, and U ^k , V ^k and R ^k are three global matrix parameters set according to domain knowledge. 8. The highly robust distributed data recovery method based on compressed sensing according to claim 3, characterized in that, in S302, the calculation method used to compare the difference between the current split result and the previous split result is: Wherein, Lab_Diff represents the obtained partition difference result, N is the number of sub-matrices, Labt(i) represents the split result of the t-th iteration, and Labt-1(i) represents the split result of the previous iteration before the t-th iteration. 9. According to claim 3, a highly robust distributed data recovery method based on compressed sensing is characterized in that the method predicts data recovery performance by adjusting the partitioning scheme in the data decomposition stage, and the data recovery performance includes recovery effect and recovery time, wherein the recovery effect is predicted based on the number of sub-matrices, and the recovery time is predicted based on the sum of the squares of the sub-matrix sizes. 10. The highly robust distributed data recovery method based on compressed sensing according to claim 9, characterized in that the recovery time prediction method is: Where N is the number of sub-matrices to be split, k′ is the time coefficient, m×n is the input matrix size, Square_Sum is the sum of squares of cluster sizes, The recovery effect is measured using the normalized mean absolute error (NMAE), defined as Where A represents the ideal data matrix, represents the restored data matrix outputted in step S3, and M represents the original matrix with defects.