CN115935147A - Traffic data recovery and abnormal value detection method represented by low-rank and sparse tensor - Google Patents

Traffic data recovery and abnormal value detection method represented by low-rank and sparse tensor Download PDF

Info

Publication number
CN115935147A
CN115935147A CN202211492049.4A CN202211492049A CN115935147A CN 115935147 A CN115935147 A CN 115935147A CN 202211492049 A CN202211492049 A CN 202211492049A CN 115935147 A CN115935147 A CN 115935147A
Authority
CN
China
Prior art keywords
tensor
traffic data
rank
low
space
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211492049.4A
Other languages
Chinese (zh)
Inventor
贺洋
赵嘉悦
夏井新
安成川
陆振波
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Southeast University
Original Assignee
Southeast University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Southeast University filed Critical Southeast University
Priority to CN202211492049.4A priority Critical patent/CN115935147A/en
Publication of CN115935147A publication Critical patent/CN115935147A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Traffic Control Systems (AREA)

Abstract

The invention discloses a traffic data recovery and abnormal value detection method expressed by low rank and sparse tensor, which comprises the following steps: constructing the traffic data observation containing the missing and abnormal values into three-dimensional tensor of position, date and time according to three dimensions of place, date and time
Figure DDA0003963716690000011
Based on the space-time traffic data, the space-time traffic data is further decomposed and expressed as low-rank tensor representing traffic mode
Figure DDA0003963716690000012
And a sparse tensor epsilon representing the outliers; aiming at the characteristics of a traffic mode and an abnormal value, respectively adopting a non-convex relaxation function based on logarithm and an l1 norm to constrain the two parts, and constructing a traffic data recovery and abnormal value detection model based on low-rank sparse tensor expression on the basis of the constraint; converting the multivariate optimization problem of the model into four univariate quantum optimization problems according to the ADMM frameworkInitialization tensor
Figure DDA0003963716690000014
Sequentially updating
Figure DDA0003963716690000015
Four variables; to be provided with
Figure DDA0003963716690000016
As input, iterative optimization is carried out by utilizing a cross direction multiplier algorithm until a convergence condition is met, and a converged low-rank tensor is obtained
Figure DDA0003963716690000013
And a sparse tensor epsilon. The invention can synchronously realize the accurate robust recovery and abnormal value detection of the traffic data.

Description

Traffic data recovery and abnormal value detection method represented by low-rank and sparse tensor
Technical Field
The invention relates to the field of intelligent traffic, in particular to a low-rank and sparse tensor expression traffic data recovery and abnormal value detection method.
Background
Traffic data has high-dimensional data characteristics, but has a low-dimensional data structure, which is characterized by periodicity in time and correlation in space of traffic data, which is low-rank in nature. To better exploit the low rank nature of traffic data, researchers typically represent traffic data as three-dimensional tensors in location, time, and date. The current traffic data recovery method based on low rank tensor expression mainly comprises the following steps: low rank tensor decomposition, low rank tensor completion, etc.
Abnormal values in traffic data have the characteristics of random positions, sparse distribution, uncertain damage degree and the like. In consideration of the sparse characteristic of the abnormal value of the traffic data, the conventional abnormal value detection method is mainly based on Robust Principal Component Analysis (RPCA) and is subsequently expanded to Tensor Robust Principal Component Analysis (TRPCA).
The existing abnormal value detection method has the following defects:
(1) The existing traffic data recovery method based on low-rank representation is poor in consideration of abnormal values, the assumption that traffic data observation values are real values is often implied, abnormal value damage generally exists in the traffic data actually, the observation is directly used as a true value to carry out data recovery, large negative effects are generated on the recovery precision, and the model is lack of robustness.
(2) The existing abnormal value detection method mostly adopts the characteristics of the abnormal value, and is still insufficient for the depiction of the traffic mode level. In the existing robust principal component analysis method, convex relaxation nuclear norm minimization is mostly adopted for constraint on structural traffic modes, and the problem of excessive relaxation exists.
(3) The traditional method is independent of the consideration of traffic data recovery and abnormal value processing. The abnormal values in the traffic data observation are detected and removed, and then the traffic data with the abnormal values removed is recovered, so that the data recovery efficiency and the abnormal value detection efficiency are not high.
Disclosure of Invention
The invention aims at: the method for recovering the traffic data and detecting the abnormal values represented by the low-rank and sparse tensors is provided, the accurate and complete traffic data are accurately and robustly recovered from partial data observation of deletion and abnormal value damage by utilizing the time-space correlation of the traffic data, and the corresponding abnormal values are detected.
In order to realize the functions, the invention designs a traffic data recovery and abnormal value detection method expressed by low rank and sparse tensor, aiming at each space-time traffic data which is collected at different positions of a road network of a target area and contains a common traffic mode and an occasional abnormal value and data loss in each space-time traffic data, the following steps S1-S4 are executed to complete the recovery of the lost data in each space-time traffic data and the detection of the abnormal value:
step S1: aiming at each collected space-time traffic data, the space-time traffic data is observed and constructed into a three-dimensional tensor of position, date and time according to three dimensions of the collected position, date and time
Figure BDA0003963716670000021
Representing traffic patterns in spatiotemporal traffic data as low rank tensor @>
Figure BDA0003963716670000022
Representing the anomaly value as a sparse tensor ε, introducing an auxiliary variable->
Figure BDA0003963716670000023
Receiving space-time traffic data observation information and establishing the observation information and the low-rank tensor->
Figure BDA0003963716670000024
Relationship constraints of the sparse tensor epsilon;
step S2: aiming at the low rank tensor corresponding to the traffic mode
Figure BDA0003963716670000025
And sparse tensor epsilon corresponding to the abnormal value, respectively adopting a non-convex relaxation function and l based on logarithm 1 Norm, for low rank tensor>
Figure BDA0003963716670000026
Constraining the sparse tensor epsilon, and constructing a space-time traffic data recovery and abnormal value detection model;
and step S3: based on a space-time traffic data recovery and abnormal value detection model, a multivariable optimization objective function of the space-time traffic data recovery and abnormal value detection model is constructed by introducing an augmented Lagrange function, and the multivariable optimization problem is decomposed into the multivariate optimization objective functions respectively aiming at the space-time traffic data recovery and abnormal value detection model based on an ADMM framework
Figure BDA0003963716670000027
ε、/>
Figure BDA0003963716670000028
In which->
Figure BDA0003963716670000029
Representing a lagrange multiplier;
and step S4: respectively initializing each univariate to obtain the initial tensor of each univariate
Figure BDA00039637166700000210
ε 0 />
Figure BDA00039637166700000211
Based on the initial tensors, carrying out iterative updating on the univariates by adopting an ADMM method until a preset convergence condition is reached, and obtaining a converged low-rank tensor/based on>
Figure BDA00039637166700000212
And the sparse tensor epsilon is used for completing missing data recovery in the air-to-air traffic data and detecting abnormal values.
As a preferred technical scheme of the invention: the three-dimensional tensor constructed in step S1
Figure BDA00039637166700000213
In the form of
Figure BDA00039637166700000214
Wherein n is 1 Is the number of acquisition positions where the data acquisition device is located, n 2 Number of dates, n, of the collected spatiotemporal traffic data 3 Representing the number of time segments for collecting the space-time traffic data every natural day;
based on low rank tensor
Figure BDA00039637166700000215
Sparse tensor ε constructing auxiliary variables->
Figure BDA00039637166700000216
The following formula:
Figure BDA00039637166700000219
wherein, P Ω Is defined as follows:
Figure BDA00039637166700000217
in the formula, observation index set of omega space-time traffic data, y ijk Is a three-dimensional tensor
Figure BDA00039637166700000218
The data located at (i, j, k).
As a preferred technical scheme of the invention: the specific steps of step S2 are as follows:
step S21: low rank tensor corresponding to traffic mode
Figure BDA0003963716670000031
And constructing a low-rank tensor completion model objective function based on rank minimization according to a sparse tensor epsilon corresponding to the abnormal value, wherein the low-rank tensor completion model objective function is as follows:
Figure BDA0003963716670000032
Figure BDA0003963716670000033
where λ is the regularization term weight with respect to the sparse tensor ε;
step S22: a logarithm-based non-convex relaxation function is used as an approximation function f (X) of the rank function, as follows:
Figure BDA0003963716670000034
in which X is an arbitrary two-dimensional matrix, σ i (X) represents the ith singular value of the matrix X, and epsilon is a constant of a preset value range so that sigma is i (X) + ε is a positive number;
step S23: the space-time traffic data recovery and abnormal value detection model is constructed according to the following formula:
Figure BDA0003963716670000035
Figure BDA0003963716670000036
in the formula (I), the compound is shown in the specification,
Figure BDA0003963716670000037
ε k respectively representing low rank tensor->
Figure BDA00039637166700000315
An auxiliary tensor developed along three modalities of the sparse tensor ε, where k =1,2,3, </or >>
Figure BDA0003963716670000038
Is->
Figure BDA0003963716670000039
Matrix developed along the kth mode, α k Is related to>
Figure BDA00039637166700000310
Regular weight of (a) k Is about epsilon k The canonical weight of (1).
As a preferred technical scheme of the invention: the specific steps of step S3 are as follows:
step S31: based on space-time traffic data recovery and an abnormal value detection model, an augmented Lagrangian multiplier is introduced to construct an augmented Lagrangian function of the model as follows:
Figure BDA00039637166700000311
Figure BDA00039637166700000312
in the formula, ρ k A penalty term weight for the kth modality,
Figure BDA00039637166700000313
lagrange multiplier terms for the kth mode;
step S32: based on ADMM framework, multivariate optimization problem is decomposed to aim at respectively
Figure BDA00039637166700000314
The univariate quantum optimization problem of epsilon is as follows:
Figure BDA0003963716670000041
Figure BDA0003963716670000042
Figure BDA0003963716670000043
Figure BDA0003963716670000044
in the formula, l represents the number of iterations.
As a preferred technical scheme of the invention: the specific steps of step S4 are as follows:
step S41: three-dimensional tensor constructed in step S1
Figure BDA0003963716670000045
As model inputs, the variables are initialized as follows:
Figure BDA0003963716670000046
step S42: according to the initial tensor
Figure BDA0003963716670000047
ε 0 、/>
Figure BDA0003963716670000048
Updates in turn>
Figure BDA0003963716670000049
ε 1 、/>
Figure BDA00039637166700000410
Calculate->
Figure BDA00039637166700000411
The modality auxiliary tensor->
Figure BDA00039637166700000412
The following formula:
Figure BDA00039637166700000413
in the formula (I), the compound is shown in the specification,
Figure BDA00039637166700000414
for weighted singular value threshold operators, E k(k) Is epsilon k Of the modal expansion matrix, M k(k) Is->
Figure BDA00039637166700000415
Of a modal expansion matrix, T k(k) Is->
Figure BDA00039637166700000416
The modal expansion matrix of (a); />
Auxiliary tensor according to each modality
Figure BDA00039637166700000417
Calculating a low rank tensor pick>
Figure BDA00039637166700000418
The following formula:
Figure BDA00039637166700000419
based on an updated low rank tensor
Figure BDA00039637166700000420
Calculate->
Figure BDA00039637166700000421
The following formula:
Figure BDA00039637166700000422
based on updated
Figure BDA00039637166700000423
Calculating epsilon 1 Is greater than or equal to>
Figure BDA00039637166700000424
Figure BDA00039637166700000425
Figure BDA0003963716670000051
Wherein
Figure BDA0003963716670000052
Representing a point-by-point product;
based on updated
Figure BDA0003963716670000053
ε 1 Calculate->
Figure BDA0003963716670000054
The following formula:
Figure BDA0003963716670000055
step S43: to be updated
Figure BDA0003963716670000056
ε 1 、/>
Figure BDA0003963716670000057
For input, the following formula is iteratively calculated until a preset convergence condition is reached:
Figure BDA0003963716670000058
Figure BDA0003963716670000059
Figure BDA00039637166700000510
Figure BDA00039637166700000511
l＝l+1；
wherein
Figure BDA00039637166700000512
Step S44: outputting the repaired complete traffic data low-rank tensor
Figure BDA00039637166700000513
And an outlier sparse tensor epsilon.
Has the advantages that: compared with the prior art, the invention has the advantages that:
(1) In the process of recovering the traffic data, the existence of data observation abnormal values is considered, the traffic data are expressed into a low-rank traffic mode and a sparse abnormal value, the negative influence of random abnormal values on the data recovery is eliminated, and a reliable scheme is provided for the recovery of robust traffic data;
(2) And aiming at the respective characteristics of the traffic mode and the abnormal value in the traffic data, a traffic data recovery and abnormal value detection method is constructed, and the abnormal value is synchronously detected in the traffic data recovery process. The method has high data recovery and abnormal value detection efficiency, can quickly form large-range application, and has low realization cost.
Drawings
Fig. 1 is a flowchart of a traffic data recovery and outlier detection method with low rank and sparse tensor representation according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a traffic data recovery and outlier detection method with low rank and sparse tensor representation according to an embodiment of the present invention;
figure 3 is a schematic illustration of a modal expansion of a three-dimensional tensor;
FIG. 4 is a graph of data recovery accuracy of the present invention versus prior art schemes at different damage rates;
FIG. 5 is a graph comparing data recovery and anomaly detection using the present invention and prior art schemes.
Detailed Description
The invention is further described below with reference to the accompanying drawings. The following examples are only for illustrating the technical solutions of the present invention more clearly, and the protection scope of the present invention is not limited thereby.
Technical terms related to the embodiments of the present invention are explained as follows:
space-time traffic data: recording time-varying traffic state (such as traffic flow and speed) observation of different positions of a road network based on a high-dimensional time sequence acquired by a fixed sensor or a mobile sensor;
and (3) traffic data recovery: recovering accurate and complete space-time traffic data from partial space-time traffic data observation comprising various deletions (random and non-random deletions) and abnormal values;
abnormal value detection: detecting abnormal values deviating from normal traffic states from the space-time traffic data observation;
low rank tensor representation: representing the part of the space-time traffic data representing the frequent traffic pattern as a low-rank tensor;
sparse tensor representation: the part of the space-time traffic data representing sporadic outliers is represented as a sparse tensor.
Spatiotemporal traffic data is a key input for many applications in Intelligent Transportation Systems (ITS), the quality of which directly affects the efficiency of the intelligent transportation system. With the development of traffic awareness technology, the size and dimension of spatiotemporal traffic data are continuously increasing. Meanwhile, due to sensor faults, communication faults and other reasons, the space-time traffic data has the problems of deletion and abnormal value damage at the same time, and the practicability and effectiveness of the space-time traffic data in the application of the intelligent traffic system are directly influenced. How to utilize the space-time correlation of traffic data, accurately and robustly recover accurate and complete space-time traffic data from partial data observation of deletion and abnormal value damage, and detect corresponding abnormal values, which has important significance for the intelligent traffic field.
The embodiment of the invention provides a low-rank and sparse tensor expressed traffic data recovery and abnormal value detection method, which has the advantages that a flow chart and a schematic diagram refer to fig. 2 and fig. 3, data acquisition equipment is adopted to acquire space-time traffic data at different positions of a target area road network, the space-time traffic data comprises a frequent traffic mode and accidental abnormal values, the frequent traffic mode refers to periodic characteristics of traffic flow, such as peak and flat peak characteristics, and the accidental abnormal values deviate from the traffic mode and present abrupt burr characteristics; in addition, when the space-time traffic data generally has data missing situations, the following steps S1-S4 are executed to complete the missing data recovery and abnormal value detection in the space-time traffic data:
step S1: aiming at each collected space-time traffic data, the space-time traffic data observation is constructed into a three-dimensional tensor with the position, the date and the time according to the three dimensions of the collected position, the collected date and the collected time
Figure BDA0003963716670000071
Representing traffic patterns in spatiotemporal traffic data as low-rank tensors>
Figure BDA0003963716670000072
Representing the anomaly value as a sparse tensor ε, introducing an auxiliary variable->
Figure BDA0003963716670000073
Receive space-timeTraffic data observation information is established and compared with the low-rank tensor>
Figure BDA0003963716670000074
Relationship constraints of the sparse tensor epsilon; />
Step S1 constructed three-dimensional tensor
Figure BDA0003963716670000075
In the form of +>
Figure BDA0003963716670000076
Wherein n is 1 Number of acquisition positions where data acquisition devices are located, n 2 Number of dates, n, of the collected spatiotemporal traffic data 3 Representing the number of time segments for collecting the time-space traffic data every natural day;
based on low rank tensor
Figure BDA0003963716670000077
Sparse tensor ε constructing auxiliary variables->
Figure BDA0003963716670000078
The following formula:
Figure BDA0003963716670000079
wherein, P Ω Is defined as follows:
Figure BDA00039637166700000710
in the formula, observation index set of omega space-time traffic data, y ijk Is a three-dimensional tensor
Figure BDA00039637166700000711
Of (i, j, k).
Step S2: aiming at the low rank tensor corresponding to the traffic mode
Figure BDA00039637166700000712
And sparse tensor epsilon corresponding to the abnormal value, respectively adopting a non-convex relaxation function and l based on logarithm 1 Norm, for low rank tensor>
Figure BDA00039637166700000713
Constraining the sparse tensor epsilon, and constructing a space-time traffic data recovery and abnormal value detection model;
the specific steps of step S2 are as follows:
step S21: low rank tensor corresponding to traffic mode
Figure BDA00039637166700000714
And constructing a low-rank tensor completion model objective function based on rank minimization according to a sparse tensor epsilon corresponding to the abnormal value, wherein the low-rank tensor completion model objective function is as follows:
Figure BDA00039637166700000715
Figure BDA00039637166700000716
where λ is the regularized term weight on the sparse tensor ε;
step S22: a logarithm-based non-convex relaxation function is used as an approximation function f (X) of the rank function, as follows:
Figure BDA00039637166700000717
in which X is an arbitrary two-dimensional matrix, σ i (X) represents the ith singular value of the matrix X, and epsilon is a constant of a preset value range so that sigma is i (X) + Epsilon is a positive number, and Epsilon is 10 -6 ～10 -4 ；
Step S23: the space-time traffic data recovery and abnormal value detection model is constructed according to the following formula:
Figure BDA0003963716670000081
Figure BDA0003963716670000082
in the formula (I), the compound is shown in the specification,
Figure BDA0003963716670000083
ε k respectively representing low rank tensor pick>
Figure BDA0003963716670000084
An auxiliary tensor developed along three modalities of the sparse tensor ε, where k =1,2,3, </or >>
Figure BDA0003963716670000085
Is->
Figure BDA0003963716670000086
Matrix developed along the k-th mode, α k Is related to>
Figure BDA0003963716670000087
Regular weight of (a) k Is about epsilon k The modal expansion of the three-dimensional tensor refers to fig. 3.
And step S3: based on a space-time traffic data recovery and abnormal value detection model, a multivariable optimization objective function of the space-time traffic data recovery and abnormal value detection model is constructed by introducing an augmented Lagrange function, and the multivariable optimization problem is decomposed into the multivariate optimization objective functions respectively aiming at the space-time traffic data recovery and abnormal value detection model based on an ADMM framework
Figure BDA0003963716670000088
ε、/>
Figure BDA0003963716670000089
In which->
Figure BDA00039637166700000810
Representing a lagrange multiplier; />
The specific steps of step S3 are as follows:
step S31: based on space-time traffic data recovery and an abnormal value detection model, an augmented Lagrange multiplier is introduced to construct an augmented Lagrange function of the model as follows:
Figure BDA00039637166700000811
Figure BDA00039637166700000812
in the formula, ρ k A penalty term weight for the kth modality,
Figure BDA00039637166700000813
lagrange multiplier terms for the kth mode;
step S32: based on ADMM framework, multivariate optimization problem is decomposed to aim at respectively
Figure BDA00039637166700000814
The univariate quantum optimization problem of epsilon is as follows:
Figure BDA00039637166700000815
Figure BDA00039637166700000816
Figure BDA00039637166700000817
Figure BDA00039637166700000818
in the formula, l represents the number of iterations.
And step S4: respectively initializing each univariate to obtain the initial tensor of each univariate
Figure BDA00039637166700000819
ε 0 />
Figure BDA00039637166700000820
Based on the initial tensors, carrying out iterative updating on the univariates by adopting an ADMM method until a preset convergence condition is reached, and obtaining a converged low-rank tensor (or greater than or equal to)>
Figure BDA0003963716670000091
And the sparse tensor epsilon is used for completing missing data recovery in the time-air traffic data and detection of abnormal values.
The specific steps of step S4 are as follows:
step S41: the three-dimensional tensor constructed in step S1
Figure BDA0003963716670000092
As model inputs, the variables are initialized as follows:
Figure BDA0003963716670000093
step S42: according to the initial tensor
Figure BDA0003963716670000094
ε 0 、/>
Figure BDA0003963716670000095
Updates in turn>
Figure BDA0003963716670000096
ε 1 、/>
Figure BDA0003963716670000097
Calculate->
Figure BDA0003963716670000098
The modality auxiliary tensor->
Figure BDA0003963716670000099
The following formula:
Figure BDA00039637166700000910
in the formula (I), the compound is shown in the specification,
Figure BDA00039637166700000911
for weighted singular value threshold operators, E k(k) Is epsilon k Of the modal expansion matrix, M k(k) Is->
Figure BDA00039637166700000912
A modal expansion matrix of (T) k(k) Is->
Figure BDA00039637166700000913
The modal expansion matrix of (a); />
Auxiliary tensor according to each modality
Figure BDA00039637166700000914
Calculating a low rank tensor +>
Figure BDA00039637166700000915
The following formula:
Figure BDA00039637166700000916
based on an updated low rank tensor
Figure BDA00039637166700000917
Calculate->
Figure BDA00039637166700000918
The following formula:
Figure BDA00039637166700000919
based on updated
Figure BDA00039637166700000920
Calculating epsilon 1 Is greater than or equal to>
Figure BDA00039637166700000921
Figure BDA00039637166700000922
Wherein
Figure BDA00039637166700000923
Representing a point-by-point product;
based on updated
Figure BDA00039637166700000924
ε 1 Counting/or>
Figure BDA00039637166700000925
The following formula:
Figure BDA00039637166700000926
step S43: to be updated
Figure BDA0003963716670000101
ε 1 、/>
Figure BDA0003963716670000102
For input, the following formula is iteratively calculated until a preset convergence condition is reached:
Figure BDA0003963716670000103
Figure BDA0003963716670000104
Figure BDA0003963716670000105
Figure BDA0003963716670000106
l＝l+1；
wherein
Figure BDA0003963716670000107
Step S44: outputting the repaired complete traffic data low-rank tensor
Figure BDA0003963716670000108
And an outlier sparse tensor epsilon.
The pseudo code of the ADMM method is specifically as follows:
inputting: auxiliary tensor obtained by first iteration
Figure BDA0003963716670000109
Lagrange multiplier->
Figure BDA00039637166700001010
Low rank tensor pick>
Figure BDA00039637166700001011
Sparse tensor ε 1 ；
And (3) outputting: low rank tensor after data recovery and anomaly detection
Figure BDA00039637166700001012
And a sparse tensor ε;
parameters are as follows: low rank tensor
Figure BDA00039637166700001013
Each mode weight->
Figure BDA00039637166700001014
Modal weights λ of sparse tensor ε 1 ＝λ 2 ＝λ 3 Learning rate ρ k ＝ρ＝ρ 0 Convergence threshold e =1e-6;
Figure BDA00039637166700001015
/>
for k＝1:3do
Figure BDA00039637166700001016
Figure BDA00039637166700001017
Figure BDA00039637166700001018
for k＝1:3do
Figure BDA00039637166700001019
Figure BDA00039637166700001020
l＝l+1
wherein o is l An objective function value representing the l-th iteration, k representing the index of the different modalities of the tensor (k =1,2, 3),
Figure BDA0003963716670000111
representing a weighted singular value threshold operator, λ k Represents l 1 Norm under epsilon constraint k The weight of (a) is calculated,
Figure BDA0003963716670000112
the dot-by-dot product is represented.
Weighted singular value threshold operator
Figure BDA0003963716670000113
In which U (sigma) V T Is an arbitrary matrix
Figure BDA0003963716670000114
Singular value decomposition of->
Figure BDA0003963716670000115
Is->
Figure BDA0003963716670000116
In a mode expansion matrix, based on the number of the preceding frames>
Figure BDA0003963716670000117
Representing a matrix +>
Figure BDA0003963716670000118
Of the 1 st singular value, ε k Is constant, usually 10 -6 ～10 -4 ，τ＝α k /ρ k 。
Examples of the invention are as follows, all from three data sets:
the first embodiment is as follows: peMS highway traffic data set P: the data set contains traffic volume collected by the california bureau of transportation performance measurement system (PeMS) at a resolution of 5 minutes (i.e., 288 time intervals per day) from 228 coil detectors during the working days of months 5 and 6 in 2012. The tensor size is 228 × 288 × 44.
The second embodiment: seattle highway traffic speed data set S: the data set contains highway traffic speed for 323 coil detectors in seattle, usa for the first 4 weeks of 1 month in 2015 with a resolution of 5 minutes (i.e. 288 time intervals per day). The tensor size is 323 × 288 × 28.
Example three: guangzhou city traffic speed data set G: the data set contains traffic speeds collected over 214 road segments in Guangzhou, china from 8/1/2016 to 30/9/10 minutes of resolution (i.e., 144 time intervals per day). The tensor size is 214 × 144 × 61.
In order to make the experimental result universal, 40% of data is randomly selected as input (the data loss rate is 60%), 10%, 20%, 30%, 40%, 50%, 60% and 70% of data are respectively damaged randomly in each loss scene to carry out experiments to detect the data recovery capability of the invention under various conditions, the data recovery precision (MAPE) is used as an index for measuring the data recovery precision, compared with other existing schemes, the experimental result is shown in FIG. 4, FIGS. 4a to 4c show the data recovery precision of the invention and the existing schemes aiming at data sets P, S and G under different damage rates, BTMF, BGCP, haLRTC and TNN in the figure are all the existing schemes, and RTC-PFNC is a traffic data recovery and abnormal value detection method expressed by low-rank and sparse tensors designed by the invention. Referring to fig. 5, fig. 5 a-5 i show the experimental results of data recovery and anomaly detection performed on data sets P, S, and G respectively by using the present invention and the existing schemes HaLRTC, LRTC-TNN, respectively, and the experimental results show that the low-rank and sparse tensor-represented traffic data recovery and anomaly detection method designed by the present invention achieves good effects on spatio-temporal traffic data recovery and anomaly detection of each data set.
The embodiments of the present invention have been described in detail with reference to the drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the gist of the present invention.

Claims (5)

1. A method for recovering traffic data and detecting abnormal values represented by low-rank and sparse tensors is characterized in that the following steps S1-S4 are executed for each space-time traffic data which is collected at different positions of a road network of a target area and contains a common traffic mode and sporadic abnormal values and data loss in each space-time traffic data, and recovery of the lost data in each space-time traffic data and detection of the abnormal values are completed:
step S1: aiming at each collected space-time traffic data, the space-time traffic data observation is constructed into a three-dimensional tensor with the position, the date and the time according to the three dimensions of the collected position, the collected date and the collected time
Figure FDA0003963716660000011
Representing traffic patterns in spatiotemporal traffic data as low rank tensor @>
Figure FDA0003963716660000012
Expressing an anomaly value as a sparse tensor ε introducing an auxiliary variable>
Figure FDA0003963716660000013
Receiving space-time traffic data observation information and establishing the observation information and the low-rank tensor->
Figure FDA0003963716660000014
Relationship constraints of the sparse tensor epsilon;
step S2: aiming at the low rank tensor corresponding to the traffic mode
Figure FDA0003963716660000015
And sparse tensor epsilon corresponding to the abnormal value, respectively adopting a non-convex relaxation function and l based on logarithm 1 Norm, for low rank tensor>
Figure FDA0003963716660000016
Constraining the sparse tensor epsilon, and constructing a space-time traffic data recovery and abnormal value detection model;
and step S3: based on a space-time traffic data recovery and abnormal value detection model, a multivariable optimization objective function of the space-time traffic data recovery and abnormal value detection model is constructed by introducing an augmented Lagrange function, and the multivariable optimization problem is decomposed into the multivariate optimization objective functions respectively aiming at the space-time traffic data recovery and abnormal value detection model based on an ADMM framework
Figure FDA0003963716660000017
In which->
Figure FDA0003963716660000018
Representing a lagrange multiplier;
and step S4: respectively initializing each univariate to obtain initial tensor of each univariate
Figure FDA0003963716660000019
Based on the initial tensors, carrying out iterative updating on the univariates by adopting an ADMM method until a preset convergence condition is reached, and obtaining a converged low-rank tensor (or greater than or equal to)>
Figure FDA00039637166600000110
And the sparse tensor epsilon is used for completing missing data recovery in the time-air traffic data and detection of abnormal values.
2. The method for recovering traffic data and detecting abnormal values represented by low-rank and sparse tensors according to claim 1, wherein the three-dimensional tensor constructed in the step S1
Figure FDA00039637166600000111
Is in the form of->
Figure FDA00039637166600000112
Wherein n is 1 Number of acquisition positions where data acquisition devices are located, n 2 Number of dates, n, of the collected spatiotemporal traffic data 3 Representing the number of time segments for collecting the time-space traffic data every natural day;
based on low rank tensor
Figure FDA00039637166600000113
Sparse tensor ε constructing auxiliary variables->
Figure FDA00039637166600000114
The following formula:
Figure FDA00039637166600000115
wherein, P Ω Is defined as follows:
Figure FDA00039637166600000116
in the formula, observation index set of omega space-time traffic data, y ijk Is a three-dimensional tensor
Figure FDA0003963716660000021
The data located at (i, j, k).
3. The method for recovering the traffic data and detecting the abnormal value represented by the low rank and sparse tensor according to claim 2, wherein the step S2 comprises the following specific steps:
step S21: low rank tensor corresponding to traffic mode
Figure FDA0003963716660000022
And constructing a low-rank tensor completion model objective function based on rank minimization according to a sparse tensor epsilon corresponding to the abnormal value, wherein the low-rank tensor completion model objective function is as follows:
Figure FDA0003963716660000023
Figure FDA0003963716660000024
where λ is the regularized term weight on the sparse tensor ε;
step S22: a logarithm-based non-convex relaxation function is used as an approximation function f (X) of the rank function, specifically as follows:
Figure FDA0003963716660000025
in which X is an arbitrary two-dimensional matrix, σ i (X) represents the ith singular value of the matrix X, and epsilon is a constant of a preset value range so that sigma is i (X) + ε is a positive number;
step S23: the space-time traffic data recovery and abnormal value detection model is constructed according to the following formula:
Figure FDA0003963716660000026
Figure FDA0003963716660000027
in the formula (I), the compound is shown in the specification,
Figure FDA0003963716660000028
ε k respectively representing low rank tensor->
Figure FDA0003963716660000029
An auxiliary tensor developed along three modalities of the sparse tensor ε, where k =1,2,3, </or >>
Figure FDA00039637166600000210
Is->
Figure FDA00039637166600000211
Matrix developed along the k-th mode, α k Is related to>
Figure FDA00039637166600000212
Regular weight of (a) ("lambda") k Is about epsilon k The canonical weight of (1).
4. The method for recovering traffic data and detecting abnormal values represented by low rank and sparse tensor according to claim 3, wherein the step S3 comprises the following steps:
step S31: based on space-time traffic data recovery and an abnormal value detection model, an augmented Lagrangian multiplier is introduced to construct an augmented Lagrangian function of the model as follows:
Figure FDA00039637166600000213
Figure FDA00039637166600000214
in the formula, ρ k A penalty term weight for the kth modality,
Figure FDA00039637166600000215
lagrange multiplier terms for the kth mode;
step S32: based on ADMM framework, multivariate optimization problem is decomposed to aim at respectively
Figure FDA0003963716660000031
Figure FDA0003963716660000032
The univariate quantum optimization problem of (a) is as follows:
Figure FDA0003963716660000033
Figure FDA0003963716660000034
Figure FDA0003963716660000035
Figure FDA0003963716660000036
in the formula, l represents the number of iterations.
5. The method for recovering the traffic data and detecting the abnormal value represented by the low-rank and sparse tensor according to the claim 1, wherein the specific steps of the step S4 are as follows:
step S41: the three-dimensional tensor constructed in step S1
Figure FDA0003963716660000037
As model inputs, the variables are initialized as follows:
Figure FDA0003963716660000038
step S42: according to the initial tensor
Figure FDA0003963716660000039
Updates in turn>
Figure FDA00039637166600000310
Calculate->
Figure FDA00039637166600000311
The modality auxiliary tensor->
Figure FDA00039637166600000312
The following formula:
Figure FDA00039637166600000313
in the formula (I), the compound is shown in the specification,
Figure FDA00039637166600000314
for weighted singular value threshold operators, E k(k) Is epsilon k Mode expansion matrix of, M k(k) Is->
Figure FDA00039637166600000315
Of a modal expansion matrix, T k(k) Is->
Figure FDA00039637166600000316
The modal expansion matrix of (a);
auxiliary tensor according to each modality
Figure FDA00039637166600000317
Calculating a low rank tensor pick>
Figure FDA00039637166600000318
The following formula:
Figure FDA00039637166600000319
based on an updated low rank tensor
Figure FDA00039637166600000320
Counting/or>
Figure FDA00039637166600000321
The following formula:
Figure FDA0003963716660000041
based on updated
Figure FDA0003963716660000042
Calculating epsilon 1 Is greater than or equal to>
Figure FDA0003963716660000043
Figure FDA0003963716660000044
Wherein
Figure FDA0003963716660000045
Figure FDA0003963716660000046
Representing a point-by-point product;
based on updated
Figure FDA0003963716660000047
Calculate->
Figure FDA0003963716660000048
The following formula:
Figure FDA0003963716660000049
step S43: to be updated
Figure FDA00039637166600000410
For input, the following formula is iteratively calculated until a preset convergence condition is reached:
Figure FDA00039637166600000411
Figure FDA00039637166600000412
Figure FDA00039637166600000413
Figure FDA00039637166600000414
l＝l+1；
wherein
Figure FDA00039637166600000415
Step S44: outputting the repaired complete traffic data low-rank tensor
Figure FDA00039637166600000416
And an outlier sparse tensor epsilon. />
CN202211492049.4A 2022-11-25 2022-11-25 Traffic data recovery and abnormal value detection method represented by low-rank and sparse tensor Pending CN115935147A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211492049.4A CN115935147A (en) 2022-11-25 2022-11-25 Traffic data recovery and abnormal value detection method represented by low-rank and sparse tensor

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211492049.4A CN115935147A (en) 2022-11-25 2022-11-25 Traffic data recovery and abnormal value detection method represented by low-rank and sparse tensor

Publications (1)

Publication Number Publication Date
CN115935147A true CN115935147A (en) 2023-04-07

Family

ID=86655260

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211492049.4A Pending CN115935147A (en) 2022-11-25 2022-11-25 Traffic data recovery and abnormal value detection method represented by low-rank and sparse tensor

Country Status (1)

Country Link
CN (1) CN115935147A (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116628435A (en) * 2023-07-21 2023-08-22 山东高速股份有限公司 Road network traffic flow data restoration method, device, equipment and medium
CN116627953A (en) * 2023-05-24 2023-08-22 首都师范大学 Method for repairing loss of groundwater level monitoring data
CN116698416A (en) * 2023-06-08 2023-09-05 上海理工大学 Low-rank characteristic multichannel synchronous extraction method for early fault detection
CN117271988A (en) * 2023-11-23 2023-12-22 广东工业大学 Tensor wheel-based high-dimensional signal recovery method and device
CN117935549A (en) * 2024-01-23 2024-04-26 江苏中路交通发展有限公司 Expressway traffic abnormality detection method

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116627953A (en) * 2023-05-24 2023-08-22 首都师范大学 Method for repairing loss of groundwater level monitoring data
CN116627953B (en) * 2023-05-24 2023-10-27 首都师范大学 Method for repairing loss of groundwater level monitoring data
CN116698416A (en) * 2023-06-08 2023-09-05 上海理工大学 Low-rank characteristic multichannel synchronous extraction method for early fault detection
CN116698416B (en) * 2023-06-08 2024-04-02 上海理工大学 Low-rank characteristic multichannel synchronous extraction method for early fault detection
CN116628435A (en) * 2023-07-21 2023-08-22 山东高速股份有限公司 Road network traffic flow data restoration method, device, equipment and medium
CN116628435B (en) * 2023-07-21 2023-09-29 山东高速股份有限公司 Road network traffic flow data restoration method, device, equipment and medium
CN117271988A (en) * 2023-11-23 2023-12-22 广东工业大学 Tensor wheel-based high-dimensional signal recovery method and device
CN117271988B (en) * 2023-11-23 2024-02-09 广东工业大学 Tensor wheel-based high-dimensional signal recovery method and device
CN117935549A (en) * 2024-01-23 2024-04-26 江苏中路交通发展有限公司 Expressway traffic abnormality detection method

Similar Documents

Publication Publication Date Title
CN115935147A (en) Traffic data recovery and abnormal value detection method represented by low-rank and sparse tensor
CN108549908B (en) Chemical process fault detection method based on multi-sampling probability kernel principal component model
CN111274525B (en) Tensor data recovery method based on multi-linear augmented Lagrange multiplier method
CN111325403B (en) Method for predicting residual life of electromechanical equipment of highway tunnel
CN109213753B (en) Industrial system monitoring data recovery method based on online PCA
CN114841888B (en) Visual data completion method based on low-rank tensor ring decomposition and factor prior
CN113673590A (en) Rain removing method, system and medium based on multi-scale hourglass dense connection network
CN114528755A (en) Power equipment fault detection model based on attention mechanism combined with GRU
CN113935249B (en) Upper-layer ocean thermal structure inversion method based on compression and excitation network
CN113094860A (en) Industrial control network flow modeling method based on attention mechanism
CN116032557A (en) Method and device for updating deep learning model in network security anomaly detection
CN106874881B (en) A kind of anti-joint sparse expression method for tracking target in the part of multi-template space time correlation
CN115829424A (en) Traffic data restoration method based on non-parametric non-convex relaxation low-rank tensor completion
CN111723857B (en) Intelligent monitoring method and system for running state of process production equipment
CN116894180B (en) Product manufacturing quality prediction method based on different composition attention network
CN117074627B (en) Medical laboratory air quality monitoring system based on artificial intelligence
CN117131654A (en) Target observation method based on nonlinear optimal disturbance of pre-analysis initial guess condition
CN114119508A (en) Shield tunnel surrounding rock quality judgment method based on monitoring video
CN111259762B (en) Pantograph abnormity detection method
CN111045861B (en) Sensor data recovery method based on deep neural network
CN111462014A (en) Single-image rain removing method based on deep learning and model driving
CN116125922B (en) Complex industrial process monitoring method and system based on parallel dictionary learning
CN117935549A (en) Expressway traffic abnormality detection method
CN117056327A (en) Tensor weighted gamma norm-based industrial time series data complement method
CN117687297A (en) Spacecraft control system anomaly detection method and system based on graph neural network fusion of self-adaptive correlation analysis

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination