CN109101638B - A kind of Dam Deformation Monitoring continuity missing data complementing method - Google Patents

A kind of Dam Deformation Monitoring continuity missing data complementing method Download PDF

Info

Publication number
CN109101638B
CN109101638B CN201810946824.6A CN201810946824A CN109101638B CN 109101638 B CN109101638 B CN 109101638B CN 201810946824 A CN201810946824 A CN 201810946824A CN 109101638 B CN109101638 B CN 109101638B
Authority
CN
China
Prior art keywords
data
deformation monitoring
time
completion
missing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201810946824.6A
Other languages
Chinese (zh)
Other versions
CN109101638A (en
Inventor
毛莺池
张建华
高建
陈豪
平萍
王龙宝
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hohai University HHU
Original Assignee
Hohai University HHU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hohai University HHU filed Critical Hohai University HHU
Priority to CN201810946824.6A priority Critical patent/CN109101638B/en
Publication of CN109101638A publication Critical patent/CN109101638A/en
Application granted granted Critical
Publication of CN109101638B publication Critical patent/CN109101638B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0793Remedial or corrective actions
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2216/00Indexing scheme relating to additional aspects of information retrieval not explicitly covered by G06F16/00 and subgroups
    • G06F2216/03Data mining

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention discloses a kind of Dam Deformation Monitoring continuity missing data complementing method, this method solves the problems, such as that continuity lacks completion in Dam Deformation Monitoring data.Firstly, being pre-processed to continuity missing in deformation measurement data;Secondly, respectively from global space, length of a game, local space, local time and semantic angle, space-time similitude and functional similarity between calculating deformation monitoring measuring point carry out global and local space, temporal interpolation and semantic interpolation to missing data in deformation measurement data;Finally, construction depth neural network model indicates ability using deep neural network, realizes non-linear fusion, complete continuity missing data completion in Dam Deformation Monitoring using above-mentioned deformation monitoring missing data completion PRELIMINARY RESULTS as input.

Description

Dam deformation monitoring continuity missing data completion method
Technical Field
The invention relates to a continuity data missing data completion method, in particular to a dam deformation monitoring missing data completion method based on multi-view deep fusion, which completes dam deformation monitoring continuity missing data completion by capturing a complex nonlinear space-time relation through a deep neural network and belongs to the technical field of data mining.
Background
A large number of sensors are deployed in a concrete dam and cooperate with each other to continuously monitor the real-time status of dam deformation. Data generated by the sensor has space-time characteristics, but due to influence factors such as hardware, communication errors and serious wireless interference of the sensor, a large amount of loss of original sensor data is generated, and in extreme cases, continuous data loss is generated. These data deletions not only affect real-time monitoring, but also are not conducive to further research analysis and decision making. Therefore, it is important to complement the missing data.
Studies on data missing completion have been advanced with single view completion, such as K-nearest neighbor based local spatial interpolation, Kriging (Kriging) interpolation, and multi-channel singular spectral analysis based on principal component analysis. However, these methods do not capture the spatio-temporal relationship among the data well and the interpolation completion quality is not high. With the rapid development of sparse representation, the matrix completion technique is widely applied to random data loss caused by unstable wireless transmission. Different from the method, the matrix completion utilizes the low-rank characteristic in the data, and the random data missing completion is well completed through the space-time relation.
However, for the missing of continuous data of whole row or whole column, it is very difficult to complete with the existing method, wherein the main reasons are that history related data cannot be found for the missing continuous data, and stable continuous input data is lacking. When the continuous data is missing, the existing matrix completion technology does not work. Existing methods address this difficulty by global initialization for historical inputs that are not stable. Such as multi-view non-negative matrix factorization based, collaborative filtering based data completion, and multi-view fusion based missing data completion, these studies have shifted from single-view to multi-view fusion learning. However, the effect is not obvious because linear fusion is adopted to complete missing data completion.
Another challenge to address continuity deficiency completion is to capture complex nonlinear spatiotemporal relationships in the data source. Aiming at the problems that information redundancy is caused by linear fusion in multiple views in the prior work and complex nonlinear space-time relation in data cannot be well captured, the invention discloses a method for complementing continuous missing data of dam deformation monitoring by simultaneously fusing views such as space-time and semantics and the like by utilizing a deep neural network technology, thereby completing the complementing of the continuous missing data of dam deformation monitoring and obtaining better accuracy and universality.
Disclosure of Invention
The purpose of the invention is as follows: the invention discloses a dam deformation monitoring continuity missing completion method aiming at stable history input of dam deformation monitoring continuity missing data missing and complex nonlinear space-time relation contained in the history input.
The technical scheme is as follows: a dam deformation monitoring continuity missing data completion method firstly defines the technical name of the invention as follows:
define 1 measurement point set: each sensor is arranged as a measuring point, the set of all deformation monitoring measuring points form a measuring point set, and S is { S ═ S { (S)1,…,Si,…,Sm}。
Define 2 set of timestamps: each deformation monitoring measuring point generates data at each moment, a set formed by all recorded moments forms a timestamp set, and T is { T ═ T1,…,ti,…,tn}。
Definition of3, monitoring a matrix: a deformation Monitoring data Matrix (Monitoring Matrix) is formed by a dam deformation Monitoring measuring point set and a time stamp set together, and the MM is formally definedS×TEach of which is a physical element mmi,tAnd representing deformation monitoring data of the measuring point i at the time stamp t, wherein the measuring point and the dam deformation monitoring measuring point have the same meaning.
Definition 4 continuous deletion: the continuity loss in dam deformation monitoring is divided into time continuity loss and space continuity loss. The lack of time continuity means that a single measuring point is lost in a certain sliding window when the deformation monitoring measuring point is under the window. The lack of spatial continuity means that data loss occurs to all dam deformation monitoring measuring points at the same timestamp.
The method realizes nonlinear fusion through a neural network, reduces redundant information, completes dam deformation monitoring continuity missing data completion, and specifically comprises the following seven steps:
(1) preprocessing the continuity loss of the dam deformation monitoring data by using Inverse Distance Weighted Interpolation (IDW) and Simple Exponential Smoothing interpolation (SES);
(2) from the perspective of global space, calculating the spatial similarity between deformation monitoring measuring points by using a reverse distance weighted interpolation method, and performing global space interpolation on missing data to obtain a deformation monitoring missing data completion preliminary result MM _ gs;
(3) from the perspective of global time, calculating the time similarity among deformation monitoring measuring points by using a bidirectional simple exponential smoothing interpolation method, and performing global time interpolation on missing data to obtain a deformation monitoring missing data completion preliminary result MM _ gt;
(4) from the perspective of local space, calculating local spatial similarity among deformation monitoring measuring points by using a User Collaborative Filtering (UCF) method, and performing local spatial interpolation on missing data to obtain a deformation monitoring missing data completion preliminary result MM _ ls;
(5) from the perspective of local time, performing local time interpolation on the deformation monitoring missing data by using a MassDiffusion Collaborative Filtering (MD-CF) method based on a measuring point-time bipartite graph to obtain a deformation monitoring missing data completion preliminary result MM _ lt;
(6) from the semantic perspective, calculating the functional similarity between deformation monitoring measuring points by using a text feature Structure (SE) nesting method, and performing semantic interpolation on the missing data to obtain a deformation monitoring missing data completion preliminary result MM _ sem;
(7) and (3) constructing an artificial neural network model, taking the initial missing data completion result of the steps (2) to (6) as input, training by using a deep neural network, realizing nonlinear fusion, and completing dam deformation monitoring continuous missing data completion.
In the step (1), the continuity missing pre-processing of the dam deformation monitoring data is carried out for the sparsity problem caused by continuity missing, the pre-estimation of the continuity missing data of the original dam deformation monitoring is carried out, and the method is mainly divided into three processes, namely IDW interpolation completion continuity missing data, SES interpolation completion continuity missing data and linear fusion to generate completion initialization values. The method comprises the following specific steps:
(1.1) interpolation of global space view to complement continuity missing data: the method is inverse distance weighted interpolation. Calculating the distance between the measuring point of the deformation monitoring missing data and all dam candidate monitoring measuring points, giving the weight to each adjacent candidate monitoring measuring point, and utilizing a formulaAnd completing the continuous missing data. Wherein, mmi,tFor monitoring data at point i at time stamp t, diIs the space distance between each candidate measuring point and the target measuring point, α is an attenuation factor, di To give a weight to point i, MMgsMM as a preliminary result of global spatial completiongsDeformation monitoring representing each completion in a continuous absenceAnd (4) data.
(1.2) the global time view completes the continuity missing data: the method is a bidirectional simple exponential smoothing interpolation. In the invention, not only the historical data of the target timestamp but also the future data of the target timestamp are considered, and exponential smoothing weighting is carried out from two directions to complete the data completion of the missing of the target timestamp. Given a target timestamp t, for mmi,tThe candidate time stamp t of the measurement point i is obtained from the monitoring dataxThe monitoring data is distributed with respective weight value ofPerforming exponential weighting by using formulaObtaining a completion result, wherein | t in the formulax-t | is a candidate timestamp txTime interval with target timestamp t, β being a smoothing factor, txX is more than or equal to 1 and less than or equal to n represents all time stamps, MMgtDeformation monitoring data representing each completion in the continuity deficiency is obtained as a global temporal completion preliminary result.
(1.3) linearly fusing the global spatiotemporal view completion result: and performing linear fusion on the global space view completion result and the global time view completion result, and taking the average value of the global space view completion result and the global time view completion result as each initial value of the continuity loss of the deformation monitoring.
The step (2) utilizes the global spatial correlation to complete the data global spatial view after the dam deformation monitoring continuity loss preprocessing, and comprises the following specific steps:
(2.1) calculating the space distance d between each candidate deformation monitoring measuring point i and the target measuring pointiGiving corresponding weight d to each candidate deformation monitoring measuring point by using distancei
(2.2) estimating a missing value through the given weight, taking the obtained result as a completion preliminary result of the global space view, wherein the calculation mode is thatMMgsThe method is used as a global space estimation result of each continuity missing completion in dam deformation monitoring data after global space-time fusion preprocessing.
And (3) completing the data after the continuity loss pretreatment of dam deformation monitoring by using the global time correlation in a global time view, wherein the specific steps are as follows:
(3.1) giving corresponding weight values to respective candidate timestamps of the dam deformation monitoring measuring points i according to bidirectional simple exponential smoothing, wherein the weight values of the candidate timestamps areWhere | tx-t | is candidate dataMm from target datai,tThe time interval of (c).
(3.2) estimating a missing value through the given weight, taking the obtained result as a pre-estimated value of the global time view, and calculating the result in a mode ofWherein, MMgtβ is a smoothing factor, t is a global time completion estimation result of each continuity missing completion in dam deformation monitoring data after global space-time fusion preprocessingxAnd x is more than or equal to 1 and less than or equal to n represents all the time stamps.
And (4) completing the data after the dam deformation monitoring continuity loss preprocessing in the local space view by utilizing the local space correlation. Modeling is performed through data-driven user collaborative filtering in the recommendation system. And taking each dam deformation monitoring measuring point as a user, and taking a time stamp as a project. The specific implementation steps are as follows:
(4.1) calculating the data in different time stamps by measuring the data in different time stampsSimilarity. Considering the measurement scale problem of each measurement point, adopting a sliding window omega to pass through the monitoring data of the measurement point uAnd monitoring data of measuring point vCalculating the cosine similarity of the data correction of the two measuring points, whereinAnda time stamp is represented which is a time stamp,indicates that the measurement point u is atThe resulting data. The cosine similarity is calculated in the manner ofWherein, sim (S)u,Sv) For the similarity of point u and point v,andmean values of the data of two measuring points, IuAnd IvRespectively representing the set of timestamps of the measuring points u and v without missinguvThe measuring points u and v are time stamp sets with data at the same time stamp. mm isu,t,mmv,tThe entity data in the monitoring matrix MM represents the monitoring data of the u measuring point and the v measuring point at the time stamp of t.
(4.2) sorting the similarity of the users in a descending order, and selecting the first k measuring points to form a nearest neighbor set V of the target measuring points, wherein V is{V1,V2,…,Vk},So that
(4.3) utilization ofWeight assignment is carried out according to the similarity to obtain a missing data completion preliminary result MMls。MMlsThe estimation result is used as a local space completion estimation result of each continuity missing completion in dam deformation monitoring data after global space-time fusion preprocessing.
And (5) completing the data after the dam deformation monitoring continuity loss preprocessing in a local time view by utilizing local time correlation. The invention applies a substance diffusion method in physics to data missing completion, and in a substance diffusion algorithm based on a graph model, a bipartite graph is used for representing the relation between a user and a project. The stations are users, and each time stamp is an item. When a certain measuring point is not lost at a certain time, one side is connected, and the nodes of the same type are not connected. Each step of substance diffusion can be used to find the degree of association between two nodes in the network structure.
Set of deformation monitoring measurement points S ═ S1,…,Si,…,SmT, set of timestamps T ═ T1,…,ti,…,tnIf it is time stamp tjAt the measuring point SiWith data, there is an edge a between themij1, otherwise aij0. Whether different timestamps generate data by the same measuring point or not can be obtained through the measuring point-time bipartite graph, and therefore whether similarity exists between the timestamps is directly judged. The specific calculation steps are as follows:
(5.1) setting a time stamp tiInitial energy of e0Wherein e is0Is expressed as:wherein,represents the measurement point u at tiThe monitoring data of (2) is obtained,represents the average monitoring data of the measuring point u at tiWhen data exists at the moment, the edge is positioned in the point-time bipartite graphOtherwise, the value is 0, and m is the number of measuring points.
(5.2) energy is first spread from the time node to the survey node. The measuring point u distributes its energy to tiMeasurement points with data, at tiThe energy of the measuring point u with the monitoring data is recordedThe calculation formula is expressed as:where k (t)i) Is the time t in the point-time bipartite graphiIs at tiThe number of measuring points with monitoring data, and the measuring point u at the time tiIf no data is missing, the edge in the point-time bipartite graphOtherwise it is 0.
(5.3) the energy is diffused to the time node from the measuring point node along the edge in the bipartite graph according to the direction opposite to the first time, namely the measuring point node distributes the currently owned energy to t according to the degree of the measuring point per seiTime node with monitoring data, time tjThe final energy of the node is the energy diffused from all the measuring point nodes connected with the nodeThe sum of the quantities is accumulated. After two diffusions, the final time node has energy representing time tjFrom tiThe obtained energy proportion reflects the similarity between two time nodes and is recorded asIs calculated by the formulaWhere k (u) is the degree of point u in the point-time bipartite graph. Measured point u at time tjIf no data is missing, thenOtherwise it is 0. Measured point u at time tiData not missing thenOtherwise it is 0.
(5.4) according to the similarity calculation formula, carrying out similarity degree sorting to obtain the target timestamp tiNearest neighbor set Nt
(5.5) using the traditional collaborative filtering algorithm to carry out weight distribution according to the similarity to obtain the MM of the primary result of completing the deformation monitoring missing dataltThe calculation formula is expressed as:whereinRepresents the measurement point u at tiTime stamped monitoring data. MM (Measure and Regulation)ltThe estimation result is used as the local time completion estimation result of each continuous missing completion in the dam deformation monitoring data after global space-time fusion preprocessing.
And (6) completing the data after the dam deformation monitoring continuity missing preprocessing in the semantic view by utilizing semantic similarity. And (5) excavating text characteristics of the measuring points, and completing dam deformation monitoring missing data completion. The specific implementation steps are as follows:
(6.1) constructing a graph for representing the functional similarity between the measuring points of the dam deformation monitoring area, wherein each vertex in the graph represents one measuring point, and each edge connects two vertexes to represent two measuring points as neighbors. Defining a semantic graph of the monitoring data as G ═ (V, E, D), a measuring point set S, wherein each measuring point represents a vertex, namely V ═ S,e is the set of edges and D is the similarity between the edges.
(6.2) measuring the measuring point S by Dynamic Time Warping (DTW)iAnd SjSimilarity between themThe similarity calculation formula isWherein α controls the degree of weight attenuation with distance, DTW (S)i,Sj) Is a measuring point SiAnd SjDynamic time warping distance in between. Any two vertices in the graph can be reached, and the graph has interoperability.
(6.3) encoding each station into a computable low-dimensional vector and retaining structural information using graph embedding methods in the graph. For each measuring point SiOutputting the embedding feature vector MM using a pattern embedding methodi. In order to integrate learning embedded feature vectors, the feature vectors are put into a full connection layer to obtain a dam deformation monitoring missing data completion preliminary result MMi. The calculation method comprises the following steps: MM (Measure and Regulation)i=f(WfeMMi+bfe). Wherein WfeAnd bfeTo learn the parameters, f is a linear activation function. The graph embedding method adopted by the invention is linear embedding. MM (Measure and Regulation)iDam deformation monitoring data after global space-time fusion preprocessingAnd (4) continuously missing a supplemented semantic estimation result from a single measuring point.
And (7) obtaining a final completion result through a multi-view deep fusion learning framework based on a neural network, so as to realize better accuracy. The method comprises the following specific steps:
and (7.1) taking the dam deformation monitoring missing data completion preliminary result of the steps (2) to (6) as input, and training in a neural network with 6 hidden layers. The neurons of the 6 hidden layers are 64, 128, 256, 64, 32, respectively. After normalization, each layer is processed and activated using a modified Linear Unit (RELU).
And (7.2) selecting the optimal hyper-parameter by adopting five-fold cross validation to obtain a more reliable and stable model. MM for obtaining final deletion completion result by adopting a linear activation function at an output layercompAnd completing dam deformation monitoring continuity missing data completion.
Has the advantages that: compared with the prior art, the dam deformation monitoring continuity missing data completion method provided by the invention is mainly characterized by solving the sparsity problem caused by continuity missing, reducing information redundancy through deep neural network deep fusion and completing dam deformation monitoring continuity missing data completion.
Drawings
FIG. 1 is an overall framework diagram of a multi-view fusion learning method according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a deformation monitoring continuity loss preprocessing operation according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating IDW completion continuity loss flow operation according to an embodiment of the present invention;
FIG. 4 is a two-way simple exponential smoothing completion continuity deficiency flow diagram of an embodiment of the present invention;
FIG. 5 is a working diagram of a UCF completion continuity loss flow of an embodiment of the present invention;
FIG. 6 is a plot of point-time bipartite graph according to an embodiment of the invention;
FIG. 7 is a working diagram of an MD-CF completion continuity loss flow of an embodiment of the present invention;
FIG. 8 is a depth fusion map of an embodiment of the present invention.
Detailed Description
The present invention is further illustrated by the following examples, which are intended to be purely exemplary and are not intended to limit the scope of the invention, as various equivalent modifications of the invention will occur to those skilled in the art upon reading the present disclosure and fall within the scope of the appended claims.
Fig. 1 provides an overall frame diagram of a dam deformation monitoring continuity missing data completion method, which is divided into two parts, namely dam deformation monitoring continuity missing data pre-estimation and dam deformation monitoring continuity missing data fusion completion. And respectively obtaining respective completion results of missing data through IDW, SES, UCF, MD-CF and SE, and completing dam deformation monitoring continuity missing data completion through nonlinear fusion of a neural network.
The implementation process of the invention is described as follows:
(1) preprocessing two types of continuity loss in dam deformation monitoring continuity loss data by utilizing reverse distance weighted interpolation and bidirectional simple exponential smoothing interpolation to complete dam deformation monitoring continuity loss pre-estimation;
(2) from the perspective of the global space, calculating the spatial similarity between deformation monitoring measuring points by using a reverse distance weighted interpolation method, and performing global spatial interpolation on the missing data to obtain a dam deformation monitoring missing data completion preliminary result 1;
(3) from the perspective of overall time, calculating time similarity among deformation monitoring measuring points by using a bidirectional simple exponential smoothing interpolation method, and performing overall time interpolation on missing data to obtain a dam deformation monitoring missing data completion preliminary result 2;
(4) from the perspective of local space, calculating local space similarity among deformation monitoring measuring points by using a user collaborative filtering method, and performing local space interpolation on missing data to obtain a dam deformation monitoring missing data completion preliminary result 3;
(5) from the angle of local time, local time interpolation is carried out on the missing data by using an energy diffusion collaborative filtering method based on a measuring point-time bipartite graph, and a dam deformation monitoring missing data completion preliminary result 4 is obtained;
(6) from the semantic perspective, calculating the functional similarity between the measuring points by using a text characteristic structure nesting method, and performing semantic interpolation on the missing data to obtain a dam deformation monitoring missing data completion preliminary result 5;
(7) and (3) constructing an artificial neural network model, taking the initial missing data completion result of the steps (2) to (6) as input, training by using a deep neural network, realizing nonlinear fusion, and completing dam deformation monitoring continuous missing data completion.
Fig. 2 is a flow chart of the continuous loss pretreatment work, and it can be seen that the dam deformation monitoring continuous loss pretreatment specifically comprises the following steps:
(1.1) the global space view completes the continuity missing data: inverse distance weighted interpolation is used. Calculating the distance d between the measuring point where the missing data is located and all dam candidate monitoring measuring points iiAssigning the weight of each dam candidate monitoring measuring point as di Using the formulaAnd calculating a global space completion result. Wherein, mmi,tIs the monitoring data, MM, of the measuring point i at the time stamp tgsMonitoring continuity loss number as dam deformationAnd completing the preliminary result according to the global space.
(1.2) complementing dam deformation monitoring continuity missing data by using a global time view: the method is a bidirectional simple exponential smoothing interpolation. In the invention, not only the historical data of the target timestamp but also the future data of the target timestamp are considered, and exponential smoothing weighting is carried out from two directions to complete the data completion of the missing of the target timestamp. Given a target timestamp t, for mmi,tThe candidate time stamp t of the measurement point i is obtained from the monitoring dataxThe monitoring data at is assigned respective candidate timestamps txHas a weight value ofPerforming exponential weighting by using formulaObtaining a completion result, wherein | t in the formulax-t | is a candidate timestamp txTime interval with target timestamp t, β being a smoothing factor, txX is more than or equal to 1 and less than or equal to n represents all time stamps, MMgtThe method is used as a dam deformation monitoring continuity missing data global time completion preliminary result.
(1.3) linearly fusing the global spatiotemporal view completion result: and (4) performing linear fusion on the global space view completion result and the global time view completion result, and taking the average value of the global space view completion result and the global time view completion result as each initial value for monitoring continuity loss of the dam deformation.
Wherein the content (2) and the content (3) are processed in the same way as the step (1.1) and the step (1.2), as shown in fig. 3 and fig. 4.
And the content (4) utilizes the user collaborative filtering to mine the spatial similarity of the local measuring points, and the continuity missing data completion is carried out. And taking the measuring points as users, taking the time stamps as projects, and mining the spatial similarity of the local measuring points as a basis for completing the continuity loss. Fig. 5 shows a flow chart of UCF for continuous deletion completion, and it can be seen that the specific steps are as follows:
and (4.1) calculating similarities in different time stamps by measuring data of the time stamps. Considering the measurement scale problem of each measurement point, adopting a sliding window omega to pass through the monitoring data of the measurement point uAnd monitoring data of measuring point vCalculating the cosine similarity of the data correction of the two measuring points, whereinAnda time stamp is represented which is a time stamp,indicates that the measurement point u is atThe resulting data. The cosine similarity is calculated in the manner ofWherein, sim (S)u,Sv) For the similarity of point u and point v,andmean values of the data of two measuring points, IuAnd IvRespectively representing the set of timestamps of the measuring points u and v without missinguvThe measuring points u and v are time stamp sets with data at the same time stamp. mm isu,t,mmv,tThe entity data in the monitoring matrix MM represents the monitoring data of the u measuring point and the v measuring point at the time stamp of t.
(4.2) decreasing the user similaritySequencing, selecting the first k measuring points to form a nearest neighbor set V of the target measuring points, wherein V is { V ═ V1,V2,…,Vk},So that
(4.3) utilization ofWeight distribution is carried out according to similarity, and a preliminary dam deformation monitoring missing data completion result MM is obtainedls,MMlsAnd representing a local space estimation result of each continuity missing data completion in the dam deformation monitoring data after global space-time fusion preprocessing.
The content (5) completes dam deformation monitoring continuity missing data from a local time view by utilizing local time correlation. Aiming at sparsity caused by continuity loss, the invention combines a substance diffusion method in physics with a bipartite graph to be applied to data loss completion. Fig. 6 shows a point-time bipartite graph, and fig. 7 shows a flow chart of MD-CF filling dam deformation monitoring continuity loss, which shows the following specific calculation steps:
(5.1) setting a time stamp tiInitial energy of e0Wherein e is0Is expressed as:wherein,represents the measurement point u at tiThe monitoring data of (2) is obtained,represents the average monitoring data of the measuring point u at tiIf data exist at the moment, a point-time bipartite graph is obtainedMiddle edgeOtherwise, the value is 0, and m is the number of measuring points.
(5.2) energy is first spread from the time node to the survey node. The measuring point u distributes its energy to tiMeasurement points with data, at tiThe energy of the measuring point u with the monitoring data is recordedThe calculation formula is expressed as:where k (t)i) Is the time t in the point-time bipartite graphiIs at tiThe number of measuring points with monitoring data, and the measuring point u at the time tiIf no data is missing, the edge in the point-time bipartite graphOtherwise it is 0.
(5.3) the energy is diffused to the time node from the measuring point node along the edge in the bipartite graph according to the direction opposite to the first time, namely the measuring point node distributes the currently owned energy to t according to the degree of the measuring point per seiTime node with monitoring data, time tjThe final energy of the node is the sum of the energy diffused from all the measuring point nodes connected with the node. After two diffusions, the final time node has energy representing time tjFrom tiThe obtained energy proportion reflects the similarity between two time nodes and is recorded asIs calculated by the formulaWhere k (u) is the degree of point u in the point-time bipartite graph. Measured point u at time tjIf no data is missing, thenOtherwise it is 0. Measured point u at time tiData not missing thenOtherwise it is 0.
(5.4) according to the similarity calculation formula, carrying out similarity degree sorting to obtain the target timestamp tiNearest neighbor set Nt
(5.5) carrying out weight distribution according to the similarity by utilizing the traditional collaborative filtering algorithm to obtain a missing data completion preliminary result MMltThe calculation formula is expressed as:whereinRepresents the measurement point u at tiTime stamped monitoring data, MMltAnd representing a local time estimation result of each continuity missing data completion in the dam deformation monitoring data after global space-time fusion preprocessing.
And the content (6) completes dam deformation monitoring continuity missing data by utilizing semantic similarity. And (5) digging text characteristics of the measuring points, and completing data completion of dam deformation monitoring continuity loss. The specific implementation steps are as follows:
(6.1) constructing a graph for representing the functional similarity between the measuring points of the dam deformation monitoring area, wherein each vertex in the graph represents one measuring point, and each edge connects two vertexes to represent two measuring points as neighbors. Defining a semantic graph of the monitoring data as G ═ (V, E, D), a measuring point set S, wherein each measuring point represents a vertex, namely V ═ S,e is the set of edges and D is the similarity between the edges.
(6.2) measuring the measuring point S by Dynamic Time Warping (DTW)iAnd SjSimilarity between themThe similarity calculation formula isWherein α controls the degree of weight attenuation with distance, DTW (S)i,Sj) Is a measuring point SiAnd SjDynamic time warping distance in between. Any two vertices in the graph can be reached, and the graph has interoperability.
(6.3) encoding each station into a computable low-dimensional vector and retaining structural information using graph embedding methods in the graph. For each measuring point SiOutputting the embedding feature vector MM using a pattern embedding methodi. In order to integrate learning and embedding the feature vectors, the feature vectors are put into a full-link layer to obtain a missing data completion preliminary result MMi. The calculation method comprises the following steps: MM (Measure and Regulation)i=f(WfeMMi+bfe). Wherein WfeAnd bfeTo learn the parameters, f is a linear activation function. The graph embedding method adopted by the invention is linear embedding. MM (Measure and Regulation)iRepresenting a semantic estimation result of dam deformation monitoring data single measuring point continuity missing data completion after global space-time fusion preprocessing.
And the content (7) obtains a final completion result of dam deformation monitoring continuity loss through a multi-view deep fusion learning framework based on a neural network. From fig. 8, which is a depth fusion map, it can be seen that the method includes the following specific steps:
and (7.1) taking the dam deformation monitoring missing data completion preliminary result of the steps (2) to (6) as input, and training in a neural network with 6 hidden layers. The neurons of the 6 hidden layers are 64, 128, 256, 64, 32, respectively. After normalization, each layer is processed and activated using a modified Linear Unit (ReLU).
And (7.2) selecting the optimal hyper-parameter by adopting five-fold cross validation to obtain a more reliable and stable model. MM for obtaining final deletion completion result by adopting a linear activation function at an output layercompAnd completing dam deformation monitoring continuity missing data completion.

Claims (7)

1. A dam deformation monitoring continuity missing data completion method is characterized by comprising the following steps:
(1) preprocessing the continuity loss of dam deformation monitoring data by utilizing reverse distance weighted interpolation and bidirectional simple exponential smoothing interpolation;
(2) from the perspective of the global space, calculating the spatial similarity between the dam deformation monitoring measuring points by using a reverse distance weighted interpolation method, and performing global spatial interpolation on the missing data to obtain a dam deformation monitoring missing data completion preliminary result MM _ gs;
(3) from the perspective of overall time, calculating the time similarity between dam deformation monitoring measuring points by using a bidirectional simple exponential smoothing interpolation method, and performing overall time interpolation on the missing data to obtain a dam deformation monitoring missing data completion preliminary result MM _ gt;
(4) from the perspective of local space, calculating local space similarity among deformation monitoring measuring points by using User Collaborative Filtering (UCF), and performing local space interpolation on missing data to obtain a dam deformation monitoring missing data completion preliminary result MM _ ls;
(5) from the angle of local time, local time interpolation is carried out on deformation monitoring missing data by using an energy diffusion collaborative filtering method based on a measuring point-time bipartite graph to obtain a dam deformation monitoring missing data completion preliminary result MM _ lt;
(6) from the semantic perspective, by using a text feature structure nesting (Structural Embedding) SE method, calculating the functional similarity between dam deformation monitoring points, and performing semantic interpolation on the missing data to obtain a dam deformation monitoring missing data completion preliminary result MM _ sem;
(7) and (3) constructing an artificial neural network model, taking the initial missing data completion result of the steps (2) to (6) as input, training by using a deep neural network, realizing nonlinear fusion, and completing dam deformation monitoring continuous missing data completion.
2. The dam deformation monitoring continuity missing data completion method according to claim 1, wherein the step (2) utilizes global spatial correlation to perform completion on dam deformation monitoring continuity missing data in a global spatial view, and comprises the following specific steps:
(2.1) calculating the space distance d between each candidate deformation monitoring measuring point i and the target deformation monitoring measuring pointiGiving corresponding weight d to each candidate deformation monitoring measuring point by using distancei The candidate deformation monitoring measuring points are dam deformation monitoring measuring points except the target deformation monitoring measuring point;
(2.2) estimating missing values through the given weight values, and taking the obtained result as a global nullThe primary result of the complementation of the interview is calculated asMMgsAs a global space completion preliminary result.
3. The dam deformation monitoring continuity missing data completion method according to claim 1, wherein the step (3) utilizes global time correlation to perform completion on dam deformation monitoring continuity missing data in a global time view, and comprises the following specific steps:
(3.1) giving corresponding weight values of respective candidate timestamps of the measuring points i according to bidirectional simple exponential smoothing, wherein the candidate timestamps are txHas a weight value ofWhere | tx-t | is candidate deformation monitoring dataData mm of target deformation monitoringi,tThe time interval of (c);
(3.2) estimating a missing value through the given weight, taking the obtained result as a pre-estimated value of the global time view, and calculating the result in a mode ofWherein, MMgtAs a preliminary result of global time completion, β is a smoothing factor, txAnd x is more than or equal to 1 and less than or equal to n represents all the time stamps.
4. The dam deformation monitoring continuity missing data completion method according to claim 1, wherein the step (4) is implemented by the following steps:
(4.1) calculating similarities in different timestamps through data of the deformation monitoring measuring points in different timestamps; considering the measurement scale problem of each measurement point, adopting a sliding window omega to pass through the monitoring data of the measurement point uAnd monitoring data of measuring point vCalculating the cosine similarity of the data correction of the two measuring points, whereinAnda time stamp is represented which is a time stamp,indicates that the measurement point u is atThe generated data; the cosine similarity is calculated in the manner ofWherein, sim (S)u,Sv) For the similarity of point u and point v,andmean values of the data of two measuring points, IuAnd IvRespectively representing the set of timestamps of the measuring points u and v without missinguvThe measuring points u and v have time stamp sets with data in the same time stamp; mm isu,t,mmv,tRepresenting the monitoring data of the u measuring point and the v measuring point in the t time stamp for the entity data in the monitoring matrix MM;
(4.2) sorting the similarity of the users in a descending order, and selecting the first k measuring points to form a nearest neighbor set V of the target measuring points, wherein V is { V ═ V1,V2,…,Vk},So that
(4.3) utilization ofWeight assignment is carried out according to the similarity to obtain a missing data completion preliminary result MMls
5. The dam deformation monitoring continuity missing data completion method according to claim 1, wherein the step (5) comprises the steps of:
(5.1) setting a time stamp tiInitial energy of e0Wherein e is0Is expressed as:wherein,indicating that the dam deformation monitoring measuring point u is at tiThe monitoring data of (2) is obtained,representing the average monitoring data of a dam deformation monitoring measuring point u, wherein the measuring point u is at tiWhen data exists at the moment, the edge is positioned in the point-time bipartite graphOtherwise, the number is 0, and m is the number of the measuring points;
(5.2) energy is diffused from the time node to the measuring point node for the first time; the measuring point u distributes its energy to tiMeasurement points with data, at tiThe energy of the measuring point u with the monitoring data is recordedThe calculation formula is expressed as:where k (t)i) Is the time t in the point-time bipartite graphiIs at tiThe number of measuring points with monitoring data, and the measuring point u at the time tiIf no data is missing, the edge in the point-time bipartite graphOtherwise, the value is 0;
(5.3) the energy is diffused to the time node from the measuring point node along the edge in the bipartite graph according to the direction opposite to the first time, namely the measuring point node distributes the currently owned energy to t according to the degree of the measuring point per seiTime node with monitoring data, time tjThe final energy of the node is the accumulated sum of the energies diffused from all the measuring point nodes connected with the node; after two diffusions, the final time node has energy representing time tjFrom tiThe obtained energy proportion reflects the similarity between two time nodes and is recorded as Is calculated by the formulaWherein, the meaning of the set S to which u belongs in the formula is S ═ S1,…,Si,…,Sm}, k (u) is the degree of point u in the point-time bipartite graph; measured point u at time tjIf no data is missing, thenOtherwise, the value is 0; measured point u is atTime tiData not missing thenOtherwise, the value is 0;
(5.4) according to the similarity calculation formula, carrying out similarity degree sorting to obtain the target timestamp tiNearest neighbor set Nt
(5.5) using the traditional collaborative filtering algorithm to carry out weight distribution according to the similarity to obtain the MM of the primary result of completing the deformation monitoring missing dataltThe calculation formula is expressed as:whereinRepresents the measurement point u at tiTime stamped monitoring data.
6. The dam deformation monitoring continuity missing data completion method according to claim 1, characterized in that the step (6) is implemented by the following steps:
(6.1) constructing a graph for representing the functional similarity among the measuring points of the dam deformation monitoring area, wherein each vertex in the graph represents one measuring point, and each edge is connected with two vertices to represent two measuring points as neighbors; defining a semantic graph G (V, E, D) of dam deformation monitoring data, and a measuring point set S, wherein each measuring point represents a vertex, namely V (S),e is a set of edges, and D is the similarity between all edges;
(6.2) measuring point S by Dynamic Time Warping (Dynamic Time Warping, DTW)iAnd SjSimilarity between themThe similarity calculation formula isWherein α controls the degree of weight attenuation with distance, DTW (S)i,Sj) Is a measuring point SiAnd SjDynamic time warping distance between; any two vertexes in the graph can be reached, and the graph has interoperability;
(6.3) encoding each station into a computable low-dimensional vector and maintaining structural information using graph embedding in the graph; for each measuring point SiOutputting the embedding feature vector MM using a pattern embedding methodi(ii) a In order to integrate learning embedded feature vectors, the feature vectors are put into a full connection layer to obtain a dam deformation monitoring missing data completion preliminary result MMi(ii) a The calculation method comprises the following steps: MM (Measure and Regulation)i=f(WfeMMi+bfe) (ii) a Wherein WfeAnd bfeTo learn the parameters, f is a linear activation function.
7. The dam deformation monitoring continuity missing data completion method according to claim 1, characterized in that the concrete steps of the step (7) are as follows:
(7.1) taking the dam deformation monitoring missing data completion preliminary result of the steps (2) - (6) as input, training in a neural network with 6 hidden layers, and activating by using a modified linear function after processing of each layer by adopting normalization operation;
(7.2) selecting the optimal hyper-parameter by adopting five-fold cross validation, and obtaining a final deficiency completion result MM by adopting a linear activation function in an output layercompAnd completing dam deformation monitoring continuity missing data completion.
CN201810946824.6A 2018-08-20 2018-08-20 A kind of Dam Deformation Monitoring continuity missing data complementing method Active CN109101638B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810946824.6A CN109101638B (en) 2018-08-20 2018-08-20 A kind of Dam Deformation Monitoring continuity missing data complementing method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810946824.6A CN109101638B (en) 2018-08-20 2018-08-20 A kind of Dam Deformation Monitoring continuity missing data complementing method

Publications (2)

Publication Number Publication Date
CN109101638A CN109101638A (en) 2018-12-28
CN109101638B true CN109101638B (en) 2019-05-28

Family

ID=64850397

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810946824.6A Active CN109101638B (en) 2018-08-20 2018-08-20 A kind of Dam Deformation Monitoring continuity missing data complementing method

Country Status (1)

Country Link
CN (1) CN109101638B (en)

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109710893B (en) * 2019-01-23 2023-04-07 江西理工大学 Time-space interpolation method for correcting abnormal data of mine slope deformation monitoring
CN110147367B (en) * 2019-05-14 2022-05-27 中国科学院深圳先进技术研究院 Temperature missing data filling method and system and electronic equipment
CN110274995A (en) * 2019-06-18 2019-09-24 深圳市美兆环境股份有限公司 Fill the determination method, apparatus and computer equipment of data
CN112749470B (en) * 2019-10-31 2023-07-14 北京华航无线电测量研究所 Layout optimization fitting method for structural deformation sensor
CN111080009B (en) * 2019-12-13 2021-04-16 北京瑞莱智慧科技有限公司 Time series-based data prediction and completion method, device, medium, and apparatus
CN111060069B (en) * 2019-12-24 2022-04-12 河南理工大学 Settling amount obtaining method and device for structure missing monitoring point
CN111121711B (en) * 2019-12-24 2022-01-18 河南理工大学 Method and device for acquiring movement deformation of missing monitoring point in mining influence area
CN111177205B (en) * 2019-12-31 2023-04-21 重庆中电自能科技有限公司 New energy station data sharing method and system
CN111798940B (en) * 2020-06-28 2024-06-25 南方科技大学 Method and device for predicting superconducting material based on deep neural network algorithm
CN112381171B (en) * 2020-11-25 2023-04-07 河海大学 Multi-sensor node missing data filling method based on combined model
CN113139571B (en) * 2021-03-09 2022-08-02 河海大学 Dam safety monitoring data completion method based on space-time multi-view fusion
CN112836180B (en) * 2021-04-19 2021-07-23 北京瑞莱智慧科技有限公司 Sensing data determination method and device, electronic equipment and readable storage medium
CN113806349B (en) * 2021-11-18 2022-02-18 浙江大学 Spatiotemporal missing data completion method, device and medium based on multi-view learning
CN115727985A (en) * 2022-11-18 2023-03-03 大连海事大学 Portable indicator diagram measuring device
CN116595327B (en) * 2023-07-19 2023-09-29 水利部交通运输部国家能源局南京水利科学研究院 Sluice deformation monitoring data preprocessing system and method
CN116628068B (en) * 2023-07-25 2024-07-05 杭州衡泰技术股份有限公司 Data handling method, system and readable storage medium based on dynamic window
CN117473234B (en) * 2023-12-28 2024-04-30 广州地铁设计研究院股份有限公司 Deformation monitoring data preprocessing method, device, equipment and storage medium

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104217002A (en) * 2014-09-14 2014-12-17 北京航空航天大学 Traffic information filling method based on high-quality data acquisition
CN105225486A (en) * 2015-10-09 2016-01-06 哈尔滨工业大学深圳研究生院 Fill up the method and system of disappearance floating car data
CN105893610A (en) * 2016-04-26 2016-08-24 中国科学院信息工程研究所 Deficiency-source completion method of multi-source heterogeneous large data
CN106776484A (en) * 2016-12-08 2017-05-31 南京航空航天大学 A kind of large deformation flexible body dynamic stress metrical information complementing method based on space-time
CN107680377A (en) * 2017-11-06 2018-02-09 浙江工商大学 Traffic flow data based on trend fitting intersects complementing method
CN107833153A (en) * 2017-12-06 2018-03-23 广州供电局有限公司 A kind of network load missing data complementing method based on k means clusters
CN107992536A (en) * 2017-11-23 2018-05-04 中山大学 Urban transportation missing data complementing method based on tensor resolution
CN108010320A (en) * 2017-12-21 2018-05-08 北京工业大学 A kind of complementing method of the road grid traffic data based on adaptive space-time constraint low-rank algorithm

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104217002A (en) * 2014-09-14 2014-12-17 北京航空航天大学 Traffic information filling method based on high-quality data acquisition
CN105225486A (en) * 2015-10-09 2016-01-06 哈尔滨工业大学深圳研究生院 Fill up the method and system of disappearance floating car data
CN105893610A (en) * 2016-04-26 2016-08-24 中国科学院信息工程研究所 Deficiency-source completion method of multi-source heterogeneous large data
CN106776484A (en) * 2016-12-08 2017-05-31 南京航空航天大学 A kind of large deformation flexible body dynamic stress metrical information complementing method based on space-time
CN107680377A (en) * 2017-11-06 2018-02-09 浙江工商大学 Traffic flow data based on trend fitting intersects complementing method
CN107992536A (en) * 2017-11-23 2018-05-04 中山大学 Urban transportation missing data complementing method based on tensor resolution
CN107833153A (en) * 2017-12-06 2018-03-23 广州供电局有限公司 A kind of network load missing data complementing method based on k means clusters
CN108010320A (en) * 2017-12-21 2018-05-08 北京工业大学 A kind of complementing method of the road grid traffic data based on adaptive space-time constraint low-rank algorithm

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
"多视角数据缺失补全问题研究";杨旭;《万方数据》;20170228;第1-69页

Also Published As

Publication number Publication date
CN109101638A (en) 2018-12-28

Similar Documents

Publication Publication Date Title
CN109101638B (en) A kind of Dam Deformation Monitoring continuity missing data complementing method
Meng et al. Multi-fidelity Bayesian neural networks: Algorithms and applications
WO2021093468A1 (en) Video classification method and apparatus, model training method and apparatus, device and storage medium
Laloy et al. Inversion using a new low-dimensional representation of complex binary geological media based on a deep neural network
US8682624B2 (en) Deterministic version of the multiple point geostatistics simulation/reconstruction method with the simulated/reconstructed values are directly taken from the training images without prior estimation of the conditional
CN112532439B (en) Network flow prediction method based on attention multi-component space-time cross-domain neural network model
CN106529721B (en) A kind of ad click rate forecasting system and its prediction technique that depth characteristic is extracted
CN112291807B (en) Wireless cellular network traffic prediction method based on deep migration learning and cross-domain data fusion
CN102928877B (en) Based on the seismic attribute fusion method of D-S evidence theory
CN113297936B (en) Volleyball group behavior identification method based on local graph convolution network
CN109272029B (en) Well control sparse representation large-scale spectral clustering seismic facies partitioning method
WO2021184754A1 (en) Video comparison method and apparatus, computer device and storage medium
CN111047078B (en) Traffic characteristic prediction method, system and storage medium
Yu et al. A deep residual computation model for heterogeneous data learning in smart Internet of Things
CN114693064B (en) Building group scheme generation performance evaluation method
Kumar et al. Ensemble-based assimilation of nonlinearly related dynamic data in reservoir models exhibiting non-Gaussian characteristics
CN116935649A (en) Urban traffic flow prediction method for multi-view fusion space-time dynamic graph convolution network
CN113657582B (en) Gas concentration prediction method based on space-time data and space-time graph neural network
CN103473366B (en) A kind of various visual angles are across the sorting technique of data field picture material identification and device
Crocker et al. Using explainability to design physics-aware CNNs for solving subsurface inverse problems
Son et al. Partial convolutional LSTM for spatiotemporal prediction of incomplete data
CN111210051B (en) User electricity consumption behavior prediction method and system
CN116960954A (en) Short-term wind power prediction method and device, storage medium and computer equipment
CN116306277A (en) Landslide displacement prediction method and device and related components
Sung et al. Development of HT-BP neural network system for the identification of well test interpretation model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
EE01 Entry into force of recordation of patent licensing contract
EE01 Entry into force of recordation of patent licensing contract

Application publication date: 20181228

Assignee: HUANENG LANCANG RIVER HYDROPOWER Inc.

Assignor: HOHAI University

Contract record no.: X2021320000034

Denomination of invention: A method to complete the missing data of dam deformation monitoring continuity

Granted publication date: 20190528

License type: Common License

Record date: 20210525