CN110569890A - A method for detecting abnormal patterns in hydrological data based on similarity measure - Google Patents
A method for detecting abnormal patterns in hydrological data based on similarity measure Download PDFInfo
- Publication number
- CN110569890A CN110569890A CN201910784182.9A CN201910784182A CN110569890A CN 110569890 A CN110569890 A CN 110569890A CN 201910784182 A CN201910784182 A CN 201910784182A CN 110569890 A CN110569890 A CN 110569890A
- Authority
- CN
- China
- Prior art keywords
- pattern
- patterns
- distance
- meta
- similarity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 230000002159 abnormal effect Effects 0.000 title claims abstract description 36
- 238000011524 similarity measure Methods 0.000 title claims abstract description 14
- 238000001514 detection method Methods 0.000 claims abstract description 23
- 238000005259 measurement Methods 0.000 claims abstract description 15
- 239000011159 matrix material Substances 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000001186 cumulative effect Effects 0.000 claims description 2
- 230000005856 abnormality Effects 0.000 claims 1
- 238000005516 engineering process Methods 0.000 abstract description 3
- 238000007405 data analysis Methods 0.000 abstract description 2
- 230000011218 segmentation Effects 0.000 abstract description 2
- 230000007423 decrease Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 3
- 230000002547 anomalous effect Effects 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 238000000691 measurement method Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Testing And Monitoring For Control Systems (AREA)
Abstract
本发明公开一种基于相似性度量的水文数据异常模式检测方法,该方法基于关键点的线性分段表示KPRA‑PLR算法,根据关键点的定义对水文数据进行切割,对每个子序列通过PLR算法进行直线拟合,用直线的斜率ai和时间间隔Δt表示该子序列;分段后的每个子序列称为元模式,相邻元模式之间组合得到序列模式,对元模式、序列模式的相似性度量分别使用加权距离、SDTW算法,然后计算每个序列模式的异常分数即该模式与其他模式的平均距离的倒数;异常分数即为序列模式Sx的k‑近邻距离,根据k‑近邻局部检测原理计算局部异常因子LOF。本发明利用相似性度量的方法检测出的异常模式更为精准,从数据分析的角度为水文数据异常模式检测提供一种新的技术。
The invention discloses a method for detecting abnormal patterns of hydrological data based on similarity measurement. The method is based on the linear segment representation of key points and represents KPRA-PLR algorithm, cuts hydrological data according to the definition of key points, and performs PLR algorithm for each subsequence. Line fitting is performed, and the subsequence is represented by the slope a i of the straight line and the time interval Δt ; each subsequence after segmentation is called a meta-pattern, and the sequence pattern is obtained by combining adjacent meta-patterns. Using the weighted distance and SDTW algorithm respectively, the similarity measure of each sequence pattern is calculated by using the weighted distance and the SDTW algorithm, and then calculating the anomaly score of each sequence pattern, which is the inverse of the average distance between the pattern and other patterns; the anomaly score is the k-nearest neighbor distance of the sequence pattern S x , according to k- The nearest neighbor local detection principle calculates the local outlier factor LOF. The abnormal pattern detected by the method of similarity measurement in the present invention is more accurate, and a new technology is provided for the abnormal pattern detection of hydrological data from the perspective of data analysis.
Description
技术领域technical field
本发明属于水文检测技术,具体涉及一种基于相似性度量的水文数据异常模式检测方法。The invention belongs to the hydrological detection technology, in particular to a method for detecting abnormal patterns of hydrological data based on similarity measurement.
背景技术Background technique
水文时间序列是观测系统按照时间顺序获取到的一系列物理量(水位、流量、降雨量等)的观测值。水文时间序列是一种常见的复杂数据类型,它客观记录了观测系统按照时间顺序取得的观测信息。在我国,随着水利信息化的发展,水文时间序列会传输到水利信息共享平台,由工作人员处理进行保存到国家水文库中。由于采集设备的测量误差、人工操作误差、水文自身演进规律变化等影响因素都会导致水文时间序列中存在一些异常数据。Hydrological time series is a series of observed values of physical quantities (water level, flow, rainfall, etc.) obtained by the observation system in time sequence. Hydrological time series is a common complex data type, which objectively records the observation information obtained by the observation system in chronological order. In my country, with the development of water conservancy informatization, the hydrological time series will be transmitted to the water conservancy information sharing platform, processed by the staff, and stored in the national hydrological library. Due to the measurement error of the acquisition equipment, the error of manual operation, and the change of the evolution law of the hydrology itself, there will be some abnormal data in the hydrology time series.
异常检测(Anomaly Detection)主要目标是挖掘出隐藏在大型数据集中的不一般数据,即与其它数据存在明显差异的、有潜在意义的信息。而水文时间序列中异常数据检测方法与技术还处在探索阶段,业界没有完美的异常检测方案,目前针对水文时间序列异常点检测的方法有Box-Plot方法、Benford法则。The main goal of Anomaly Detection is to mine unusual data hidden in large data sets, that is, potentially meaningful information that is significantly different from other data. However, the methods and technologies for detecting abnormal data in hydrological time series are still in the exploratory stage, and there is no perfect anomaly detection solution in the industry. At present, the methods for detecting abnormal points in hydrological time series include Box-Plot method and Benford's rule.
发明内容SUMMARY OF THE INVENTION
发明目的:针对上述现有技术精准度不高,并且异常情况的检测速度慢问题,本发明提供一种基于相似性度量的水文数据异常模式检测方法,为检测水文数据存在的异常模式提供理论依据。Purpose of the invention: Aiming at the problems of low accuracy and slow detection of abnormal conditions in the prior art, the present invention provides a method for detecting abnormal patterns in hydrological data based on similarity measurement, which provides a theoretical basis for detecting abnormal patterns in hydrological data. .
技术方案:一种基于相似性度量的水文数据异常模式检测方法,所述方法基于关键点的分段线性表示法KPRA-PLR算法、时间序列模式相似性度量的方法和基于k-近邻局部异常检测算法,包括如下步骤:Technical solution: A method for detecting anomalous patterns in hydrological data based on similarity measurement, the method is based on the piecewise linear representation of key points KPRA-PLR algorithm, a method for similarity measurement of time series patterns, and local anomaly detection based on k-nearest neighbors The algorithm includes the following steps:
(1)基于关键点的分段线性表示法KPRA-PLR算法:首先根据关键点的定义对原始水文时间序列进行分割,得到每一段子序列,然后根据PLR算法对每段子序列进行直线拟合,得到直线的斜率,解出相邻两条直线间的交点,得到每个子序列起始时刻til和终止时刻tir,用(ai,til,tir)表示每一段子序列又称元模式;(1) KPRA-PLR algorithm based on piecewise linear representation of key points: First, the original hydrological time series is divided according to the definition of key points to obtain each subsequence, and then each subsequence is fitted with a straight line according to the PLR algorithm, Obtain the slope of the straight line, solve the intersection between two adjacent straight lines, and obtain the start time t il and end time t ir of each subsequence, and use ( ai , t il , t ir ) to represent each subsequence, also known as the element model;
(2)时间序列模式相似性度量的方法:将相邻元模式之间组合为序列模式;在度量两个模式相似性时,建立两个模式之间的相似性距离度量函数,用距离大小表示两者之间的相似度,每一模式的平均相似度代表该模式的异常分数;(2) Method for measuring the similarity of time series patterns: combine adjacent meta-patterns into sequential patterns; when measuring the similarity of two patterns, establish a similarity distance measure function between the two patterns, which is expressed by the distance The similarity between the two, the average similarity of each pattern represents the abnormal score of the pattern;
(3)k-近邻局部异常检测算法:用异常分数值表示该模式的k-近邻距离,得出k-近邻局部可达距离、局部可达密度、局部异常因子,最大的异常因子即为异常模式。(3) k-nearest neighbor local anomaly detection algorithm: the k-nearest neighbor distance of the model is represented by the anomaly score value, and the k-nearest neighbor local reachable distance, local reachability density, and local anomaly factor are obtained. The largest anomaly factor is the anomaly. model.
进一步的,步骤(1)包括如下过程:Further, step (1) includes the following process:
(11)将关键点定义为S={s1,s2,...,sm}为时间序列X的所有局部极值点,当极值点sk满足|sk-sk-1|<ε且|tk-tk-1|<δ时,其中ε和δ为给定的常数,tk表示各个水文物理量的观测时间,则称sk为非重要局部极值点,集合S中其他极值点为关键点;(11) Define the key point as S={s 1 , s 2 , ..., s m } as all local extreme points of the time series X, when the extreme point sk satisfies |s k -s k-1 When |<ε and |t k -t k-1 |<δ, where ε and δ are given constants, and t k represents the observation time of each hydrological physical quantity, then s k is called an unimportant local extreme point. Other extreme points in S are key points;
(12)然后根据线性分段表示算法,通过相邻两个关键点用最小二乘法拟合直线,得到直线对应的斜率,解出相邻两条直线间的交点,得到每个子序列起始时刻和终止时刻til和tir;每个子序列的模式表示为(ai,til,tir)称为元模式;(12) Then, according to the linear segment representation algorithm, use the least squares method to fit a straight line through two adjacent key points to obtain the slope corresponding to the straight line, solve the intersection between the two adjacent straight lines, and obtain the starting time of each subsequence and the termination moments t il and t ir ; the pattern of each subsequence is represented as ( ai , t il , t ir ) is called a meta-pattern;
(13)通过调整参数,根据得到的拟合直线和原曲线形态吻合程度确定适合当前数据的常数ε和δ值。(13) By adjusting the parameters, determine the constant ε and δ values suitable for the current data according to the degree of conformity between the obtained fitted straight line and the original curve shape.
步骤(2)所述的时间序列模式相似性度量的方法中,对元模式、序列模式分别建立相似性距离度量函数,具体如下:In the method for measuring the similarity of time series patterns described in step (2), a similarity distance measure function is established for the meta-pattern and the sequence pattern respectively, and the details are as follows:
(21)对于两个元模式之间的相似程度,采用加权距离方法,由PLR算法得到时间序列X的模式序列为Sx={M1,M2,M3,...,Mn},Sx中任意选择两个元模式Mi=(ai,Δti),Mj=(aj,Δtj),(i,j=1,2,3,...,n),(21) For the degree of similarity between two meta-patterns, using the weighted distance method, the pattern sequence of the time series X obtained by the PLR algorithm is S x ={M 1 , M 2 , M 3 ,..., Mn } , two meta-patterns M i =(a i ,Δt i ), M j =(a j ,Δt j ),(i,j=1,2,3,...,n) are arbitrarily selected in S x ,
记作:Dwei(Mi,Mj)=β·|ai-aj|+(1-β)·|Δti-Δtj|Denoted as: D wei (Mi, M j )=β·|a i -a j |+(1-β)·|Δt i -Δt j |
式中,ai表示拟合直线的斜率,Δtj表示j点相邻的时间间隔,Mi表示元模式,Dwei(Mi,Mj)为元模式Mi和Mj的加权距离,0≤β≤1;In the formula, a i represents the slope of the fitted line, Δt j represents the time interval adjacent to point j , Mi represents the meta-pattern, D wei (Mi, M j ) is the weighted distance between the meta-patterns Mi and M j , 0≤ β≤1;
(22)将相邻的元模式两两相组合得到序列模式,序列模式的相似性度量是在元模式的相似性基础上,使用动态时间弯曲距离法来度量序列模式的相似性;(22) Combining adjacent meta-patterns in two phases to obtain sequential patterns, the similarity measurement of sequential patterns is based on the similarity of meta-patterns, using dynamic time-warped distance method to measure the similarity of sequential patterns;
时间序列X=(x1,x2,x3,...,xn)和时间序列Y=(y1,y2,...,ym),通过KPRA-PLR算法进行降维、线性表示得到的模式序列分别为:Time series X=(x 1 , x 2 , x 3 ,..., x n ) and time series Y=(y 1 , y 2 ,..., y m ), through the KPRA-PLR algorithm for dimensionality reduction, The pattern sequences obtained by linear representation are:
Sx={M1,M2,M3,...,Mp},S x = {M 1 , M 2 , M 3 , ..., M p },
Sy={N1,N2,N3,...,Nt},S y = {N 1 , N 2 , N 3 , ..., N t },
其中(i=1,2,3,...,p;j=1,2,3,...,t),Mi=(ai,Δti)是Sx的元模式,where ( i = 1 , 2, 3 , .
Nj=(aj,Δtj)是Sy的元模式;建立两个模式序列间的距离矩阵计算公式如下:N j = (a j , Δt j ) is the meta-pattern of S y ; the calculation formula for establishing the distance matrix between two pattern sequences is as follows:
DM=(aij)p×t DM=(a ij ) p×t
式中,aij=Dwei(Mi,Nj)表示为模式序列Sx的第i个元模式Mi和Sy的第j个元模式Nj的加权距离。aij值的大小体现元模式Mi,Nj之间相似程度,当元模式Mi,Nj之间越相似,aij值越接近0;当元模式Mi,Nj之间差别越大,aij值越大;In the formula, a ij =D wei (M i , N j ) is expressed as the weighted distance of the i-th meta-pattern Mi of the pattern sequence S x and the j-th meta-pattern N j of S y . The value of a ij reflects the degree of similarity between the meta-patterns Mi and N j . When the meta-patterns Mi and N j are more similar, the value of a ij is closer to 0 ; The larger the value of a ij , the larger the value of a ij;
(23)与时间序列间的动态时间弯曲距离相似,在距离矩阵DM=(aij)p×t中搜索最佳弯曲路径W=w1,w2,..,wk,并计算累积距离最小的弯曲路径,把这个最小值称为模式序列Sx和Sy的动态时间弯曲距离简记为Ddtw(Sx,Sy),得到如下公式:(23) Similar to the dynamic time-curved distance between time series, search for the best curved path W=w 1 , w 2 , .. , w k in the distance matrix DM=(a ij ) p×t , and calculate the cumulative distance The minimum curved path, which is called the dynamic time curved distance of the mode sequence S x and S y , is abbreviated as D dtw (S x , S y ), and the following formula is obtained:
式中,w就是计算两个元模式之间的距离大小。In the formula, w is to calculate the distance between two meta-patterns.
步骤(3)所述的k-近邻局部异常检测算法中,异常分数值等于该模式的k-近邻距离,具体计算过程如下:In the k-nearest neighbor local anomaly detection algorithm described in step (3), the anomaly score value is equal to the k-nearest neighbor distance of the mode, and the specific calculation process is as follows:
设k是一个正整数,实例x与实例t之间的k-近邻可达距离用如下公式表示为:Let k be a positive integer, the k-nearest neighbor reachable distance between instance x and instance t is expressed as:
rdist(x,t)=max{kdistance(t),d(x,t)}r dist(x, t) = max{k distance(t) , d(x, t)}
实例x的局部可达密度定义为实例x的k个最近邻点的平均可达距离的倒数,局部可达密度公式如下:The local reachability density of an instance x is defined as the reciprocal of the average reachable distance of the k nearest neighbors of the instance x, and the local reachability density formula is as follows:
在数据点集D中,实例x的局部异常系数的LOF(x)可以定义为:In the data point set D, the LOF(x) of the local outlier coefficients of the instance x can be defined as:
其中,根据DTW算法计算每个序列模式的异常分数,rdist(x,t)为每个序列模式的异常分数,实例x表示一个序列模式的异常分数,按照公式计算局部可达密度,lof表示异常因子。Among them, the abnormal score of each sequence pattern is calculated according to the DTW algorithm, r dist(x, t) is the abnormal score of each sequence pattern, the instance x represents the abnormal score of a sequence pattern, and the local reachability density is calculated according to the formula, lof represents abnormal factor.
更进一步的,对于DTW方法存在的奇异点问题,基于空间维度、时间维度的改进方法,通过自适应权重的方式,不引入额外权重系数对空间维度(数值)和时间维度(梯度)加权求和,进而计算积累路径最小的距离。Further, for the singularity problem of the DTW method, the improved method based on the spatial dimension and the time dimension, through the method of adaptive weight, does not introduce additional weight coefficients to weight the sum of the spatial dimension (value) and the time dimension (gradient). , and then calculate the minimum distance of the accumulated path.
更进一步的,所述的DTW方法中的奇异点问题包括基于SDTW算法作为时间序列相似性度量方法,具体过程如下:Further, the singularity problem in the DTW method includes the SDTW algorithm as the time series similarity measurement method, and the specific process is as follows:
定义特征Fs(x(i))为:Define the feature F s (x(i)) as:
式中max(|Δx|)表示时间序列中所有时间点的最大梯度,某点的梯度用相邻点的差分表示比如x(i)-x(i-1);Max(|Δx|)的作用是将时间序列中各点的梯度大小约束在[-1,1]之中,从而以比率的形式将时间维度融与空间维度相结合,得到矩阵中各元素之间距离d(i,j)的表达式如下:In the formula, max(|Δx|) represents the maximum gradient of all time points in the time series, and the gradient of a point is represented by the difference of adjacent points, such as x(i)-x(i-1); Max(|Δx|) The function is to constrain the gradient size of each point in the time series to [-1, 1], so as to combine the time dimension with the space dimension in the form of a ratio, and obtain the distance d(i, j) between the elements in the matrix ) is expressed as follows:
d(i,j)=(Fs(x(i))-Fs(y(j)))2。d(i,j)=( Fs (x(i))- Fs (y(j))) 2 .
有益效果:本发明一套水文数据异常模式检测方案,实现KPRA-PLR算法对原始序列进行分割、针对元模式的相似性度量使用加权距离、针对序列模式的相似性度量使用改进的DTW算法、基于相似性度量的异常模式检测原理使用k-近邻局部异常检测算法,对比常用的异常模式检测方法——符号化SAX方法,不用固定分段以及固定模式类别个数,本发明利用相似性度量的方法检测出的异常模式更为精准,从数据分析的角度为水文数据异常模式检测提供了参考。Beneficial effects: a set of hydrological data abnormal pattern detection scheme of the present invention realizes the KPRA-PLR algorithm to segment the original sequence, uses weighted distance for the similarity measure of the meta pattern, uses the improved DTW algorithm for the similarity measure of the sequence pattern, based on The anomaly pattern detection principle of similarity measurement uses the k-nearest neighbor local anomaly detection algorithm. Compared with the commonly used abnormal pattern detection method—the symbolic SAX method, the method of similarity measurement is used without fixed segmentation and fixed number of pattern categories. The detected abnormal patterns are more accurate, which provides a reference for the detection of abnormal patterns in hydrological data from the perspective of data analysis.
附图说明Description of drawings
图1为本发明中方法的整体结构示意图;Fig. 1 is the overall structure schematic diagram of the method in the present invention;
图2为本发明中KPRA-PLR算法拟合直线示意图;Fig. 2 is KPRA-PLR algorithm fitting straight line schematic diagram in the present invention;
图3为实施例中模式185~195流量变化示意图;FIG. 3 is a schematic diagram of flow changes in modes 185 to 195 in the embodiment;
图4为实施例中模式282~287流量变化示意图;FIG. 4 is a schematic diagram of flow changes in modes 282 to 287 in the embodiment;
图5为实施例中模式290~302流量变化示意图;FIG. 5 is a schematic diagram of flow changes in modes 290 to 302 in the embodiment;
图6为实施例中模式363~370流量变化示意图。FIG. 6 is a schematic diagram of flow changes in modes 363 to 370 in the embodiment.
具体实施方式Detailed ways
为了详细的说明本发明所公开的技术方案,下面结合说明书附图和具体实施例做进一步的阐述。In order to describe the technical solutions disclosed in the present invention in detail, further description is made below with reference to the accompanying drawings and specific embodiments of the description.
一种基于相似性度量的水文数据异常模式检测方法,如图1所示,发明的基于相似度度量的水文数据异常模式检测方法,包括基于关键点的线性分段表示KPRA-PLR算法、时间序列模式相似性度量方法、K-近邻原理三个模块。本实施例中水文数据选用选择龙门水文测站汛期小时流量(单位为m3/s)数据,采用该测站2000/6/1 6:00:00—2015/9/30 17:39:00的实测汛期小时流量数据进行实验。A method for detecting abnormal patterns in hydrological data based on similarity measure, as shown in Figure 1, the invented method for detecting abnormal pattern in hydrological data based on similarity measure includes KPRA-PLR algorithm based on linear segment representation of key points, time series There are three modules of pattern similarity measurement method and K-nearest neighbor principle. In this embodiment, the hydrological data is selected to select the hourly flow (unit is m 3 /s) data of the Longmen hydrological station during the flood season, and the station is used from 2000/6/1 6:00:00 to 2015/9/30 17:39:00 The measured hourly flow data in the flood season were used for experiments.
首先,基于关键点的线性分段表示KPRA-PLR算法,根据关键点定义中的公式,对以上两种水文数据的降维采用一样的阈值ε和δ,分别得到历史水文时间序列(2000年—2015年汛期数据)和实时水文时间序列(2016年—2017年汛期数据)的关键点,据统计:实时水文数据得到关键点189个,历史水文数据得到2844个关键点。关键点按照[t,i]格式表示,其中t代表着时间序列角标,i代表着时间序列对应时刻观测值大小。截取部分实时水文时间得到的关键点放入表格中展示,如下表1所示。First, the KPRA-PLR algorithm is based on the linear segment representation of key points. According to the formula in the definition of key points, the same thresholds ε and δ are used for the dimensionality reduction of the above two hydrological data, and the historical hydrological time series (2000- 2015 flood season data) and real-time hydrological time series (2016-2017 flood season data) key points, according to statistics: real-time hydrological data obtained 189 key points, historical hydrological data obtained 2844 key points. The key points are represented in the format of [t,i], where t represents the time series index, and i represents the size of the observation value at the corresponding time of the time series. The key points obtained by intercepting part of the real-time hydrological time are displayed in a table, as shown in Table 1 below.
表1关键点展示Table 1 shows the key points
通过关键点对原始时间序列进行分割得到一个个子序列,使用PLR算法对子序列进行模式表示,得到一个个元模式。根据定义,相邻的元模式进行组合得到序列模式,采用不一样的组合方式,得到的序列模式会不同,本章采用相邻两两之间组合得到序列模式,用Mi=(ai,Δti)表示,其中ai代表着直线斜率,表示曲线的走势,Δti代表着直线的长短,表示曲线的长短。The original time series is divided by key points to obtain sub-sequences, and the sub-sequences are represented by the PLR algorithm to obtain the sub-sequences. According to the definition, adjacent meta-patterns are combined to obtain sequential patterns. If different combination methods are used, the obtained sequential patterns will be different. This chapter uses the combination of adjacent meta-patterns to obtain sequential patterns, using Mi = (a i , Δt i ), where a i represents the slope of the straight line, which represents the trend of the curve, and Δt i represents the length of the straight line, which represents the length of the curve.
据统计:实时水文序列得到188个元模式,通过相邻的两个元模式组成序列模式有94个;历史水文序列共计2844个元模式,通过相邻的两个元模式组成序列模式有1422个。对实时数据得到的序列模式进行部分展示如下表2所示。According to statistics: 188 meta-patterns are obtained from the real-time hydrological sequence, and 94 are formed by two adjacent meta-patterns; there are 2,844 meta-patterns in the historical hydrological sequence, and 1,422 are formed by two adjacent meta-patterns. . A partial presentation of the sequence patterns obtained from real-time data is shown in Table 2 below.
表2部分序列模式表Table 2 Partial sequence pattern table
所述时间序列模式相似性度量方法中,本章实验主要是挖掘实时水文数据里的异常模式,故根据KPRA-PLR算法对实时水文数据进行分割、模式表示为元模式,相邻两个元模式组成序列模式,共得到序列模式94个,对历史水文数据进行分割、模式表示为元模式,相邻两个元模式组成序列模式,共得到序列模式1422个。其中每一个序列模式与历史水文数据得到的1422个序列模式两两计算距离,展示部分相似性度量结果如下表3所示。In the method for measuring the similarity of time series patterns, the experiments in this chapter are mainly to mine abnormal patterns in real-time hydrological data. Therefore, the real-time hydrological data is segmented according to the KPRA-PLR algorithm, and the pattern is represented as a meta-pattern, which consists of two adjacent meta-patterns. Sequence patterns, a total of 94 sequence patterns were obtained. The historical hydrological data were segmented, and the patterns were represented as meta-patterns. Two adjacent meta-patterns formed a sequence pattern, and a total of 1422 sequence patterns were obtained. The distance between each sequence pattern and the 1422 sequence patterns obtained from the historical hydrological data is calculated in pairs, and the results of some similarity measures are shown in Table 3 below.
表3相似性度量结果Table 3 Similarity measurement results
K-近邻原理中,使用SDTW算法计算实时水文数据得到的序列模式与历史水文数据得到的序列模式之间的相似性,相似性度量结果如上表4和表3所示。然后计算实时水文数据中每个序列模式的异常分数,异常分数计算方式为:与历史数据得到的1422个序列模式之间平均距离的倒数。计算完异常分数后,可采用基于相似度的点异常检测算法检测异常模式,有两种方法可以使用,一种是k-近邻原理;另一种是基于聚类的方法。In the K-nearest neighbor principle, the SDTW algorithm is used to calculate the similarity between the sequence pattern obtained from real-time hydrological data and the sequence pattern obtained from historical hydrological data. The similarity measurement results are shown in Tables 4 and 3 above. Then calculate the anomaly score of each sequence pattern in the real-time hydrological data. The anomaly score is calculated as: the reciprocal of the average distance between the 1422 sequence patterns obtained from the historical data. After calculating the anomaly score, a similarity-based point anomaly detection algorithm can be used to detect anomalous patterns. There are two methods that can be used, one is the k-nearest neighbor principle; the other is a clustering-based method.
本章使用k-近邻方法计算局部异常因子,选取Top-k最大的局部异常因子对应的序列模式即为异常模式。K-近邻原理重要的参数就是近邻数K,通过不断调整近邻数K值,计算对应的局部异常因子,选取Top-k最大的局部异常因子如下表4所示。In this chapter, the k-nearest neighbor method is used to calculate the local anomaly factor, and the sequence pattern corresponding to the local anomaly factor with the largest Top-k is selected as the anomaly pattern. An important parameter of the K-nearest neighbor principle is the number of neighbors K. By continuously adjusting the value of the number of neighbors K, the corresponding local anomaly factor is calculated, and the local anomaly factor with the largest Top-k is selected as shown in Table 4 below.
表4 Top-k局部异常因子表Table 4 Top-k local abnormal factor table
实验结果分析,将局部异常因子从大到小排序,模式185~195最大,模式282~287次之。从图3,图4可以发现当前测站小时流量在2个小时内突然暴涨,序列模式的斜率ai很大,反映出序列模式的增长速度很快,而后又下降。这样的序列模式在2000年—2015年流量数据中是存在的,但出现的次数很少,并且能和该序列模式的走势、时间长度相类似的就更少了。图3,图4表示的两个序列模式短短2个小时流量增加趋势太快,其次模式185~195的模式增减时间偏长,模式282~287的增减时间长度过于太短,因此模式185~195,模式282~287判定为异常模式。According to the analysis of the experimental results, the local abnormal factors are sorted from large to small, the mode 185-195 is the largest, and the mode 282-287 is the second. From Figure 3 and Figure 4, it can be found that the hourly flow of the current station suddenly surges within 2 hours, and the slope a i of the sequence pattern is very large, reflecting the rapid growth rate of the sequence pattern, and then decreases. Such a sequence pattern exists in the flow data from 2000 to 2015, but the number of occurrences is very small, and the trend and time length of the sequence pattern are similar. Figure 3 and Figure 4 show that the traffic flow increases too fast in just 2 hours. Secondly, the increase and decrease time of the modes 185 to 195 is too long, and the increase and decrease time of the modes 282 to 287 is too short. Therefore, the mode 185 to 195, and modes 282 to 287 are judged as abnormal modes.
将局部异常因子从大到小排序,模式290~302第三位,模式363~370第四位。从图5,图6可以发现该流量短时间内缓慢增长,然后长时间的在下降,序列模式的斜率ai是负值,反映出序列模式的下降程度,图5该序列模式整体时间过长,根据关键点的定义,1520并不是关键点,出现了下降-上升走势,前1个小时更是暴增,后面的下降速度很快从2640降到755,在2000年—2015年流量数据中能和模式290~302的增减趋势、时间长度相类似的很少。图6该序列模式只有短短1个小时流量缓慢增加,然后长时间处于下降状态,很少出现这样的走势,加上下降时间过长,就显得“奇怪”,因此模式290~302,模式363~370判定为异常模式。Sort the local anomaly factors from large to small, mode 290-302 is the third, mode 363-370 is the fourth. From Figure 5 and Figure 6, it can be found that the flow increases slowly for a short time, and then decreases for a long time. The slope a i of the sequence mode is a negative value, reflecting the degree of decline of the sequence mode. The overall time of the sequence mode in Figure 5 is too long , according to the definition of the key point, 1520 is not the key point, there is a downward-rising trend, the first hour is a sharp increase, and the subsequent decline rate quickly drops from 2640 to 755. In the traffic data from 2000 to 2015 There are very few that can be similar to the increasing and decreasing trends and time lengths of patterns 290-302. Figure 6 This sequence pattern only has a slow increase in traffic for a short period of 1 hour, and then it is in a state of decline for a long time. Such a trend seldom occurs, and if the decline time is too long, it appears "strange". Therefore, patterns 290 to 302, pattern 363 ~370 is determined as abnormal mode.
Claims (6)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910784182.9A CN110569890A (en) | 2019-08-23 | 2019-08-23 | A method for detecting abnormal patterns in hydrological data based on similarity measure |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910784182.9A CN110569890A (en) | 2019-08-23 | 2019-08-23 | A method for detecting abnormal patterns in hydrological data based on similarity measure |
Publications (1)
Publication Number | Publication Date |
---|---|
CN110569890A true CN110569890A (en) | 2019-12-13 |
Family
ID=68775898
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201910784182.9A Withdrawn CN110569890A (en) | 2019-08-23 | 2019-08-23 | A method for detecting abnormal patterns in hydrological data based on similarity measure |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110569890A (en) |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112183721A (en) * | 2020-09-16 | 2021-01-05 | 河海大学 | A Construction Method of Combined Hydrological Prediction Model Based on Adaptive Differential Evolution |
CN112597539A (en) * | 2020-12-28 | 2021-04-02 | 上海观安信息技术股份有限公司 | Unsupervised learning-based time series anomaly detection method and system |
CN112966017A (en) * | 2021-03-01 | 2021-06-15 | 北京青萌数海科技有限公司 | Abnormal subsequence detection method with indefinite length in time sequence |
CN113190406A (en) * | 2021-04-30 | 2021-07-30 | 上海爱数信息技术股份有限公司 | IT entity group anomaly detection method under cloud native observability |
CN115729981A (en) * | 2022-11-29 | 2023-03-03 | 中国长江电力股份有限公司 | Similar water regime data mining method based on editing distance and application thereof |
CN116429220A (en) * | 2023-06-14 | 2023-07-14 | 济宁安泰矿山设备制造有限公司 | A Method for Abnormal Detection of Water Conservancy Engineering |
CN116703485A (en) * | 2023-08-04 | 2023-09-05 | 山东创亿智慧信息科技发展有限责任公司 | Advertisement accurate marketing method and system based on big data |
CN116957634A (en) * | 2023-09-19 | 2023-10-27 | 贵昌集团有限公司 | Information intelligent acquisition processing method for electronic commerce platform |
-
2019
- 2019-08-23 CN CN201910784182.9A patent/CN110569890A/en not_active Withdrawn
Cited By (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112183721B (en) * | 2020-09-16 | 2022-04-26 | 河海大学 | A Construction Method of Combined Hydrological Prediction Model Based on Adaptive Differential Evolution |
CN112183721A (en) * | 2020-09-16 | 2021-01-05 | 河海大学 | A Construction Method of Combined Hydrological Prediction Model Based on Adaptive Differential Evolution |
CN112597539A (en) * | 2020-12-28 | 2021-04-02 | 上海观安信息技术股份有限公司 | Unsupervised learning-based time series anomaly detection method and system |
CN112966017B (en) * | 2021-03-01 | 2023-11-14 | 北京青萌数海科技有限公司 | Abnormal subsequence detection method for indefinite length in time sequence |
CN112966017A (en) * | 2021-03-01 | 2021-06-15 | 北京青萌数海科技有限公司 | Abnormal subsequence detection method with indefinite length in time sequence |
CN113190406A (en) * | 2021-04-30 | 2021-07-30 | 上海爱数信息技术股份有限公司 | IT entity group anomaly detection method under cloud native observability |
CN115729981A (en) * | 2022-11-29 | 2023-03-03 | 中国长江电力股份有限公司 | Similar water regime data mining method based on editing distance and application thereof |
CN115729981B (en) * | 2022-11-29 | 2024-02-13 | 中国长江电力股份有限公司 | Editing distance-based similar water condition data mining method and application thereof |
CN116429220A (en) * | 2023-06-14 | 2023-07-14 | 济宁安泰矿山设备制造有限公司 | A Method for Abnormal Detection of Water Conservancy Engineering |
CN116429220B (en) * | 2023-06-14 | 2023-09-26 | 济宁安泰矿山设备制造有限公司 | A water conservancy project anomaly detection method |
CN116703485B (en) * | 2023-08-04 | 2023-10-20 | 山东创亿智慧信息科技发展有限责任公司 | Advertisement accurate marketing method and system based on big data |
CN116703485A (en) * | 2023-08-04 | 2023-09-05 | 山东创亿智慧信息科技发展有限责任公司 | Advertisement accurate marketing method and system based on big data |
CN116957634A (en) * | 2023-09-19 | 2023-10-27 | 贵昌集团有限公司 | Information intelligent acquisition processing method for electronic commerce platform |
CN116957634B (en) * | 2023-09-19 | 2023-11-21 | 贵昌集团有限公司 | Information intelligent acquisition processing method for electronic commerce platform |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110569890A (en) | A method for detecting abnormal patterns in hydrological data based on similarity measure | |
CN108038044B (en) | Anomaly detection method for continuous monitored object | |
CN103729550B (en) | Multiple-model integration Flood Forecasting Method based on propagation time cluster analysis | |
CN108959778B (en) | A Remaining Life Prediction Method of Aero-engine Based on Consistency of Degradation Modes | |
CN108667684B (en) | Data flow anomaly detection method based on local vector dot product density | |
CN108876021B (en) | Medium-and-long-term runoff forecasting method and system | |
CN106951680A (en) | A kind of Hydrological Time Series abnormal patterns detection method | |
CN110175416A (en) | Three Gorges Reservoir water temperature prediction method based on principal component analysis and neural network | |
CN113048807B (en) | A method for detecting abnormal back pressure of air-cooled unit | |
CN107293118B (en) | A short-term prediction method for traffic speed dynamic interval | |
CN115031776B (en) | Method for monitoring and analyzing sedimentation of drainage pipe network | |
CN101587154B (en) | Quick mode estimation mode estimating method suitable for complicated node and large scale metric data | |
CN110490228A (en) | A kind of Hydro-Model Parameter Calibration Technology dynamic rating method based on CPP | |
CN110532297A (en) | A kind of symbolism Hydrological Time Series abnormal patterns detection method based on hierarchical clustering | |
CN113807562A (en) | Prediction method of ocean surface temperature | |
CN112116013A (en) | A voltage sag event normalization method based on waveform characteristics | |
CN117251682A (en) | Runoff prediction method and device based on deep learning model | |
CN113640675A (en) | Aviation lithium battery abnormity detection method based on Snippets characteristic extraction | |
CN110046593A (en) | The complex electric energy quality disturbance recognition methods of S-transformation and random forest is improved based on segmentation | |
CN112084461A (en) | A KNN real-time correction method based on historical flood learning | |
CN113779113B (en) | Flood dynamic prediction method and system based on similarity mining of rain and flood spatiotemporal process | |
CN110647717B (en) | A parameter selection method of hydrological model based on correlation dimension | |
CN110032585B (en) | Time sequence double-layer symbolization method and device | |
CN107871140B (en) | A Slope-based Elasticity Similarity Measurement Method | |
CN117972549A (en) | River water level information missing restoration method based on water conservancy big data |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
WW01 | Invention patent application withdrawn after publication |
Application publication date: 20191213 |
|
WW01 | Invention patent application withdrawn after publication |