WO2017185296A1 - 一种基于多支撑点索引的离群检测方法及其系统 - Google Patents

一种基于多支撑点索引的离群检测方法及其系统 Download PDF

Info

Publication number
WO2017185296A1
WO2017185296A1 PCT/CN2016/080505 CN2016080505W WO2017185296A1 WO 2017185296 A1 WO2017185296 A1 WO 2017185296A1 CN 2016080505 W CN2016080505 W CN 2016080505W WO 2017185296 A1 WO2017185296 A1 WO 2017185296A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
index
outlier
objects
data set
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/080505
Other languages
English (en)
French (fr)
Inventor
毛睿
许红龙
陆敏华
廖好
李荣华
王毅
刘刚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen University
Original Assignee
Shenzhen University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen University filed Critical Shenzhen University
Priority to PCT/CN2016/080505 priority Critical patent/WO2017185296A1/zh
Publication of WO2017185296A1 publication Critical patent/WO2017185296A1/zh
Priority to US15/876,218 priority patent/US20180143945A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/243—Classification techniques relating to the number of classes
    • G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection

Definitions

  • the present invention relates to the field of computers, and in particular, to an outlier detection method based on multi-support point indexing and a system thereof.
  • Outliers are data points that are distinctive in a dataset, and their performance is so different from other points that one suspects that the data is not a random bias but is produced by a completely different mechanism. Outliers are also called abnormal points or abnormal objects. Outlier detection is also called anomaly detection, deviation detection or outlier mining. It is to detect outliers in the data set according to a certain algorithm, such as detecting TOP-n outliers, or all qualified deviations. Group point. In other words, outlier detection is the mining of a large number of points in the massive data that are significantly different from the mainstream data.
  • the detection algorithms for outliers mainly include the ORCA algorithm and the iORCA algorithm.
  • the ORCA algorithm uses a method of randomly scrambling the order of data sets in order to obtain an average approximate linear time complexity.
  • the time complexity is still as high as O(n 2 )!
  • pruning efficiency is less than ideal due to the slower rise in the outlier threshold.
  • the required detection time is still too long.
  • the shortcomings of the iORCA algorithm include: First, using only one support point, while saving the indexing time, it leads to the distortion of the data space, reduces the index quality, and does not play the pruning efficiency well; secondly, the iORCA algorithm is as soon as possible. The outlier threshold is raised, and the area farther away from the support point is preferentially detected, but other sparse areas are ignored, but the lifting speed of the outlier threshold has limitations; again, the iORCA algorithm does not provide a support point selection algorithm, and the support point The quality of the algorithm is closely related to the performance of the algorithm. In other words, the support point selection method adopted by the iORCA algorithm is only random selection, and the effect is unstable. Finally, the iORCA algorithm uses only one termination rule to determine whether to stop detecting outliers and fail to fully play. The metric space "triangular inequality" acts to further reduce the number of distance calculations.
  • the object of the present invention is to provide an outlier detection method based on multi-support point indexing and a system thereof, which aims to solve the problem that the single support point used in the prior art causes data space distortion and the outlier detection speed is not high. problem.
  • the invention provides an outlier detection method based on multi-support point index, the method comprising:
  • Selecting a support point step reading in a data set, and selecting a plurality of support points in the data set to form a support point set;
  • An indexing step calculating a distance by using each object in the data set and the selected plurality of support points and using the distance as a coordinate to form a multi-dimensional data space, and using the multi-dimensional data space to establish an index;
  • Outlier detection step dividing the index into data blocks, and detecting the outliers block by block for the data blocks.
  • the step of selecting a support point specifically includes:
  • the midpoint of the number of segments closer to the initial reference point is preferentially added to the set of support points.
  • the step of establishing an index specifically includes:
  • the obtained multiple Hilbert code values are sorted to establish a Hilbert index.
  • the outlier detection step specifically includes:
  • the nearest neighbors are searched in a spiral order from the objects in the current data block, and it is determined that the objects that are unlikely to be outliers are from the detected current
  • the data block is removed until the TOP is updated after all objects in the current data block have been processed. n outliers and outlier thresholds, and enter the next data block;
  • the TOP n is out of the group.
  • the present invention also provides an outlier detection system based on a multi-support point index, the system comprising:
  • An indexing module is configured to calculate a distance by using each object in the data set and the selected plurality of support points and use the distance as a coordinate to form a multi-dimensional data space, and use the multi-dimensional data space to establish an index;
  • An outlier detection module is configured to divide an index into data blocks, and perform block-by-block detection of outliers on the data blocks.
  • the selected support point module is specifically configured to:
  • the midpoint of the number of segments closer to the initial reference point is added to the set of support points.
  • the indexing module is specifically configured to:
  • the obtained multiple Hilbert code values are sorted to establish a Hilbert index.
  • the outlier detection module is specifically configured to:
  • the nearest neighbors are searched in a spiral order from the objects in the current data block, and it is determined that the objects that are unlikely to be outliers are from the detected current
  • the data block is removed until the TOP is updated after all objects in the current data block have been processed. n outliers and outlier thresholds, and enter the next data block;
  • the TOP n is out of the group.
  • the technical solution provided by the present invention is to reduce data space distortion, select multiple support points in the data set, and establish an index, and ensure that the indexing time overhead is extremely small (relative to the total time of outlier detection);
  • the threshold is used to preferentially detect all sparse regions in the dataset, including farther regions and other sparse regions.
  • an approximate dense region support point selection algorithm is proposed, and the quality is relatively good in a very short time. Support points; to further reduce the number of distance calculations, speed up outlier detection, and use multiple pruning rules to exclude non-outliers and non-k nearest neighbors more significantly.
  • the technical solution provided by the invention establishes an index by selecting a plurality of support points and calculating a distance from a global data set, avoids data space distortion caused by a single support point, and preferentially detects all sparse areas in the data set, thereby improving the outlier degree more quickly. Threshold to improve the speed of outlier detection.
  • FIG. 1 is a flowchart of an outlier detection method based on a multi-support point index according to an embodiment of the present invention
  • step S11 shown in FIG. 1 according to an embodiment of the present invention
  • step S12 shown in FIG. 1 according to an embodiment of the present invention
  • step S13 shown in FIG. 1 according to an embodiment of the present invention
  • FIG. 5 is a schematic diagram showing the internal structure of an outlier detection system 10 based on a multi-support point index according to an embodiment of the present invention.
  • Outlier The degree of outliers of an object indicates the degree of its outliers. The average of the distances of its nearest neighbors is used as the outlier, or its distance from the kth nearest neighbor as the outlier.
  • Data block A unit of outlier detection consisting of several objects in a data set, such as 1000 objects commonly used as a data block;
  • Outlier threshold the outliers of the nth outliers of the TOP n outliers
  • Spiral order for example, there is an index sequence of 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, and if starting from 5, its spiral order is 5, 4, 6, 3, 7, 2 , 8 ..., or 5, 6, 4, 7, 3, 8, 2, ... is the meaning of one after the other, and so on;
  • Midpoint of quantity The midpoint calculated from the number, the number of objects larger than the object, and the number of objects smaller than the object, no more than 1, or equal.
  • a specific embodiment of the present invention provides an outlier detection method based on a multi-support point index, and the method mainly includes the following steps:
  • An indexing step calculating a distance by using each object in the data set and the selected plurality of support points, and using the distance as a coordinate to form a multi-dimensional data space, and using the multi-dimensional data space to establish an index;
  • An outlier detection step dividing an index into data blocks, and performing block-by-block detection of outliers on the data blocks.
  • the outlier detection method based on multi-support point index provides index by calculating distances between multiple support points and global data sets, avoiding data space distortion caused by single support points, and prioritizing all sparse areas in the data set. Detection can improve the outlier threshold faster and improve the outlier detection speed.
  • FIG. 1 is a flowchart of an outlier detection method based on a multi-support point index according to an embodiment of the present invention.
  • a support point step is selected: reading a data set, and selecting a plurality of support points in the data set to form a support point set.
  • the selecting support point step S11 specifically includes sub-steps S111-S118, as shown in FIG. 2.
  • FIG. 2 is a detailed flowchart of step S11 shown in FIG. 1 according to an embodiment of the present invention.
  • step S111 after reading the data set, the initial reference point is randomly selected, and the point farthest from the initial reference point is selected as the reference point.
  • step S112 the distances of the respective objects in the data set from the reference point are calculated.
  • step S113 the sorting is performed in descending order of distance.
  • step S114 the data set is divided into a plurality of segments of equal distance.
  • step S115 the plurality of segments are sorted according to the size of the number of objects included.
  • step S116 it is judged whether or not the number of objects included in each segment is equal.
  • step S117 if the number of objects included in each segment is not equal, the midpoints of the number of segments are sequentially added to the set of support points.
  • step S118 if the number of objects included in each segment is equal, the midpoint of the number of segments closer to the initial reference point is added to the set of support points.
  • the data set is divided by equal distance increments on the basis of the equidistant division of the data set from the reference point to the object farthest from the data set.
  • maximum distance d f proposed is divided into n segments, the respectively divided at the reference point distance d f / n, 2d f / n, «, (n-1) d f / n , etc., so that the The data set is divided into n segments that are equidistant but the number of objects is not necessarily equal.
  • the method for determining the dense region is to first count the number of objects included in each segment, and then sort by the number, and the larger one is the candidate region selected by the support point.
  • the temporary reference point is randomly selected as the initial reference point, and the object with the farthest distance from the data set is searched, and the distance between each object in the data set and the reference point is calculated by using the object as a base point.
  • the method of "equal division + number of midpoints" is adopted, and the middle points of the divided segments are added to the support point candidate set.
  • Calculate the number of objects in each segment and then sort the number of objects in descending order. For segments with equal number of objects, the segment closest to the reference point among the segments is obtained, and the midpoint of the number is taken as the first support point.
  • the midpoint of the segment closer to the support point is preferentially selected as the support point.
  • the size ie, the number of segments
  • the number of segments should generally be more than twice the number of support points.
  • the size of the support points should not be too small to ensure the quality of the support points.
  • one data block can be used. In the case of a large number of support points, it should be used more. More data blocks.
  • step S12 an indexing step is established: a multi-dimensional data space is formed by the selected plurality of support points, and an index is established by using the multi-dimensional data space.
  • the indexing step S12 specifically includes sub-steps S121-S125, as shown in FIG.
  • FIG. 3 is a detailed flowchart of step S12 shown in FIG. 1 according to an embodiment of the present invention.
  • step S121 a corresponding number of support points in the set of support points are selected according to the multidimensional data dimension to be converted.
  • step S122 each object in the data set is mapped to a distance value from each support point to form a multidimensional data space.
  • step S123 the multidimensional data space is mapped to an integer coordinate value.
  • step S124 the Hilbert coded value of each pair of integer coordinate values is directly calculated using the Hilbert index mapping algorithm.
  • step S125 the obtained plurality of Hilbert code values are sorted to establish a Hilbert index.
  • the support point selection algorithm After reading the data set, according to the multidimensional data dimension to be converted, using the support point selection algorithm, selecting a corresponding number of support points, and mapping each object of the data set to a distance value from each support point. , forming a multidimensional data space (ie, real coordinate values).
  • the real coordinate values are mapped to integer coordinate values, and then the Hilbert coded value of each pair of integer coordinate values is directly calculated using the Hilbert index mapping algorithm, thus completing the encoding of the metric space objects, and then sorting the encoded values, that is, Create a Hilbert index.
  • step S13 the outlier detection step: dividing the index into data blocks, and performing block-by-block detection of outliers on the data blocks.
  • the outlier detection step S13 specifically includes sub-steps S131-S135, as shown in FIG.
  • FIG. 4 is a detailed flowchart of step S13 shown in FIG. 1 according to an embodiment of the present invention.
  • step S131 the Hilbert index is divided into data blocks, and the data blocks are sorted from sparse to dense according to the encoded values as an outlier detection order.
  • step S132 the set outlier threshold is initialized to 0, and the data set is read on a data block by block in the detection order.
  • step S133 if all the objects in the current data block are impossible to be outliers, the next data block is directly entered.
  • step S134 if there is an object in the current data block that may be an outlier, the nearest neighbor is searched in a spiral order from the object in the current data block, and the object that is impossible to be the outlier is judged. Remove from the current data block being detected until all objects in the current data block have been processed and update TOP n Outliers and outlier thresholds, and enter the next data block.
  • step S1335 when all the data blocks are processed, the TOP n outlier point is output.
  • the pseudo code description algorithm is taken as an example for description, and the input: the nearest neighbor number k, the number of outliers to be detected n, the data set D; output: TOP n out of the group. Then the above step S13 includes:
  • the index data is divided into data blocks (for example, 1000 objects as one data block), and Hilbert code value increments are calculated for the data blocks and sorted in descending order.
  • the outliers are detected block by block in the order of the data blocks. For each data block, when starting the detection, first call the pruning rule three to determine whether it may contain outliers. If not, it will go directly to the next data block; if there is, start from the object in the data block. Search for nearest neighbors in a spiral order. For each object in the detected data block B, first use the pruning rule to determine whether it is an outlier, if not, remove it from the data block B, and enter the detection of the next object; It may be an outlier, then continue to search for its k nearest neighbor.
  • the pruning rule 2 Before calculating the distance, use the pruning rule 2 to determine whether it is possible to be k nearest neighbor. If it is not possible to be its nearest neighbor, then the distance between the two is not calculated, and the detection of the next object is directly performed; if possible, two are calculated. The distance of the person, and try to update its k nearest neighbor, and judge whether its current outlier is less than the threshold c. If it is less, it can no longer be an outlier, and it is removed from the data block B.
  • the three major pruning rules are as follows:
  • Pruning rule 1 Exclude objects that are not out of the group.
  • the distance between the support point p i and its k nearest neighbor and the object x is less than c, so the object x has at least k objects in the range of the radius c, and the outlier must be smaller than c.
  • Pruning rule 2 Exclude objects that are not k nearest neighbors.
  • all objects of data block B have more than k nearest neighbors in the range of distance c.
  • the objects in the data block may have been largely removed. For the remaining objects, try to join TOP one by one. n Outliers and update the outlier threshold c. After all the data blocks have been detected, the TOP n outliers are output.
  • the technical solution provided by the present invention is to reduce data space distortion, select multiple support points in the data set, and establish an index, and ensure that the indexing time overhead is extremely small (relative to the total time of outlier detection);
  • the threshold is used to preferentially detect all sparse regions in the dataset, including farther regions and other sparse regions.
  • an approximate dense region support point selection algorithm is proposed, and the quality is relatively good in a very short time. Support points; to further reduce the number of distance calculations, speed up outlier detection, and use multiple pruning rules to exclude non-outliers and non-k nearest neighbors more significantly.
  • the technical solution provided by the invention establishes an index by selecting a plurality of support points and calculating a distance from a global data set, avoids data space distortion caused by a single support point, and preferentially detects all sparse areas in the data set, thereby improving the outlier degree more quickly. Threshold to improve the speed of outlier detection.
  • the technical solution provided by the present invention can provide a high detection speed while maintaining distance-based versatility, and is compatible with various outlier definitions.
  • the technical solution provided by the invention uses three large pruning rules, and largely eliminates non-outlier points and non-k nearest neighbors, reduces the number of distance calculations, and improves the outlier detection speed.
  • the embodiment of the present invention further provides an outlier detection system 10 based on a multi-support point index, which mainly includes:
  • An indexing module 12 is configured to calculate a distance from each object in the data set and the selected plurality of support points and use the distance as a coordinate to form a multi-dimensional data space, and use the multi-dimensional data space to establish an index;
  • the outlier detection module 13 is configured to divide the index into data blocks, and perform block-by-block detection of the outliers on the data blocks.
  • the invention provides an outlier detection system 10 based on multi-support point indexing, which establishes an index by selecting a plurality of support points and a global data set to calculate a distance, avoiding data space distortion caused by a single support point, and all sparseness in the data set.
  • Area priority detection can increase the outlier threshold faster and improve the outlier detection speed.
  • the outlier detection system 10 based on the multi-support point index mainly includes a selection support point module 11, an index establishment module 12, and an outlier detection module 13.
  • the support point module 11 is selected for reading in the data set, and a plurality of support points are selected in the data set to form a support point set.
  • the selection support point module 11 is specifically configured to: after reading the data set, randomly select an initial reference point, and select a point farthest from the initial reference point as a reference point; The distance between each object in the data set and the reference point; sorted according to the distance from the smallest to the largest; the data set is divided into multiple segments of equal distance; the plurality of segments are sorted according to the size of the number of objects included; Whether the number of objects included in each segment is equal; if the number of objects included in each segment is not equal, the number of points in each segment is sequentially added to the set of support points;
  • the midpoint of the number of segments closer to the initial reference point is added to the set of support points.
  • the indexing module 12 is configured to form a multi-dimensional data space by using the selected plurality of support points, and use the multi-dimensional data space to establish an index.
  • the indexing module 12 is specifically configured to:
  • the obtained multiple Hilbert code values are sorted to establish a Hilbert index.
  • the outlier detection module 13 is configured to divide the index into data blocks, and perform block-by-block detection of the outliers on the data blocks.
  • the outlier detection module 13 is specifically configured to:
  • the nearest neighbors are searched in a spiral order from the objects in the current data block, and it is determined that the objects that are unlikely to be outliers are from the detected current
  • the data block is removed until the TOP is updated after all objects in the current data block have been processed.
  • n Outlier and outlier threshold and enter the next data block; when all data blocks are processed, output TOP n out of the group.
  • the invention provides an outlier detection system 10 based on multi-support point indexing, in order to reduce data space distortion, select multiple support points in the data set, establish an index, and ensure that the indexing time overhead is extremely small (relative to the outlier detection) In terms of total time), in order to improve the outlier threshold faster, all sparse areas in the data set, including far areas and other sparse areas, are preferentially detected. To improve the stability of the algorithm performance, an approximate dense area support point selection algorithm is proposed. In the very short time, the support points with relatively good quality are selected; in order to further reduce the number of distance calculations and speed up the outlier detection, multiple pruning rules are used to exclude non-outliers and non-k nearest neighbors more greatly. Object.
  • the outlier detection system 10 based on multi-support point index provided by the present invention establishes an index by selecting a plurality of support points and calculating a distance from a global data set, thereby avoiding data space distortion caused by a single support point, and all sparse areas in the data set. Priority detection can increase the outlier threshold faster and improve the outlier detection speed.
  • the multi-support point index-based outlier detection system 10 provided by the present invention can provide a higher detection speed while maintaining distance-based versatility, and is compatible with a plurality of outlier definitions.
  • the outlier detection system 10 based on the multi-support point index provided by the present invention uses three large pruning rules to exclude non-outlier points and non-k nearest neighbors in a large amount, reduces the number of distance calculations, and improves the outlier detection speed.
  • each unit included is only divided according to functional logic, but is not limited to the above division, as long as the corresponding function can be implemented; in addition, the specific name of each functional unit is also They are only used to facilitate mutual differentiation and are not intended to limit the scope of the present invention.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Pure & Applied Mathematics (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Mathematical Optimization (AREA)
  • Mathematical Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Operations Research (AREA)
  • Probability & Statistics with Applications (AREA)
  • Algebra (AREA)
  • Databases & Information Systems (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Image Analysis (AREA)

Abstract

一种基于多支撑点索引的离群检测方法，包括：选取支撑点步骤：读入数据集，在所述数据集中选取多个支撑点以形成支撑点集（S11）；建立索引步骤：通过数据集中每个对象与所选取的多个支撑点计算距离并将距离作为坐标，形成多维数据空间，利用所述多维数据空间建立索引（S12）；离群检测步骤：划分索引为数据块，并对所述数据块进行逐块检测离群点（S13）。还提供一种基于多支撑点索引的离群检测系统。本技术方案通过选取多个支撑点与全局数据集计算距离来建立索引，避免单支撑点导致的数据空间扭曲，对数据集中的所有稀疏区域优先检测，能更快地提升离群度阈值，提高离群检测速度。

Description

一种基于多支撑点索引的离群检测方法及其系统 技术领域
本发明涉及计算机领域，尤其涉及一种基于多支撑点索引的离群检测方法及其系统。
背景技术
离群点是数据集中与众不同的数据点，其表现与其它点如此不同，以至于使人怀疑这些数据并非随机的偏差，而是由另外一种完全不同的机制所产生的。离群点也称异常点或者异常对象。离群点检测也称为异常检测、偏差检测或离群点挖掘，它就是按照一定的算法把数据集中的离群点检测出来，例如检测出TOP-n离群点，或者所有符合要求的离群点。换言之，离群点检测就是挖掘海量数据中极少数与主流数据显著不同的点。
目前，针对离群点的检测算法主要包括ORCA算法以及 iORCA算法。
其中，ORCA算法采用随机打乱数据集顺序的方法，以便获得平均近似线性的时间复杂度。然而，在最坏情况下，时间复杂度仍然高达O(n2)！即使在平均情况下，由于离群度阀值上升速度较慢，导致剪枝效率不够理想。在数据集规模较大的情况下，所需检测时间仍然太长。
iORCA算法的缺点包括：首先，仅仅使用一个支撑点，在节省建索引时间的同时，却导致了数据空间的扭曲，降低了索引质量，不能很好地发挥剪枝效率；其次，iORCA算法为尽快提升离群度阈值，优先检测距离支撑点较远的区域，但忽略了其它稀疏区域，但离群度阈值的提升速度具有局限性；再次，iORCA算法并未提供支撑点选取算法，而支撑点的好坏与算法性能密切相关，换言之，iORCA算法采用的支撑点选取方法仅仅是随机选取，效果不稳定；最后，iORCA算法只用一个终止规则来判断是否停止检测离群点，未能充分发挥度量空间“三角不等性”作用来进一步减少距离计算次数。
 技术问题
有鉴于此，本发明的目的在于提供一种基于多支撑点索引的离群检测方法及其系统，旨在解决现有技术中采用的单支撑点导致数据空间扭曲以及离群检测速度不高的问题。
技术解决方案
本发明提出一种基于多支撑点索引的离群检测方法，所述方法包括：
选取支撑点步骤：读入数据集，在所述数据集中选取多个支撑点以形成支撑点集；
建立索引步骤：通过数据集中每个对象与所选取的多个支撑点计算距离并将距离作为坐标，形成多维数据空间，利用所述多维数据空间建立索引；
离群检测步骤：划分索引为数据块，并对所述数据块进行逐块检测离群点。
优选的，所述选取支撑点步骤具体包括：
在读入数据集之后，随机选取初始参考点，并选取与所述初始参考点距离最远的点为基准点；
计算所述数据集中的各个对象与所述基准点的距离；
按照距离的从小到大的顺序排序；
将所述数据集划分为等距离的多段；
将所述多段按照所含对象数量的大小进行排序；
判断各个分段所含对象数量是否相等；
如果各个分段所含对象数量不相等，则将各分段的数量中点按序加入支撑点集；
如果各个分段所含对象数量相等，则优先将离所述初始参考点较近的分段的数量中点加入支撑点集。
优选的，所述建立索引步骤具体包括：
按照拟转换的多维数据维数，选择所述支撑点集中的对应数量的支撑点；
将所述数据集中每个对象映射为与各个支撑点的距离值，以形成多维数据空间；
将多维数据空间映射为整数坐标值；
使用Hilbert索引映射算法直接计算每对整数坐标值的Hilbert编码数值；
将得到的多个Hilbert编码数值进行排序，以建立Hilbert索引。
优选的，所述离群检测步骤具体包括：
划分所述Hilbert索引为数据块，按编码值从稀疏到密集为这些数据块排序以作为离群检测顺序；
设置离群度阈值初始化为0，按检测顺序逐个数据块读取所述数据集；
如果当前数据块中的所有对象都不可能为离群点，则直接进入下一个数据块；
如果当前数据块中有对象可能为离群点，则从所述当前数据块中位的对象开始以螺旋式的顺序搜索最近邻，并将判断不可能是离群点的对象从被检测的当前数据块里移除，直到当前数据块中的所有对象都处理完后更新TOP n离群点和离群度阈值，并进入下一个数据块；
当所有数据块都处理完时，输出TOP n离群点。
另一方面，本发明还提供一种基于多支撑点索引的离群检测系统，所述系统包括：
选取支撑点模块，用于读入数据集，在所述数据集中选取多个支撑点以形成支撑点集；
建立索引模块，用于通过数据集中每个对象与所选取的多个支撑点计算距离并将距离作为坐标，形成多维数据空间，利用所述多维数据空间建立索引；
离群检测模块，用于划分索引为数据块，并对所述数据块进行逐块检测离群点。
优选的，所述选取支撑点模块具体用于：
在读入数据集之后，随机选取初始参考点，并选取与所述初始参考点距离最远的点为基准点；
计算所述数据集中的各个对象与所述基准点的距离；
按照距离的从小到大的顺序排序；
将所述数据集划分为等距离的多段；
将所述多段按照所含对象数量的大小进行排序；
判断各个分段所含对象数量是否相等；
如果各个分段所含对象数量不相等，则将各分段的数量中点按序加入支撑点集；
如果各个分段所含对象数量相等，则将离所述初始参考点较近的分段的数量中点加入支撑点集。
优选的，所述建立索引模块具体用于：
按照拟转换的多维数据维数，选择所述支撑点集中的对应数量的支撑点；
将所述数据集中每个对象映射为与各个支撑点的距离值，以形成多维数据空间；
将多维数据空间映射为整数坐标值；
使用Hilbert索引映射算法直接计算每对整数坐标值的Hilbert编码数值；
将得到的多个Hilbert编码数值进行排序，以建立Hilbert索引。
优选的，所述离群检测模块具体用于：
划分所述Hilbert索引为数据块，按编码值从稀疏到密集为这些数据块排序以作为离群检测顺序；
设置离群度阈值初始化为0，按检测顺序逐个数据块读取所述数据集；
如果当前数据块中的所有对象都不可能为离群点，则直接进入下一个数据块；
如果当前数据块中有对象可能为离群点，则从所述当前数据块中位的对象开始以螺旋式的顺序搜索最近邻，并将判断不可能是离群点的对象从被检测的当前数据块里移除，直到当前数据块中的所有对象都处理完后更新TOP n离群点和离群度阈值，并进入下一个数据块；
当所有数据块都处理完时，输出TOP n离群点。
 有益效果
本发明提供的技术方案为减少数据空间扭曲，在数据集中选取多个支撑点，建立索引，同时确保建立索引时间开销极小（相对于离群检测总时间而言）；为更快提升离群度阈值，优先检测数据集中的所有稀疏区域，包括较远区域和其它稀疏区域；为提高算法性能的稳定性，提出近似密集区域支撑点选取算法，在极短的时间内选取到质量相对较好的支撑点；为进一步减少距离计算次数，加快离群检测速度，使用多个剪枝规则，更大幅度地排除非离群点和非k最近邻对象。本发明提供的技术方案通过选取多个支撑点与全局数据集计算距离来建立索引，避免单支撑点导致的数据空间扭曲，对数据集中的所有稀疏区域优先检测，能更快地提升离群度阈值，提高离群检测速度。
附图说明
图1为本发明一实施方式中基于多支撑点索引的离群检测方法流程图；
图2为本发明一实施方式中图1所示的步骤S11的详细流程图；
图3为本发明一实施方式中图1所示的步骤S12的详细流程图；
图4为本发明一实施方式中图1所示的步骤S13的详细流程图；
图5为本发明一实施方式中基于多支撑点索引的离群检测系统10的内部结构示意图。
 本发明的实施方式
为了使本发明的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本发明进行进一步详细说明。应当理解，此处所描述的具体实施例仅仅用以解释本发明，并不用于限定本发明。
本发明技术方案中出现的名词及其解释如下所示：
离群度：一个对象的离群度表示其离群的程度，常用其与k最近邻的距离的平均值作为离群度，或者其与第k个最近邻的距离值作为离群度；
数据块：离群检测的一个单位，由数据集中的若干个对象组成，例如常用1000个对象作为一个数据块；
离群度阈值：TOP n离群点的第n个离群点的离群度；
螺旋顺序：例如有一个索引序列1、2、3、4、5、6、7、8、9、10，如果以5为起点，它的螺旋顺序就是5、4、6、3、7、2、8……，或者5、6、4、7、3、8、2……，就是一前一后、依此类推的意思；
数量中点：从数量上算的中点，比该对象大的对象数量，与比该对象小的对象数量，相差不超过1，或者相等。
本发明具体实施方式提供了一种基于多支撑点索引的离群检测方法，所述方法主要包括如下步骤：
S11、选取支撑点步骤：读入数据集，在所述数据集中选取多个支撑点以形成支撑点集；
S12、建立索引步骤：通过数据集中每个对象与所选取的多个支撑点计算距离并将距离作为坐标，形成多维数据空间，利用所述多维数据空间建立索引；
S13、离群检测步骤：划分索引为数据块，并对所述数据块进行逐块检测离群点。
本发明提供的一种基于多支撑点索引的离群检测方法通过选取多个支撑点与全局数据集计算距离来建立索引，避免单支撑点导致的数据空间扭曲，对数据集中的所有稀疏区域优先检测，能更快地提升离群度阈值，提高离群检测速度。
以下将对本发明所提供的一种基于多支撑点索引的离群检测方法进行详细说明。
请参阅图1，为本发明一实施方式中基于多支撑点索引的离群检测方法流程图。
在步骤S11中，选取支撑点步骤：读入数据集，在所述数据集中选取多个支撑点以形成支撑点集。
在本实施方式中，所述选取支撑点步骤S11具体包括子步骤S111-S118，如图2所示。
请参阅图2，为本发明一实施方式中图1所示的步骤S11的详细流程图。
在步骤S111中，在读入数据集之后，随机选取初始参考点，并选取与所述初始参考点距离最远的点为基准点。
在步骤S112中，计算所述数据集中的各个对象与所述基准点的距离。
在步骤S113中，按照距离的从小到大的顺序排序。
在步骤S114中，将所述数据集划分为等距离的多段。
在步骤S115中，将所述多段按照所含对象数量的大小进行排序。
在步骤S116中，判断各个分段所含对象数量是否相等。
在步骤S117中，如果各个分段所含对象数量不相等，则将各分段的数量中点按序加入支撑点集。
在步骤S118中，如果各个分段所含对象数量相等，则将离所述初始参考点较近的分段的数量中点加入支撑点集。
在本实施方式中，利用等距划分的情形对数据集在基准点到与其距离最远对象的基础上，按相等的距离增量划分数据集。假设最远距离为df，拟划分为n段，那么可分别在与基准点距离df/n、2df/n、……、(n-1)df/n等处划分，从而将数据集划分为等距离但对象数量不一定相等的n段。其确定密集区域的方法是，先统计各段所含对象数量，再按此数量排序，数量大者为支撑点选取的候选区域。
在本实施方式中，在读入数据集之后，随机选取临时参考点作为初始参考点，搜索数据集中与其距离最远的对象，以该对象为基点，计算数据集中各个对象与参考点的距离，按照从小到大的顺序排序，采用“等距划分+数量中点”的方法，取划分后的各段中位点加入支撑点候选集。计算每个段的对象数量，再对对象数量按从大到小的顺序排序。对于对象数量相等的分段，比较获得这些分段之中与参考点距离最近的分段，取其数量中点作为第一个支撑点。遇到分段所含对象数量相等时，则优先选取离支撑点较近的分段中点为支撑点。
在本实施方式中，应该注意的是，为了使支撑点候选集能选取足够数量的支撑点，其规模（也就是分段的数量）应大于拟选支撑点数量。为确保选取质量，分段数量一般应该是支撑点数量的2倍以上。此外，如果使用数据集的子集来选取支撑点，同样为了确保支撑点质量，其规模也不能过小，一般取一个数据块即可，在支撑点数量较多的情况下，就应该使用更多数据块了。
请继续参阅图1，在步骤S12中，建立索引步骤：通过所选取的多个支撑点形成多维数据空间，利用所述多维数据空间建立索引。
在本实施方式中，所述建立索引步骤S12具体包括子步骤S121-S125，如图3所示。
请参阅图3，为本发明一实施方式中图1所示的步骤S12的详细流程图。
在步骤S121中，按照拟转换的多维数据维数，选择所述支撑点集中的对应数量的支撑点。
在步骤S122中，将所述数据集中每个对象映射为与各个支撑点的距离值，以形成多维数据空间。
在步骤S123中，将多维数据空间映射为整数坐标值。
在步骤S124中，使用Hilbert索引映射算法直接计算每对整数坐标值的Hilbert编码数值。
在步骤S125中，将得到的多个Hilbert编码数值进行排序，以建立Hilbert索引。
在本实施方式中，在读取数据集之后，按照拟转换的多维数据维数，使用支撑点选取算法，选取相应数量的支撑点，将数据集每个对象映射为与各个支撑点的距离值，形成多维数据空间（即实数坐标值）。接下来将实数坐标值映射为整数坐标值，然后使用Hilbert索引映射算法，直接计算每对整数坐标值的Hilbert编码数值，这样就完成了对度量空间对象的编码，再将这些编码值排序，即建立Hilbert索引。
请继续参阅图1，在步骤S13中，离群检测步骤：划分索引为数据块，并对所述数据块进行逐块检测离群点。
在本实施方式中，所述离群检测步骤S13具体包括子步骤S131-S135，如图4所示。
请参阅图4，为本发明一实施方式中图1所示的步骤S13的详细流程图。
在步骤S131中，划分所述Hilbert索引为数据块，按编码值从稀疏到密集为这些数据块排序以作为离群检测顺序。
在步骤S132中，设置离群度阈值初始化为0，按检测顺序逐个数据块读取所述数据集。
在步骤S133中，如果当前数据块中的所有对象都不可能为离群点，则直接进入下一个数据块。
在步骤S134中，如果当前数据块中有对象可能为离群点，则从所述当前数据块中位的对象开始以螺旋式的顺序搜索最近邻，并将判断不可能是离群点的对象从被检测的当前数据块里移除，直到当前数据块中的所有对象都处理完后更新TOP n离群点和离群度阈值，并进入下一个数据块。
在步骤S135中，当所有数据块都处理完时，输出TOP n离群点。
在本实施方式中，以伪代码描述算法为例进行说明，输入：最近邻数量k，拟检测离群点数量n，数据集D；输出：TOP n离群点。则上述步骤S13包括：
在索引建立之后，对索引数据按数据块（例如1000个对象为一个数据块）划分，对数据块计算Hilbert编码值增量并按降序排序。接下来按排好顺序的数据块逐块检测离群点。对于每个数据块，刚开始检测时，先调用剪枝规则三，判断是否可能含有离群点，若无，则直接进入下一个数据块；若有，则从数据块中位的对象开始，以螺旋式的顺序搜索最近邻。对于被检数据块B里的每个对象，先使用剪枝规则一判断有没可能是离群点，若不可能，则将其从数据块B移除，并进入下一对象的检测；若可能是离群点，则继续搜索其k最近邻。在计算距离之前，使用剪枝规则二判断有没可能是k最近邻，若不可能是其k最近邻，则不计算两者的距离，直接进入下一对象的检测；若可能，则计算两者的距离，并尝试更新其k最近邻，同时判断其当前离群度是否小于阈值c，若小于，则再也不可能成为离群点，从数据块B移除。
在本实施方式中，其中三大剪枝规则如下：
（1） 剪枝规则一：排除非离群点的对象。
如果dist(x,pi)+dist(pi,nnk(pi,D))＜c，其中pi∈P;
那么x不可能为离群点。
换言之，支撑点pi及它的k最近邻与对象x的距离都小于c，所以对象x在半径c范围内至少有k个对象，其离群度必定小于c。
（2） 剪枝规则二：排除非k最近邻的对象。
如果||dist(xt,pi)-dist(xj,pi)||＞dist(xt,nnk(xt,D))，其中pi∈P;
那么xj不可能为xt 的k最近邻。
（3） 剪枝规则三：
如果dist(B,pi)+dist(pi,nnk(pi,D))＜c，其中pi∈P;
那么数据块B中的所有对象都不可能为离群点。
也就是说，数据块B的所有对象在距离c范围内都有k个以上的最近邻。
在本实施方式中，实际上，在检测完一个数据块之后，数据块里的对象可能已被大量移除。对于剩下的对象，逐个尝试加入TOP n离群点，并更新离群点阈值c。检测完所有数据块之后，输出TOP n离群点。
本发明提供的技术方案为减少数据空间扭曲，在数据集中选取多个支撑点，建立索引，同时确保建立索引时间开销极小（相对于离群检测总时间而言）；为更快提升离群度阈值，优先检测数据集中的所有稀疏区域，包括较远区域和其它稀疏区域；为提高算法性能的稳定性，提出近似密集区域支撑点选取算法，在极短的时间内选取到质量相对较好的支撑点；为进一步减少距离计算次数，加快离群检测速度，使用多个剪枝规则，更大幅度地排除非离群点和非k最近邻对象。本发明提供的技术方案通过选取多个支撑点与全局数据集计算距离来建立索引，避免单支撑点导致的数据空间扭曲，对数据集中的所有稀疏区域优先检测，能更快地提升离群度阈值，提高离群检测速度。
本发明提供的技术方案在保持基于距离的通用性的同时，能提供较高的检测速度，且兼容多种离群点定义。本发明提供的技术方案使用三大剪枝规则，大量排除非离群点和非k最近邻，减少距离计算次数，提高离群检测速度。
本发明具体实施方式还提供一种基于多支撑点索引的离群检测系统10，主要包括：
选取支撑点模块11，用于读入数据集，在所述数据集中选取多个支撑点以形成支撑点集；
建立索引模块12，用于通过数据集中每个对象与所选取的多个支撑点计算距离并将距离作为坐标，形成多维数据空间，利用所述多维数据空间建立索引；
离群检测模块13，用于划分索引为数据块，并对所述数据块进行逐块检测离群点。
本发明提供的一种基于多支撑点索引的离群检测系统10，通过选取多个支撑点与全局数据集计算距离来建立索引，避免单支撑点导致的数据空间扭曲，对数据集中的所有稀疏区域优先检测，能更快地提升离群度阈值，提高离群检测速度。
请参阅图5，所示为本发明一实施方式中基于多支撑点索引的离群检测系统10的结构示意图。在本实施方式中，基于多支撑点索引的离群检测系统10主要包括选取支撑点模块11、建立索引模块12以及离群检测模块13。
选取支撑点模块11，用于读入数据集，在所述数据集中选取多个支撑点以形成支撑点集。
在本实施方式中，所述选取支撑点模块11具体用于：在读入数据集之后，随机选取初始参考点，并选取与所述初始参考点距离最远的点为基准点；计算所述数据集中的各个对象与所述基准点的距离；按照距离的从小到大的顺序排序；将所述数据集划分为等距离的多段；将所述多段按照所含对象数量的大小进行排序；判断各个分段所含对象数量是否相等；如果各个分段所含对象数量不相等，则将各分段的数量中点按序加入支撑点集；
如果各个分段所含对象数量相等，则将离所述初始参考点较近的分段的数量中点加入支撑点集。
建立索引模块12，用于通过所选取的多个支撑点形成多维数据空间，利用所述多维数据空间建立索引。
在本实施方式中，所述建立索引模块12具体用于：
按照拟转换的多维数据维数，选择所述支撑点集中的对应数量的支撑点；
将所述数据集中每个对象映射为与各个支撑点的距离值，以形成多维数据空间；
将多维数据空间映射为整数坐标值；
使用Hilbert索引映射算法直接计算每对整数坐标值的Hilbert编码数值；
将得到的多个Hilbert编码数值进行排序，以建立Hilbert索引。
离群检测模块13，用于划分索引为数据块，并对所述数据块进行逐块检测离群点。
在本实施方式中，所述离群检测模块13具体用于：
划分所述Hilbert索引为数据块，按编码值从稀疏到密集为这些数据块排序以作为离群检测顺序；
设置离群度阈值初始化为0，按检测顺序逐个数据块读取所述数据集；
如果当前数据块中的所有对象都不可能为离群点，则直接进入下一个数据块；
如果当前数据块中有对象可能为离群点，则从所述当前数据块中位的对象开始以螺旋式的顺序搜索最近邻，并将判断不可能是离群点的对象从被检测的当前数据块里移除，直到当前数据块中的所有对象都处理完后更新TOP n离群点和离群度阈值，并进入下一个数据块；当所有数据块都处理完时，输出TOP n离群点。
本发明提供的一种基于多支撑点索引的离群检测系统10，为减少数据空间扭曲，在数据集中选取多个支撑点，建立索引，同时确保建立索引时间开销极小（相对于离群检测总时间而言）；为更快提升离群度阈值，优先检测数据集中的所有稀疏区域，包括较远区域和其它稀疏区域；为提高算法性能的稳定性，提出近似密集区域支撑点选取算法，在极短的时间内选取到质量相对较好的支撑点；为进一步减少距离计算次数，加快离群检测速度，使用多个剪枝规则，更大幅度地排除非离群点和非k最近邻对象。本发明提供的一种基于多支撑点索引的离群检测系统10通过选取多个支撑点与全局数据集计算距离来建立索引，避免单支撑点导致的数据空间扭曲，对数据集中的所有稀疏区域优先检测，能更快地提升离群度阈值，提高离群检测速度。
本发明提供的一种基于多支撑点索引的离群检测系统10在保持基于距离的通用性的同时，能提供较高的检测速度，且兼容多种离群点定义。本发明提供的一种基于多支撑点索引的离群检测系统10使用三大剪枝规则，大量排除非离群点和非k最近邻，减少距离计算次数，提高离群检测速度。
值得注意的是，上述实施例中，所包括的各个单元只是按照功能逻辑进行划分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，各功能单元的具体名称也只是为了便于相互区分，并不用于限制本发明的保护范围。
另外，本领域普通技术人员可以理解实现上述各实施例方法中的全部或部分步骤是可以通过程序来指令相关的硬件来完成，相应的程序可以存储于一计算机可读取存储介质中，所述的存储介质，如ROM/RAM、磁盘或光盘等。
以上所述仅为本发明的较佳实施例而已，并不用以限制本发明，凡在本发明的精神和原则之内所作的任何修改、等同替换和改进等，均应包含在本发明的保护范围之内。

Claims (8)

  1. 一种基于多支撑点索引的离群检测方法，其特征在于，所述方法包括：
    选取支撑点步骤：读入数据集，在所述数据集中选取多个支撑点以形成支撑点集；
    建立索引步骤：通过数据集中每个对象与所选取的多个支撑点计算距离并将距离作为坐标，形成多维数据空间，利用所述多维数据空间建立索引；
    离群检测步骤：划分索引为数据块，并对所述数据块进行逐块检测离群点。
  2. 如权利要求1所述的基于多支撑点索引的离群检测方法，其特征在于，所述选取支撑点步骤具体包括：
    在读入数据集之后，随机选取初始参考点，并选取与所述初始参考点距离最远的点为基准点；
    计算所述数据集中的各个对象与所述基准点的距离；
    按照距离的从小到大的顺序排序；
    将所述数据集划分为等距离的多段；
    将所述多段按照所含对象数量的大小进行排序；
    判断各个分段所含对象数量是否相等；
    如果各个分段所含对象数量不相等，则将各分段的数量中点按序加入支撑点集；
    如果各个分段所含对象数量相等，则优先将离所述初始参考点较近的分段的数量中点加入支撑点集。
  3. 如权利要求2所述的基于多支撑点索引的离群检测方法，其特征在于，所述建立索引步骤具体包括：
    按照拟转换的多维数据维数，选择所述支撑点集中的对应数量的支撑点；
    将所述数据集中每个对象映射为与各个支撑点的距离值，以形成多维数据空间；
    将多维数据空间映射为整数坐标值；
    使用Hilbert索引映射算法直接计算每对整数坐标值的Hilbert编码数值；
    将得到的多个Hilbert编码数值进行排序，以建立Hilbert索引。
  4. 如权利要求3所述的基于多支撑点索引的离群检测方法，其特征在于，所述离群检测步骤具体包括：
    划分所述Hilbert索引为数据块，按编码值从稀疏到密集为这些数据块排序以作为离群检测顺序；
    设置离群度阈值初始化为0，按检测顺序逐个数据块读取所述数据集；
    如果当前数据块中的所有对象都不可能为离群点，则直接进入下一个数据块；
    如果当前数据块中有对象可能为离群点，则从所述当前数据块中位的对象开始以螺旋式的顺序搜索最近邻，并将判断不可能是离群点的对象从被检测的当前数据块里移除，直到当前数据块中的所有对象都处理完后更新TOP n离群点和离群度阈值，并进入下一个数据块；
    当所有数据块都处理完时，输出TOP n离群点。
  5. 一种基于多支撑点索引的离群检测系统，其特征在于，所述系统包括：
    选取支撑点模块，用于读入数据集，在所述数据集中选取多个支撑点以形成支撑点集；
    建立索引模块，用于通过数据集中每个对象与所选取的多个支撑点计算距离并将距离作为坐标，形成多维数据空间，利用所述多维数据空间建立索引；
    离群检测模块，用于划分索引为数据块，并对所述数据块进行逐块检测离群点。
  6. 如权利要求5所述的基于多支撑点索引的离群检测系统，其特征在于，所述选取支撑点模块具体用于：
    在读入数据集之后，随机选取初始参考点，并选取与所述初始参考点距离最远的点为基准点；
    计算所述数据集中的各个对象与所述基准点的距离；
    按照距离的从小到大的顺序排序；
    将所述数据集划分为等距离的多段；
    将所述多段按照所含对象数量的大小进行排序；
    判断各个分段所含对象数量是否相等；
    如果各个分段所含对象数量不相等，则将各分段的数量中点按序加入支撑点集；
    如果各个分段所含对象数量相等，则将离所述初始参考点较近的分段的数量中点加入支撑点集。
  7. 如权利要求6所述的基于多支撑点索引的离群检测系统，其特征在于，所述建立索引模块具体用于：
    按照拟转换的多维数据维数，选择所述支撑点集中的对应数量的支撑点；
    将所述数据集中每个对象映射为与各个支撑点的距离值，以形成多维数据空间；
    将多维数据空间映射为整数坐标值；
    使用Hilbert索引映射算法直接计算每对整数坐标值的Hilbert编码数值；
    将得到的多个Hilbert编码数值进行排序，以建立Hilbert索引。
  8. 如权利要求7所述的基于多支撑点索引的离群检测系统，其特征在于，所述离群检测模块具体用于：
    划分所述Hilbert索引为数据块，按编码值从稀疏到密集为这些数据块排序以作为离群检测顺序；
    设置离群度阈值初始化为0，按检测顺序逐个数据块读取所述数据集；
    如果当前数据块中的所有对象都不可能为离群点，则直接进入下一个数据块；
    如果当前数据块中有对象可能为离群点，则从所述当前数据块中位的对象开始以螺旋式的顺序搜索最近邻，并将判断不可能是离群点的对象从被检测的当前数据块里移除，直到当前数据块中的所有对象都处理完后更新TOP n离群点和离群度阈值，并进入下一个数据块；
    当所有数据块都处理完时，输出TOP n离群点。
PCT/CN2016/080505 2016-04-28 2016-04-28 一种基于多支撑点索引的离群检测方法及其系统 Ceased WO2017185296A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2016/080505 WO2017185296A1 (zh) 2016-04-28 2016-04-28 一种基于多支撑点索引的离群检测方法及其系统
US15/876,218 US20180143945A1 (en) 2016-04-28 2018-01-22 Method and system for detecting outlier based on multiple pivots index

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2016/080505 WO2017185296A1 (zh) 2016-04-28 2016-04-28 一种基于多支撑点索引的离群检测方法及其系统

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/876,218 Continuation US20180143945A1 (en) 2016-04-28 2018-01-22 Method and system for detecting outlier based on multiple pivots index

Publications (1)

Publication Number Publication Date
WO2017185296A1 true WO2017185296A1 (zh) 2017-11-02

Family

ID=60160552

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/080505 Ceased WO2017185296A1 (zh) 2016-04-28 2016-04-28 一种基于多支撑点索引的离群检测方法及其系统

Country Status (2)

Country Link
US (1) US20180143945A1 (zh)
WO (1) WO2017185296A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110309151A (zh) * 2019-06-18 2019-10-08 精硕科技（北京）股份有限公司 一种索引建立方法、装置及计算机可读存储介质
CN110378843A (zh) * 2018-11-13 2019-10-25 北京京东尚科信息技术有限公司 数据滤波方法和装置
CN112733904A (zh) * 2020-12-30 2021-04-30 佛山科学技术学院 一种水质异常检测方法及电子设备

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111666316B (zh) * 2020-06-19 2023-09-15 南京大学 一种隔离分布核构建方法、异常数据检测方法及装置
CN114153888B (zh) * 2020-09-08 2024-11-22 大连理工大学 一种时间序列数据的异常值检测方法和装置
US11985153B2 (en) 2021-09-22 2024-05-14 The Toronto-Dominion Bank System and method for detecting anomalous activity based on a data distribution
CN114070426B (zh) * 2021-11-15 2024-04-19 上海创远仪器技术股份有限公司 针对mimo信道模拟器的异常校准数据进行剔除处理的方法、装置、处理器及其存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070239636A1 (en) * 2006-03-15 2007-10-11 Microsoft Corporation Transform for outlier detection in extract, transfer, load environment
CN103577589A (zh) * 2013-11-11 2014-02-12 浙江工业大学 一种基于支持张量数据描述的离群数据检测方法
CN104462379A (zh) * 2014-12-10 2015-03-25 深圳大学 一种基于距离的高准确率全局离群点检测算法
CN105975519A (zh) * 2016-04-28 2016-09-28 深圳大学 一种基于多支撑点索引的离群检测方法及其系统

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107480258A (zh) * 2017-08-15 2017-12-15 佛山科学技术学院 一种基于多种支撑点的度量空间离群检测方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070239636A1 (en) * 2006-03-15 2007-10-11 Microsoft Corporation Transform for outlier detection in extract, transfer, load environment
CN103577589A (zh) * 2013-11-11 2014-02-12 浙江工业大学 一种基于支持张量数据描述的离群数据检测方法
CN104462379A (zh) * 2014-12-10 2015-03-25 深圳大学 一种基于距离的高准确率全局离群点检测算法
CN105975519A (zh) * 2016-04-28 2016-09-28 深圳大学 一种基于多支撑点索引的离群检测方法及其系统

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
XUE, ANRONG ET AL.: "New Segment Method of Temporal Data for Outlier Detection", COMPUTER ENGINEERING AND DESIGN, vol. 28, no. 20, 31 October 2007 (2007-10-31), pages 4875 - 4877 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110378843A (zh) * 2018-11-13 2019-10-25 北京京东尚科信息技术有限公司 数据滤波方法和装置
CN110309151A (zh) * 2019-06-18 2019-10-08 精硕科技（北京）股份有限公司 一种索引建立方法、装置及计算机可读存储介质
CN112733904A (zh) * 2020-12-30 2021-04-30 佛山科学技术学院 一种水质异常检测方法及电子设备
CN112733904B (zh) * 2020-12-30 2022-03-25 佛山科学技术学院 一种水质异常检测方法及电子设备

Also Published As

Publication number Publication date
US20180143945A1 (en) 2018-05-24

Similar Documents

Publication Publication Date Title
WO2017185296A1 (zh) 一种基于多支撑点索引的离群检测方法及其系统
WO2019164251A1 (en) Method of performing learning of deep neural network and apparatus thereof
WO2010123185A1 (en) Semiconductor chip and method for generating digital value using process variation
WO2020125251A1 (zh) 基于联邦学习的模型参数训练方法、装置、设备及介质
WO2024186177A1 (ko) 비전-언어 트랜스포머를 사전 훈련하는 방법 및 이를 통해 사전 훈련된 비전-언어 트랜스포머를 포함하는 인공지능 시스템
WO2020242090A1 (en) Apparatus for deep representation learning and method thereof
WO2019112379A1 (ko) 스마트 팩토리 레이아웃 설계 방법 및 시스템
WO2015069087A1 (ko) 인식기 학습 방법 및 장치, 데이터 인식 방법 및 장치
WO2021091022A1 (ko) 머신 러닝 시스템 및 머신 러닝 시스템의 동작 방법
WO2017146338A1 (ko) 인덱스정보를 생성하는 데이터베이스의 아카이빙 방법 및 장치, 인덱스정보를 포함하는 아카이빙된 데이터베이스의 검색 방법 및 장치
WO2021040217A1 (ko) 에너지저장시스템(ess)의 셀 안정화 방법 및 시스템
WO2012165709A1 (ko) 인스턴스 경로 탐색 및 시각화 방법 및 장치
WO2018151356A1 (ko) 다중 스케일 곡률 기반 가시적 벡터 모델 해싱 방법
WO2018199539A1 (ko) 식별키 생성장치 및 식별키 생성방법
WO2022114639A1 (ko) 다차원의 부분집합 연관 분석에 기반한 인공지능 학습 데이터셋 공정성을 보장하는 장치 및 이를 이용한 인공지능 학습 데이터셋 공정성을 보장하는 방법
WO2023128113A1 (ko) 차량에 탑재된 정보 수집 단말과의 연동을 통해 도로 상의 위험물의 실제 크기를 추정할 수 있는 위험물 정보 관리 서버 및 그 동작 방법
WO2018155769A1 (ko) 고효율 저유체 유발 진동 단일채널펌프
WO2019203523A1 (ko) 포인트 클라우드에 관한 데이터를 프로세싱하기 위한 방법 및 장치
WO2017045418A1 (zh) 一种基于内存剩余容量调节照相机帧率的方法及系统
WO2014069767A1 (ko) 염기 서열 정렬 시스템 및 방법
WO2015053441A1 (ko) 식별키 생성 장치 및 방법
WO2015046682A1 (ko) 반도체 공정을 이용한 식별키 생성 장치 및 방법
WO2012165708A1 (ko) 온톨로지 스키마 기반 인스턴스 경로 탐색 방법 및 장치
WO2025095499A1 (ko) 양자 어닐링 기반 신규 소재 탐색 장치 및 방법
WO2024210253A1 (en) Method for threshold secret sharing and reconstruction for multi-compartment

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16899809

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 01.04.2019)

122 Ep: pct application non-entry in european phase

Ref document number: 16899809

Country of ref document: EP

Kind code of ref document: A1