WO2016072839A1

WO2016072839A1 - System and method for identifying potential anomalies in time series data

Info

Publication number: WO2016072839A1
Application number: PCT/MY2015/050137
Authority: WO
Inventors: Wooi Keat Martin CHEW; Chi Hui FOO; Venkaiah Chowdary KONERU; Srinu SOMINENI
Original assignee: Mimos Bhd
Current assignee: Mimos Bhd
Priority date: 2014-11-06
Filing date: 2015-10-30
Publication date: 2016-05-12
Anticipated expiration: 2017-05-06
Also published as: MY175323A

Abstract

A system and method for identifying potential anomalies in time series data, the time series being normalised (100), and the unit and time scales being divided (202, 206) into sections; the time series for each time section being converted (200) into symbolic representations representing a plurality of different values within a time section; symbols representing mean and slope values for each series over time being compared such that those with smaller differences are identified (400) as peers; wherein potential anomalies are identified (500) if a transition between symbols representing maximum or minimum values for one peer is different to the other peers in the group.

Description

SYSTEM AND METHOD FOR IDENTIFYING POTENTIAL ANOMALIES IN TIME SERIES DATA

Field of Invention The invention relates to a system and method for identifying potential anomalies in time series data.

Background

In recent times data sets have become so large that it is becoming increasingly difficult to analyse them, and because of the amount of data it can be difficult to spot anomalies.

One solution to this issue is to use symbolic dynamic filtering in which time-series data are partitioned, symbol sequences are generated to generalise the same, and then compared to standard sequences to see if there are any differences. For example, flight recorder data may be analysed in this manner.

However, in very heterogeneous data, standard sequences for comparison may not be available, and therefore it is problematic to spot anomalies in such data. For example, a large amount of data is generated in tracking food from a source to a destination, so that every step can be checked if required, but if a supplier manipulates an invoice, it is hard to identify such a fraudulent entry in the mass of additional data. Therefore an aim of the invention is to provide a method for identifying anomalies which overcomes the above problem.

Summary of Invention

In an aspect of the invention, there is provided a system for identifying potential anomalies in time series data which can be plotted as units against time, comprising receiving means for receiving time series data from one or more databases, processing means for normalising the time series and dividing unit and time scales into sections; and output means for outputting the results including the potential anomalies identified from the processing means; characterised in that the processing means converts the time series for each time section into symbolic representations representing a plurality of different values within a time section; means are provided to construct a scoring table in which two symbols representing different values for each series over time are compared such that those with smaller differences are identified as peers; and means are provided to compare two symbols representing different values within peer groups, wherein potential anomalies are identified if a transition for one peer is different to the other peers in the group.

In one embodiment the symbols used to identify peers represent mean and slope values. In one embodiment the symbols compared within peer groups represent maximum and minimum values.

In one embodiment the unit scale is divided into sections representing substantially equal probability. In one embodiment the symbolic representations are provided as a quadruplet codon for each series for each time section. In one embodiment the scoring table is constructed using a Markov model.

In one embodiment a transition is the difference between a pair of symbols of a time series, each symbol being in a different time segment. Typically transitions of the same pair type are compared with peers in the same time segments.

In one embodiment the data series are normalised by calculating the mean thereof, moving the mean point to 0 of the x-axis, mapping the upper and lower boundaries of the time series to a predetermined number of standard deviations on the y-axis, and scaling each point in the time series to be between the upper and lower boundaries. Typically three standard deviations are used.

In one embodiment the anomaly detection sensitivity can be changed by adjusting the level of difference required to register an anomaly. In one embodiment the results are stored in a database.

In one embodiment the system includes transmission means for notifying a user if a potential anomaly is detected. Typically the user is sent the results in which the potential anomaly is highlighted, for review and audit of the same. In a further aspect of the invention, there is provided a method of identifying anomalies in time series data which can be plotted as units against time, comprising the steps of: normalising the time series; dividing unit and time scales into sections; characterised by converting the time series for each time section into symbolic representations representing a plurality of different values within a time section; constructing a scoring table in which two symbols representing different values for each series over time are compared such that those with smaller differences are identified as peers; and comparing two symbols representing different values within peer groups, potential anomalies being identified if a transition for one peer is different to the other peers in the group.

Brief Description of Drawings

It will be convenient to further describe the present invention with respect to the accompanying drawings that illustrate possible arrangements of the invention. Other arrangements of the invention are possible, and consequently the particularity of the accompanying drawings is not to be understood as superseding the generality of the preceding description of the invention.

Figure 1 is a schematic overview of an embodiment of the invention.

Figure 2 illustrates normalisation of the time series data (a) method steps; (b) dividing y- axis into equal probability sections; (b) time series after normalisation. Figure 3 illustrates conversion of the time series data to symbolic representations (a) method steps; (b) time series prior to normalisation; (b) assigning quadruplet codons for each series for each time section. Figure 4 illustrates the sectioning of an normal probability distribution (a) area under a normal probability distribution; (b) half the area; (c) first section; (d) second section; (e) third section; (f) final apportionment.

Figure 5 illustrates a method of generating a scoring table based on a Markov model.

Figure 6 illustrates a method of identifying a peer group of candidate time series based on Markov model scoring table.

Figure 7 illustrates a method of identifying local anomalies of candidate time series with respect to the peer group.

Figure 8 illustrates a summary of an example of the method for identifying potential anomalies: (a) Figure 9 illustrates an example output

Detailed Description

With regard to Figure 1, there is illustrated a schematic view of the overall system. The system comprises an electronic processing device 4, for receiving time series data from databases 2, 2', 2", and processing the data to generate output images 6 which are stored in a database 8. If any potential anomalies are detected in the data, the relevant output images are displayed on a monitor or transmitted to a remote devicefor a user 10 to review.

With reference to Figures 2a-c, each time series are first normalised 100, by calculating 102 the mean of the time series, translating 104 the time series by moving the mean point to 0 of the x-axis, mapping 106 the upper boundary of the time series to +3 standard deviations on the y-axis, and similarly mapping 108 the lower boundary of the time series to -3 standard deviations on the y-axis., then scaling 110 each point in the time series to be between the upper and lower boundaries.

Three standard deviations are used as in a normal distribution this covers 99.73% (i.e. nearly all) of data.

With regard to Figures 3a-c, the time series are then converted 200 to symbolic representations (A-F in this example), by dividing 202 the y-axis into equal probability sections based on the normal distribution (as further described with reference to Figure 4), assigning 204 a token to each y-axis division, dividing 206 the x-axis into equal width windows representing time sections, and identifying 208 tokens corresponding to maximum, mean, minimum and slope (maximum-minimum) values of each time series within each time section, such that after all time sections have been tokenized 210, the tokens are concatenated 212 into a final symbolic representation for each time series. As an example with reference to Figure 3c, time series tsl in the first time section window has a minimum value (square) corresponding to unit section having symbol B, a mean value (triangle) corresponding to symbol D, a maximum value (circle) corresponding to symbol F, and a positive slope direction P (P for positive, N for negative) between start and end of the time section. Therefore the final symbolic representation representing max- mean-min-slope is FDBP. These quadruplet codons are determined for each series for each time section. With respect to Figures 4a-f, the division of the y-axis into equal probability sections based on the normal distribution is illustrated in greater details. Figure 4a indicates the area under a normal probability distribution with three standard deviations (i.e. from -3 to +3) which sums to approximately 1.0. Figure 4b illustrates one side which sums to 0.5. Dividing one side into three equal portions makes each area equal to 0.166 (i.e. 0.5 / 3). In Figures 4c -e, first portion (i.e. symbol D) ranges from 0.00 to 0.43 on the x-axis, second portion (i.e. symbol E) ranges from 0.43 to 0.96, and third portion (i.e. symbol F) ranges from 0.97 onwards. Figure 4f illustrates the final division of the y-axis into six portions - each represented by an alphabetical token A-F. With regard to Figure 5, a scoring table based on a Markov model is generated. A table of time series dataset symbolic representations is created 302 from which a time series is selected 304. The token probability for mean and slope columns of the selected time series is calculated 306 and summed 308. This is repeated 310 for every time series in the table until all are completed and the end scoring is determined 312. With respect to Figure 6, a peer group of candidate times series is identified 400 based on the Markov model scoring table. A candidate time series is selected 402, and the token probability sum of the candidate time series is compared 404 with other time series. The nearest of the other time series are tagged 406 as potential peers and a different count is then performed 408 between potential peers and candidate time series for verification.

With regard to Figure 7, local anomalies of candidate time series are identified 500 in the peer group. Initially, two adjacent equal-width time section windows are selected 502 from peer time series. The ascent/descent of max/min columns are compared 504 with other time series. If an anomaly is suspected 506, the selected time section of the relevant time series is flagged 508 as being a potential anomaly. The process then repeats for other time sections. If no other time sections from the candidate time series remain 510, the number of potential local anomalies in candidate time series are counted 512.

With respect to Figures 8a-e, a summary of an example of the method for identifying potential anomalies is illustrated. In Figure 8a five time series (ts l, ts2, ts3, ts4 and ts5) are converted to symbolic representations and tabulated as shown in Figure 8b for building a simple Markov model scoring table. Given a collection of symbolic representations of time series (i.e. tsl, ts2, ts3, ts4, ts5), any given symbolic representation (e.g. ts5) in that collection are scored with respect to the whole collection by the summation of the mean and slope columnar probabilities of that given symbolic representation - mean and slope are used to represent the global similarities of a given time series. For example the columnar probability of ts5 in the slope column of the 1^st window is 0.75 because 'P' appears three times in the combination 'PNPP' .

As indicated in Figure 8c, the calculated scores are used to identify peers for a given time series symbolic representation (e.g. potential peers of ts l), peers being within a selected range.

As a check, a simple difference calculation is performed between the symbolic representation of the given time series and the symbolic representation of a possible peer to ascertain if the difference is within a pre-defined range, where if there is a difference, output as T, otherwise output as Ό', such that the final count of differences indicates the similarity (smaller numbers = greater similarity). For example:

Time Series Symbolic Representation

ts l FFEP FEDN

ts2 FFFN FEDN (minus)

*0* 1 *0*0 0 + 1 + 0 + 0 = 1

Time Series Symbolic Representation

ts l FFEP FEDN

ts3 FEDP FEDN (minus)

* 1 *0 *0*0 1 + 0 + 0 + 0 = 1

Time Series Symbolic Representation

ts l FFEP FEDN

ts5 -> EDDP FFEP (minus) * 1 *0 * 1 * 1 1 + 0 + 1 + 1 = 3

Thus as illustrated in Figures 8d-e, local anomaly can be detected for max-max, max-min, min-min, and min-max by comparing the transitions in selected columns (i.e. ascent or descent) of the candidate time series with its peers. An anomaly is flagged when a time series makes a transition (ascend, descend, or flat) that no other of its peers make. Max and min are used because local anomaly detection focuses on the basic points and not the derived (i.e. mean and slope) points. The sensitivity of the method can be changed by adjusting the level of difference required to register an anomaly.

The method can be applied with varying user-defined numbers of y-axis tokens and x-axis equal windows to surface historical local anomalies. The minimum resolution for the x- axis is 2 data points. The more potential historical local anomalies are highlighted for a given time series, the higher the probability of data integrity issues for the candidate time series.

With regard to Figure 9, an output is produced summarising the results, in which the quadruplet codons 602 of the peer time series 606 used to identify potential local anomalies are illustrated, together with the non-peer time series 608. Any potentially anomalies 604 are highlighted for ease of understanding. It will be appreciated by persons skilled in the art that the present invention may also include further additional modifications made to the system which does not affect the overall functioning of the system.

Claims

1. A system for identifying potential anomalies in time series data which can be plotted as units against time, comprising:

receiving means for receiving time series data from one or more databases (2,

2', 2");

processing means for normalising the time series and dividing unit and time scales into sections; and

output means for outputting the results (6) including the potential anomalies identified from the processing means;

characterised in that the processing means converts the time series for each time section into symbolic representations representing a plurality of different values within a time section;

means are provided to construct a scoring table in which two symbols representing different values for each series over time are compared such that those with smaller differences are identified as peers; and

means are provided to compare two symbols representing different values within peer groups, wherein potential anomalies are identified if a transition for one peer is different to the other peers in the group.

2. A system according to claim 1 wherein the symbols used to identify peers represent mean and slope values. A system according to claim 1 wherein the symbols compared within peer groups represent maximum and minimum values.

A system according to claim 1 wherein the unit scale is divided into sections representing substantially equal probability.

A system according to claim 1 wherein the scoring table is constructed using a Markov model.

A system according to claim 1 wherein a transition is the difference between a pair of symbols of a time series, each symbol being in a different time segment.

A system according to claim 6 wherein transitions of the same pair type are compared with peers in the same time segments.

A system according to claim 1 wherein the data series are normalised by calculating (102) the mean thereof, moving (104) the mean point to 0 of the x-axis, mapping (106, 108) the upper and lower boundaries of the time series to three standard deviations on the y-axis, and scaling (110) each point in the time series to be between the upper and lower boundaries.

A system according to claim 1 wherein the anomaly detection sensitivity can be changed by adjusting the level of difference required to register an anomaly. A method of identifying anomalies in time series data which can be plotted as units against time, comprising the steps of:

normalising the time series;

dividing (202, 206) unit and time scales into sections;

characterised by converting (200) the time series for each time section into symbolic representations representing a plurality of different values within a time section;

constructing a scoring table in which two symbols representing different values for each series over time are compared such that those with smaller differences are identified as peers; and

comparing two symbols representing different values within peer groups, potential anomalies being identified if a transition for one peer is different to the other peers in the group.