JP2018190281A - Data processing apparatus, data processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a data processing apparatus or the like capable of detecting fault by performing analysis for multidimensional and multi-scale data.SOLUTION: A statistics data processing apparatus includes: a statistics data comparison part for comparing comparison object data with reference data which are statistics data based on data included in each combination of a point of view and a scale with respect to at least one point of view and a plurality of scales for each point of view; and a fault range determination part for determining a range deemed to be abnormal among a plurality of ranges of the point of view on the basis of comparison result by the statistics data comparison part.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a data processing device, a data processing method, and a program.

In various fields such as traffic and healthcare, time-series data of a predetermined target is detected. In some cases, it is important to immediately detect an abnormality to be detected from such time-series data.
Further, there are many cases where stream data in which the number of samples of time series data is enormous and data is always generated one after another is processed.

The method for detecting an abnormality is roughly classified into a method for detecting an abnormality using prior knowledge and a method for detecting an outlier as an abnormality.
In the method of detecting an abnormality using prior knowledge, when time-series data satisfying a specific condition occurs, the abnormality is detected as the occurrence of the abnormality, but an unknown event cannot be dealt with. In addition, in the method of detecting an abnormality using prior knowledge, a precondition is often assumed, and therefore, when the precondition changes, it is necessary to correct a specific condition each time. This method also requires prior knowledge to be learned. In order to compensate for this point, it may be possible to use machine learning, but a sufficiently high-speed process has not been established.

Under these circumstances, it is important to establish and enhance a method for detecting outliers for stream data.
For example, techniques for detecting outliers have existed for some time, but there have been cases where sufficiently high-speed processing has not been performed, particularly when stream data is targeted. In situations where it is difficult to predict when an anomaly will occur, it is necessary to constantly monitor whether an anomaly has occurred, and processing that is sufficiently faster than the rate at which the data constituting the time-series data occurs one after another It was necessary to realize.

As an example, Non-Patent Document 1 proposes a technique called t-digest. The t-digest has a data structure for estimating the distribution of a set of values. For example, the distribution can be estimated for stream data.
However, there are cases where it is difficult to realize sufficiently high-speed processing with only t-digest.

TED DUNING AND OTMAR ERTL, "COMPUTING EXTREME ACCURATE QUANTILES USING t-DIESTS", [online], [April 12, 2017 search], Internet <URL: https://github.com/tdunning/t-gest blob / master / docs / t-digest-paper / histo.pdf>

  Conventionally, it has been difficult to perform multidimensional and multiscale data analysis. In t-digest, multidimensional and multiscale were not assumed.

  The present invention has been made in consideration of such circumstances, and provides a data processing apparatus, a data processing method, and a program that can detect an abnormality by performing multidimensional and multiscale data analysis. This is the issue.

  As one configuration example, for at least one viewpoint and a plurality of scales for each viewpoint, reference data that is statistical data based on data included for each combination of the viewpoint and the scale is compared with comparison target data. A statistic comprising: a statistic data comparison unit; and an abnormal range determination unit that determines a range considered to be abnormal among the plurality of ranges of the viewpoint based on a result of comparison by the statistic data comparison unit A data processing device.

  According to the present invention, it is possible to detect anomalies by performing multidimensional and multiscale data analysis.

1 is a block diagram showing a schematic configuration of a data processing system according to an embodiment (first embodiment) of the present invention. It is a block diagram which shows the schematic structure of the statistic data processing apparatus which concerns on one Embodiment of this invention. It is a figure which shows typically the data structure of an example of the statistics data group which concerns on one Embodiment of this invention. It is a figure which shows the outline | summary of the abnormality detection process based on statistic data based on one Embodiment of this invention. It is a figure which shows an example of the statistics data used as the reference | standard which concerns on one Embodiment of this invention. It is a figure which shows another example of the statistics data used as the reference | standard which concerns on one Embodiment of this invention. It is a figure which shows an example of the statistics data used as the analysis object which concerns on one Embodiment of this invention. It is a figure which shows an example of the comparison result of the statistics data for every time range which concerns on one Embodiment of this invention. It is a figure for demonstrating an example of the determination process of the abnormal period which concerns on one Embodiment of this invention. It is a figure which shows an example of the procedure of the process for detecting the presence or absence of abnormality performed in the statistic data processing apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows the schematic structure of the data processing system which concerns on one Embodiment (2nd Embodiment) of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
[Data processing system]
FIG. 1 is a block diagram showing a schematic configuration of a data processing system 1 according to an embodiment (first embodiment) of the present invention.
The data processing system 1 includes n (n is an integer of 2 or more) terminal devices 11-1 to 11-n, a statistic data processing device 12, a database 13, and a network 21.
As the network 21, a wired or wireless arbitrary network may be used. For example, the Internet or a Wi-Fi (registered trademark) network may be used.
Although the database 13 is provided separately from the statistic data processing device 12 in this embodiment, as another example, the database 13 may be integrated with the statistic data processing device 12.

  The statistic data processing device 12 acquires and analyzes predetermined target data regarding the terminal devices 11-1 to 11-n. In the present embodiment, it is assumed that the data is stream data in which the number of samples is enormous (big data) and data is always generated one after another with respect to predetermined target time-series data.

Arbitrary data may be used as the predetermined target data. For example, data on IoT (Internet on Things) may be used, or other data may be used.
As a specific example, data in an arbitrary system may be used as the predetermined target data. For example, data related to transportation systems such as vehicles, data related to healthcare systems such as people, and factory systems that produce products Data relating to financial systems such as securities, data relating to wired or wireless communication systems, and the like may be used. In addition, in an arbitrary system, various data may be used as predetermined target data. For example, data related to temperature, data related to humidity, data related to speed, data related to acceleration, data related to images, data related to oxygen, etc. Data relating to substance concentration, data relating to quality, data relating to stock prices, data relating to communication signals, data relating to positions of terminal devices (terminal devices 11-1 to 11-n in the present embodiment), data relating to areas where terminal devices exist. Etc. may be used.

Various methods may be used as a method for detecting predetermined target data.
As an example, in each of the terminal devices 11-1 to 11-n, a configuration for detecting predetermined target data related to each of the terminal devices 11-1 to 11-n may be used. In this configuration, each of the terminal devices 11-1 to 11-n includes a detection unit that detects data of a predetermined target, and the data detected by the detection unit is subjected to statistical data processing via the network 21. Transmit to device 12. The detection unit may be, for example, a sensor or an imaging device (camera). Here, the imaging device (camera) may also be regarded as an example of a sensor.
Each of the terminal devices 11-1 to 11-n may be an IoT terminal device or may be another terminal device.

  As another example, the data processing system 1 may include a detection device (not shown) different from the terminal devices 11-1 to 11-n. In this configuration, the detection device includes a detection unit that detects predetermined target data regarding each of the terminal devices 11-1 to 11-n, and the data detected by the detection unit is transmitted to the network 21. To the statistic data processing device 12.

The detection device may include, for example, an imaging device (camera) that captures an image of a predetermined region, and the number of terminal devices 11-1 to 11-n existing in the predetermined region based on the image. Or the like may be detected as predetermined target data.
The detection apparatus may include, for example, a communication unit that communicates with the terminal devices 11-1 to 11-n existing in a predetermined area wirelessly or by wire. Based on the result of the communication, the detection apparatus Data of the number of existing terminal apparatuses 11-1 to 11-n may be detected as predetermined target data.
The detection device may include, for example, a signal acquisition unit that acquires signals transmitted from the terminal devices 11-1 to 11-n. Based on the state of the signal, the frequency of occurrence of the signal or the degree of delay, etc. May be detected as predetermined target data.

In the present embodiment, each of the terminal devices 11-1 to 11-n can communicate with other devices (for example, the statistics data processing device 12) via the network 21. Each of the terminal devices 11-1 to 11-n is connected for communication with the network 21 by wire or wireless.
As another example, each of the terminal devices 11-1 to 11-n may not have a communication function.

Moreover, as each terminal device 11-1 to 11-n, for example, a terminal device having the same configuration may be used, or a terminal device having a different configuration may be included.
Each of the terminal devices 11-1 to 11-n may be added to or attached to an object, or may be carried or attached by a person. As the said thing, arbitrary things may be sufficient, for example, vehicles, such as a motor vehicle, or an electric appliance.

  In addition, a detection device other than the terminal devices 11-1 to 11-n may be provided in the statistic data processing device 12, for example. In this configuration, the statistic data processing device 12 acquires and analyzes the data detected by the detection device.

[Statistics data processor]
FIG. 2 is a block diagram showing a schematic configuration of the statistic data processing apparatus 12 according to an embodiment of the present invention.
The statistics data processing device 12 includes an input unit 111, an output unit 112, a storage unit 113, a communication unit 114, and a control unit 115.
The control unit 115 includes a data acquisition unit 131, a statistic data processing unit 132, and a data output control unit 133.
The statistic data processing unit 132 includes a viewpoint setting unit 151, a scale setting unit 152, a statistic data group generation unit 153, a statistic data comparison unit 154, and an abnormal range determination unit 155.

The input unit 111 inputs information from the outside. The input unit 111 includes, for example, an operation unit that receives an operation performed by a user (person), and inputs information according to the operation received by the operation unit. For example, the input unit 111 is connected to an external device (for example, a recording medium) and inputs information output from the external device.
The output unit 112 outputs information. The output unit 112 has a screen, for example, and displays (outputs) information on the screen. The output unit 112 is connected to, for example, an external device (for example, a recording medium) and outputs information to the external device.
The storage unit 113 stores information. In the present embodiment, the storage unit 113 and the database 13 may be arbitrarily used.
The communication unit 114 communicates information. In the present embodiment, the communication unit 114 communicates information with other devices (for example, the terminal devices 11-1 to 11-n or another detection device) via the network 21.

The control unit 115 performs various controls in the statistic data processing apparatus 12.
In the present embodiment, the storage unit 113 stores information on a predetermined control program and its parameters. The control unit 115 is configured by using a CPU (Central Processing Unit). In the control unit 115, the CPU performs various types of control by executing the control program stored in the storage unit 113 using the parameters stored in the storage unit 113.

  In addition, about the statistics data processing apparatus 12, the structure provided with each process part 111-115 shown in FIG. 2 is an example, and another structure may be used. For example, the classification of the functions of the processing units 111 to 115 is for convenience of explanation, and is not necessarily limited to the configuration shown in FIG.

The function of the control unit 115 will be described.
The data acquisition unit 131 acquires predetermined target data as data to be analyzed.
As an example, the data acquisition unit 131 acquires data received from another device (for example, the terminal devices 11-1 to 11-n or another detection device) by the communication unit 114 as data to be analyzed. Also good.
As another example, the data acquisition unit 131 sequentially stores data acquired in the past in the database 13 and acquires the data from the database 13 as data to be analyzed when processing the data. May be.
As another example, a configuration in which predetermined target data is stored in the database 13 without passing through the statistics data processing device 12 may be used. In this case, the data acquisition unit 131 When performing processing, the data may be acquired from the database 13 as data to be analyzed.

The statistic data processing unit 132 performs a statistical process on the data acquired by the data acquisition unit 131, and acquires the resulting data (also referred to as “statistical data” in the present embodiment).
Note that the statistic data processing unit 132 may process, for example, data that is successively acquired in real time by the data acquisition unit 131 (a collection of newly increasing data). There may be a case where processing is performed on past data (a group of data that does not newly increase) acquired by the unit 131.
Further, the statistic data processing unit 132 performs detection (determination) regarding the presence or absence of abnormality based on the statistic data.

The data output control unit 133 controls the output unit 112 to output data to be output. As this output, for example, display output of characters, figures, graphs or the like is used.
Arbitrary data may be used as the data to be output. For example, data acquired by the data acquisition unit 131 or data obtained as a result of processing by the statistic data processing unit 132 is used. May be. The data obtained as a result of the processing by the statistic data processing unit 132 includes, for example, data obtained as a result of processing by the statistic data group generation unit 153 and processing by the statistic data comparison unit 154. One or more of the result data or the result data obtained by the processing by the abnormal range determination unit 155 may be used.

The statistic data processing unit 132 will be described.
The viewpoint setting unit 151 sets viewpoints (items) for the analysis target data. The viewpoint setting unit 151 may set two or more viewpoints for the data to be analyzed.
Here, any viewpoint may be used as the viewpoint, and for example, time (time), area (region), device type, or the like may be used.
In addition, the viewpoint setting unit 151 may set a predetermined viewpoint, or may set a viewpoint instructed by a user or the like. When the viewpoint is determined in advance, for example, information specifying the viewpoint is stored in the storage unit 113.

The scale setting unit 152 sets a scale (granularity) for each viewpoint. The scale setting unit 152 may set two or more scales for each viewpoint.
Here, as the scale, a scale having an arbitrary size may be used.
For example, a scale of 1 second, a scale of 1 minute, a scale of 1 hour, a scale of 1 day, or any other scale may be used as the time scale.
For example, an area scale, a city scale, a prefecture scale, a national scale, or any other scale may be used as the area scale.
For example, as the device type scale, a model scale, a manufacturer scale, an OS (Operating System) scale, or another arbitrary type (attribute) scale may be used.
For example, the scale setting unit 152 may set a predetermined scale for each viewpoint, or may set a scale instructed by a user or the like. When the scale is determined in advance, for example, information for specifying the scale is stored in the storage unit 113.

The statistics data group generation unit 153 performs statistical processing on the data acquired by the data acquisition unit 131 based on the viewpoint set by the viewpoint setting unit 151 and the scale set by the scale setting unit 152. Thus, a set of a plurality of statistical data (also referred to as “statistical data group” in the present embodiment) is generated.
In the present embodiment, the statistic data group generation unit 153 converts the data obtained as a result of multidimensional and multiscale data analysis into a statistic data group (in this embodiment, “multidimensional multiscale statistic data group”). It is also generated as
Here, in this embodiment, the dimension represents a viewpoint, and the multidimension represents a plurality of viewpoints.
In the present embodiment, the multi-scale indicates that there are a plurality of scales.

The statistical data comparison unit 154 compares the standard statistical data with the statistical data to be analyzed. In the present embodiment, the statistic data comparison unit 154 compares the statistic data serving as a reference with the statistic data to be analyzed for a plurality of different scales.
Here, the reference statistical data may be acquired by the data acquisition unit 131, for example, or statistical data included in the statistical data group generated by the statistical data group generation unit 153 is used. May be.
The statistical data to be analyzed may be acquired by the data acquisition unit 131, for example, or based on the data acquired by the data acquisition unit 131, the statistical data processing unit 132 (for example, statistical data) The statistic data generated by the comparison unit 154) may be used, or the statistic data included in the statistic data group generated by the statistic data group generation unit 153 may be used.

The abnormal range determination unit 155 is a range in which an abnormality has occurred in the statistical data to be analyzed based on the result of the comparison by the statistical data comparison unit 154 (in this embodiment, a range of time (time), (Also referred to as “abnormal period”). This determination may be, for example, an estimation determination.
In the present embodiment, the presence of an abnormality is detected when the abnormal range is determined by the abnormal range determination unit 155 for the statistical data to be analyzed.

[Example of data structure of statistical data group]
FIG. 3 is a diagram schematically showing an example of the data structure of the statistic data group 201 according to the embodiment of the present invention.
In the example of FIG. 3, the statistical data group 201 is a multidimensional multiscale statistical data group that is a result of processing time-series data by the statistical data processing unit 132. The statistic data group 201 has a structure of distribution data (in this embodiment, statistic data) extended to multi-dimensional and multi-scale for time series data.
In the example of FIG. 3, time, region, and device type are used as a plurality of viewpoints. Further, for each viewpoint, a plurality of scales (multiscale) are used.

In the example of FIG. 3, a time scale s0, a time scale s1, and a time scale s2 are used as a plurality of time scales. As an example, the time scale s0 is “10 seconds”, the time scale s1 is “20 seconds”, and the time scale s2 is “40 seconds”.
Further, as a plurality of scales related to the region, a region scale A1, a region scale A2, and a region scale A3 are used. As an example, the area scale A1 is “Tokyo”, the area scale A2 is “Kanto”, and the area scale A3 is “Japan (nationwide)”.
Further, a device type scale D1 and a device type scale D2 are used as a plurality of scales related to the device type. As an example, the device type scale D1 is “specific model A1”, and the device type scale D2 is “specific manufacturer B1”.

In the example of FIG. 3, the direction (arrow) of the axis representing the time is shown. As a time range (time division), a time range t0, a time range t1, a time range t2, and a time range t3 are used. The time range t0 is “0 seconds to less than 10 seconds”, the time range t1 is “10 seconds to less than 20 seconds”, the time range t2 is “20 seconds to less than 30 seconds”, and the time range t3 is “ 30 seconds or more and less than 40 seconds ".
An arbitrary timing may be used as the initial time value (0 seconds in the present embodiment).

Here, when one time scale, one region scale, and one device type scale are specified, and one time range corresponding to the time scale is specified, a unit statistic Data (also referred to as “unit statistic data” in the present embodiment) is specified.
The unit statistic data specified by one time scale, one area scale, one device type scale, and one time range corresponding to the time scale is the maximum time in the time range. The statistical data processing unit 132 performs a statistical calculation on the data belonging to the time scale and the time that goes back to the past by the time scale and belonging to the area scale and the device type scale. It corresponds to the obtained statistical data.

  In the example of FIG. 3, six unit statistical data 211 to 213 and 221 to 223 corresponding to the time scale s0 and the time range t3, and six unit statistical data 311 corresponding to the time scale s1 and the time range t3. .., 313, 321-323, and the six unit statistical data 411-413, 421-423 corresponding to the time scale s2 and the time range t3 are provided with symbols, and the other unit statistical data are encoded. Is omitted.

  As an example, the unit statistics data 211 corresponds to the time scale S0 (10 seconds), the area scale A1 (Tokyo), the device type scale D1 (specific model A1), and the time range t3 (30 seconds to less than 40 seconds). . The unit statistical data 211 is statistical data obtained based on data belonging to a time belonging to 30 seconds or more and less than 40 seconds, a region belonging to Tokyo, and a device type belonging to a specific model A1. It is. That is, the data is generated in the terminal devices 11-1 to 11-n that exist in Tokyo at a time of 30 seconds or more and less than 40 seconds, and the terminal device 11-1 whose device type is the specific model A1. Means that the data occurred in -11-n. The unit statistical data is statistical data obtained using such a set of data.

Here, as the unit statistic data, data of an arbitrary statistic may be used. For example, data of an arbitrary value related to order statistics may be used, or data of an average value may be used. Also good.
As a value related to order statistics, for example, a median value may be used. In general, when a plurality of pieces of data to be processed are the same, it is considered that the processing time for the median value is shorter than the processing for acquiring (calculating) the average value.
In addition, as a value related to order statistics, for example, a value of a cumulative distribution function (CDF) may be used, or a value of a probability distribution function (PDF: Probability Density Function) may be used. .

In the example of FIG. 3, the unit of each square (cuboid or cube) is unit statistical data (unit statistical data 211-213, 221-223, 311-313, 321-323, 411-413, 421-423, etc. ).
Note that the statistic data group generation unit 153 may acquire unit statistic data using an arbitrary technique. For example, the t-digest technique (for example, see Non-Patent Document 1 or the like) that is an existing technique. .) May be used to calculate and obtain unit statistics data. In the present embodiment, the statistical data group includes a plurality of unit statistical data.

In addition, as a plurality of scales used for one viewpoint, for example, a plurality of scales that are inclusive relation to each other may be used, or a plurality of scales that are not inclusive relation to each other are used. Alternatively, a plurality of scales that are inclusive in only part of them may be used.
As a plurality of scales that are inclusive of each other, for example, the scale of the region is “Tokyo”, “Kanto”, and “Japan”.
Examples of the plurality of scales that are not inclusive of each other include, for example, “Tokyo”, “Chiba”, and “Ibaraki”, taking the scale of the region as an example.
As a plurality of scales that are inclusive in only a part, taking the scale of the area as an example, there are, for example, “Tokyo”, “Kanto” (including Tokyo), and “Osaka”.

[Abnormality detection processing performed in the statistics data processing unit]
FIG. 4 is a diagram showing an outline of the abnormality detection process based on the statistic data according to the embodiment of the present invention.
In the present embodiment, as data to be analyzed (comparison target) and reference data, data regarding latency is used for the plurality of terminal devices 11-1 to 11-n. As the latency, a time (delay time) until a response comes to the request transmitted from each of the terminal devices 11-1 to 11-n is used.
Here, it is assumed that each of the terminal devices 11-1 to 11-n is owned by a different person. For convenience of explanation, it is assumed that there is a commuting rush at a fixed time on weekdays and no commuting rush on holidays. In general, there are morning commuting rush and evening commuting rush on weekdays, but in this example, only morning commuting rush is shown, and evening commuting rush is omitted.

In the example of FIG. 4, an analysis target statistical data group 522 is generated based on the analysis target data 521. In the present embodiment, the statistical data group 522 to be analyzed is generated based on data for one day that is a target for detecting the presence or absence of abnormality.
In the example of FIG. 4, the statistical data groups 511-1 to 511-30 generated based on the data for one day are used as the reference data 511 for each of the past 30 days. It has been. Here, it is assumed that no abnormality has occurred on the 30th day from which the reference data 511 is acquired, and normal reference data is acquired.

The statistic data comparison unit 154 performs processing (comparison processing 523) for comparing the reference data 511 and the statistical data group 522 to be analyzed.
Based on the result of such comparison, the abnormal range determination unit 155 performs processing (in this example, an abnormality detection process 524 for detecting an abnormal distribution) for determining an abnormal range for the statistic data. Thereby, the abnormal range determination unit 155 determines the abnormal range 525 for the statistical data group to be analyzed.
In the present embodiment, when the abnormal range 525 exists, it is detected (determined) that there is an abnormality, and when there is no abnormal range 525, it is detected (determined) that there is no abnormality.
In this example, a case where an abnormal period that is a time (time) range is detected (determined) as the abnormal range 525 will be described.

An example of statistical data is shown with reference to FIGS. Note that the statistical data shown in FIGS. 5 to 7 are artificially created for convenience of explanation, and represent the time transition of the data distribution.
FIG. 5 is a diagram showing an example of statistical data serving as a reference according to an embodiment of the present invention.
In the example of FIG. 5, statistical data on weekdays is shown.
In the graph shown in FIG. 5, the horizontal axis represents time for one day of 24 hours (from time 0 am to time 12 pm), and the vertical axis represents the latency [ms of each terminal device 11-1 to 11-n]. ].
In this graph, the latency detected about each terminal device 11-1 to 11-n is shown.
In addition, this graph shows the percentile characteristics in order statistics for the latencies of the plurality of terminal apparatuses 11-1 to 11-n. Specifically, a 10th percentile characteristic 611, a 25th percentile characteristic 612, a 50th percentile characteristic 613, a 75th percentile characteristic 614, and a 90th percentile characteristic 615 are shown.
In this example, it is assumed that there is a commuting rush at around 8 am on weekdays, and that the latency is the largest in the day.

FIG. 6 is a diagram illustrating another example of statistical data serving as a reference according to an embodiment of the present invention.
In the example of FIG. 6, statistical data on holidays is shown.
In the graph shown in FIG. 6, the horizontal axis represents time for one day of 24 hours (from time 0 am to time 12 pm), and the vertical axis represents the latency [ms of each terminal device 11-1 to 11-n]. ].
In this graph, the latency detected about each terminal device 11-1 to 11-n is shown.
In addition, this graph shows the percentile characteristics in order statistics for the latencies of the plurality of terminal apparatuses 11-1 to 11-n. Specifically, a 10th percentile characteristic 621, a 25th percentile characteristic 622, a 50th percentile characteristic 623, a 75th percentile characteristic 624, and a 90th percentile characteristic 625 are shown.
Further, in this example, it is assumed that there is no commuting rush on a holiday and a period in which the latency is particularly large as a tendency is not observed.

FIG. 7 is a diagram showing an example of statistical data to be analyzed according to an embodiment of the present invention.
In the example of FIG. 7, statistical data on weekdays is shown.
In the graph shown in FIG. 7, the horizontal axis represents time for one day of 24 hours (from time 0 am to time 12 pm), and the vertical axis represents the latency [ms of each terminal apparatus 11-1 to 11-n]. ].
In this graph, the latency detected about each terminal device 11-1 to 11-n is shown.
In addition, this graph shows the percentile characteristics in order statistics for the latencies of the plurality of terminal apparatuses 11-1 to 11-n. Specifically, the 10th percentile characteristic 631, the 25th percentile characteristic 632, the 50th percentile characteristic 633, the 75th percentile characteristic 634, and the 90th percentile characteristic 635 are shown.
In this example, this graph has a commuting rush around 8am, and the trend is that the latency is the maximum in one day, but it is abnormal compared to the reference data (here, the example in FIG. 5). Suppose that the data distribution is as follows. Specifically, in the example of FIG. 7, the 10th percentile characteristic 631 and the 25th percentile characteristic 632 are closer than normal (reference time).

In this example, statistical data for weekdays as shown in FIG. 5 (for example, Monday-Friday) as shown in FIG. 5 is acquired as 22 days of reference statistical data, which is shown in FIG. Statistical data for such a holiday (here, Saturday-Sun) is acquired for 8 days.
In this example, the statistic data processing unit 132 uses the weekday statistic data as a reference, and the holiday statistic data as another reference. For this reason, the statistical data comparison unit 154 compares the processing for comparing the statistical data for one day to be analyzed with the statistical data for weekdays for the standard 22 days, and the data for one day to be analyzed. One or more of the processes for comparing the statistical data with the statistical data of the holiday for 8 days serving as a reference is performed.

FIG. 8 is a diagram showing an example of the comparison result of the statistic data for each time range according to the embodiment of the present invention.
In the example of FIG. 8, the standard weekday statistical data shown in FIG. 5 is compared with the statistical data to be analyzed shown in FIG. Further, 22 days of data is used as standard weekday statistical data, and one day of statistical data is used for each day. Such data for 22 days is a cluster. Further, as statistical data to be analyzed, data for one day is used, and statistical data for one day of the day is used.

  In the example of FIG. 8, the time range is a range of every 30 minutes, specifically, a range from 6 o'clock to 6:30 (6:00 to 6:30), 6:30 to 7 Range from Hours (6:30 to 7:00), Range from 7:00 to 7:30 (7:00 to 7:30), from 7:30 to 8:00 (7:30 to 8:00) The range is from 8 o'clock to 8:30 (8:00 to 8:30), and from 8:30 to 9 o'clock (8:30 to 9:00).

Further, in the example of FIG. 8, a scale of 30 minutes is used as a scale (time scale) of the viewpoint “time”. This 30-minute time scale period is combined with a time range period of every 30 minutes.
Further, in the example of FIG. 8, although illustrated in a plan view, any number of viewpoints other than time may be used. In this case, the data in each time range shown in FIG. 8 is data when a scale having a predetermined size is set for each scale other than time.

In the example of FIG. 8, in each time range, 22 values (also referred to as “reference values” in this embodiment) corresponding to the standard weekday statistic data are shown, forming a cluster. ing. Further, one value (also referred to as “target value” in the present embodiment) corresponding to statistical data to be analyzed is shown.
In the example of FIG. 8, one reference value 711 in the time range (6:00 to 6:30) is denoted by a reference sign, and the other reference values are omitted from the reference sign. Moreover, the code | symbol is attached | subjected to the one target value 721-726 in each time range.

The statistic data comparison unit 154 represents a distance between a plurality of (22 in this example) reference values and one target value (target values 721 to 726 in each time range) in each time range. Is calculated.
Here, as the distance, an arbitrary distance may be used. For example, a CDF-based Jensen-Shannon distance or a PDF-based Jensen-Shannon distance may be used. The Euclidean distance may be used by characterizing the percentile value of the day.

  In the time range in which the statistical data to be analyzed does not include any abnormality, the statistical data is included in the reference cluster (in this example, the reference weekday cluster or the reference holiday cluster). In the time range in which the statistical data to be analyzed includes an abnormality, the statistical data is not included in the reference cluster.

FIG. 9 is a diagram for explaining an example of an abnormal period determination process according to an embodiment of the present invention.
In the example of FIG. 8, an example in which the time scale is 30 minutes is shown. The abnormal range determination unit 155 performs comparison processing by the statistic data comparison unit 154 for the case where the time scale is 30 minutes and for time scales having any other number of different periods (different lengths of time). To do. In this example, the abnormal range determination unit 155 causes the statistical data comparison unit 154 to perform comparison processing when the time scale is 1 minute, 10 minutes, 30 minutes, and 60 minutes (= 1 hour).

In the graph shown in FIG. 9, the horizontal axis represents the time from 6:00 (6:00) to 10:00 (10:00), and the vertical axis represents the value of LOF (Local Operator Factor). Here, as the LOF, LOFs of 22 reference value clusters and one target value 721 to 726 in each time range are used. The LOF represents the degree to which the target values 721 to 726 are out of the cluster.
In the example of FIG. 9, the LOF characteristic 811 when the time scale “1 minute” is used, the LOF characteristic 812 when the time scale “10 minutes” is used, and the time scale “30 minutes” are used. The LOF characteristic 813 at the time and the LOF characteristic 814 when the time scale “60 minutes” is used are shown. Note that the timings of the start points (in the example of FIG. 3, the start point of the time range t0) are matched in the plurality of different time scale periods.

In the statistic data processing apparatus 12, for example, a predetermined threshold value Q <b> 1 related to LOF is set in the storage unit 113.
First, the abnormal range determination unit 155 calculates the calculated LOF exceeds a predetermined threshold value Q1 for each of a plurality of predetermined different time scales (in this example, 1 minute, 10 minutes, 30 minutes, and 60 minutes). It is determined whether or not. As a result of this determination, the abnormal range determination unit 155 acquires the entire period in which the LOF exceeds the predetermined threshold value Q1 on one or more time scales as an abnormal period candidate. In the example of FIG. 9, the abnormal period candidates are from 7:00 to 9:00. As the abnormal period candidate, for example, a strict period in which the LOF exceeds a predetermined threshold Q1 may be used, or a predetermined interval (for example, a unit of 1 minute or a unit of 5 minutes) Period may be used to deviate from the exact period.

Next, the abnormal range determination unit 155 sets a plurality of time ranges for the abnormal period candidates, and causes the statistical data comparison unit 154 to perform comparison processing.
In this example, the abnormal range determination unit 155 uses the minimum time scale that has already been used for the period from 7 o'clock to 9 o'clock which is a candidate for the abnormal period, and uses the start time and end time. A time range in which each of these is shifted by 1 minute is set. Specifically, the abnormal range determination unit 155 takes the start time every minute from 7 o'clock to 8:59 and the end time every 7 minutes from 7 o'clock to 9 o'clock, and the start time and the end time Set the time range for all combinations with. In other words, from 7 o'clock to 7: 01, from 7 o'clock to 7: 02, from 7 o'clock to 7: 3 ..., from 7 o'clock to 8: 59, from 7 o'clock to 7: 02 The time ranges from 7:01 to 7: 3,..., 8:58 to 8:59, 8:58 to 9 and 8:59 to 9 are set. Note that only combinations in which the start time is earlier than the end time are used. In this example, the number of combinations of (start time, end time) is {120 × 119/2}.

In the example of FIG. 9, the abnormal range determination unit 155 determines that the LOF calculated for the time range from 7 o'clock to 8:30 is the maximum among the set time ranges. Then, the abnormal range determination unit 155 determines that the period corresponding to the time range (range from 7 o'clock to 8:30) is the abnormal range 815 (ANORMALY FACTOR).
In the present embodiment, the abnormal range determination unit 155 always (for example, every fixed period) has a LOF at each of a plurality of time scales (in this example, 1 minute, 10 minutes, 30 minutes, and 60 minutes). When it is determined whether or not the predetermined threshold value Q1 is exceeded, and the LOF exceeds the predetermined threshold value Q1, as a subsequent process, a process is performed for determining a time range that is maximum for the LOF in a plurality of time ranges. Transition.

As described above, the abnormal range determination unit 155 determines the outlier (based on the degree of deviation of the value (target value) to be analyzed from the reference value (normal value) acquired in advance. It is determined whether or not (abnormal value).
The reference value (normal value) may be learned using, for example, machine learning.

Here, the size or the number of time scales initially set for detecting abnormal period candidates may be arbitrary.
In addition, when it is determined that there is no abnormality in the time scale that is initially set in order to detect an abnormal period candidate, the abnormal range determination unit 155 ends the abnormality detection process, for example.
Further, after the abnormal period candidates are detected, the time range in the case where the abnormal period is determined by further setting the time range for the detected abnormal period candidates, as the range or the number thereof. Each may be arbitrary.

Further, in the present embodiment, the case where the abnormal period is determined for the index (time scale, time range) regarding the viewpoint of time (or time) has been described. However, as another example, the viewpoint other than the time (or time) An abnormality range may be determined for an index related to (for example, the scale of the viewpoint, the range of the viewpoint). As a viewpoint other than the time (or time), for example, an area viewpoint or a device type viewpoint may be used.
In the present embodiment, the case where the abnormal range is determined for one viewpoint (time (or time) viewpoint) has been described. As another example, the abnormal range is determined for a combination of two or more different viewpoints. The determination may be performed.
In this embodiment, the distance between the target value and the cluster of the plurality of reference values is calculated and used. However, as another configuration example, one representative in the cluster is used instead of the plurality of reference value clusters. A value may be used. As one representative value in the cluster, for example, an average value (or centroid value) of values included in the cluster may be used.

Note that, as the standard statistical data, the standard holiday statistical data shown in FIG. 6 may be used instead of the standard weekday statistical data.
As another configuration example, statistical data serving as a reference for a plurality of types (for example, weekdays and holidays) and statistical data to be analyzed may be compared in parallel. In this case, for example, at the timing at which the comparison processing is performed, the closest statistical data to be analyzed among statistical data serving as a plurality of types of reference (in this embodiment, a plurality of types of clusters) ( In the case of the present embodiment, a method may be used in which a single cluster having a minimum distance) is compared.

[Example of processing performed in statistical data processing apparatus]
FIG. 10 is a diagram illustrating an example of a processing procedure for detecting the presence / absence of an abnormality performed in the statistic data processing apparatus 12 according to an embodiment of the present invention.
In the present embodiment, the abnormal range determination unit 155 performs processing for determining the abnormal range by causing the statistic data comparison unit 154, the viewpoint setting unit 151, and the scale setting unit 152 to perform respective processes.
In addition, this example is an example and other arbitrary processing procedures may be used.

(Step S1)
The statistic data comparison unit 154 acquires data (reference data) serving as a reference for determining an abnormal range.
(Step S2)
The statistic data comparison unit 154 acquires data (target data) that is a target for the abnormal range determination.
(Step S3)
The viewpoint setting unit 151 sets the viewpoint used for the abnormal range determination process.
(Step S4)
The scale setting unit 152 sets a scale to be used for abnormal range determination processing.
(Step S5)
The statistic data comparison unit 154 compares the reference data with the target data using the set viewpoint and the set scale.
(Step S6)
The abnormal range determination unit 155 determines an abnormal range based on the result of the comparison process by the statistic data comparison unit 154, thereby detecting the presence or absence of an abnormality. In addition, the abnormal range determination unit 155 may determine the abnormal range by changing the comparison processing conditions to a plurality of types and causing the statistical data comparison unit 154 to perform comparison processing. The condition for the comparison process may be, for example, a condition regarding one or more of a viewpoint, a scale, and a range (such as a time range).

[Summary of First Embodiment]
As described above, in the data processing system 1 according to the present embodiment, the statistic data processing apparatus 12 can detect anomalies by performing multidimensional and multiscale data analysis.
In the data structure of the statistic data group according to the present embodiment, for example, it is possible to monitor and detect an event such as the occurrence of an abnormality from a plurality of viewpoints and a plurality of scales. It is possible to monitor and detect scale events in parallel. In this case, in the data structure of the statistic data group according to the present embodiment, for example, it is possible to determine which viewpoint and at which scale the generated event is an event. As a specific example, if an event occurs in a wide area scale, it can be estimated that the event is caused by a cause in a wide area, and if an event occurs in a specific area scale, It can be presumed that the event is caused by a cause limited to the region.
Here, in this embodiment, although it was set as the structure which processes statistics data about the value regarding the terminal devices 11-1 to 11-n, the structure which processes statistics data about another arbitrary value may be implemented. .

(Second Embodiment)
[Data processing system]
FIG. 11 is a block diagram showing a schematic configuration of a data processing system 1001 according to one embodiment (second embodiment) of the present invention.
The data processing system 1001 includes n terminal devices 11-1 to 11-n, a statistic data processing device 1011, a database 1012, a unit statistic data generation device 1021, and a network 21. Here, the terminal devices 11-1 to 11-n and the network 21 are the same as those shown in FIG. 1, and are given the same reference numerals for convenience of explanation.
Further, the database 1012 has a function of storing data, similar to the database 13 shown in FIG.

Differences between the data processing system 1001 according to the present embodiment and the data processing system 1 shown in FIG. 1 will be described. The example of FIG. 11 is different from the example of FIG. 1 in that the unit statistics data generation device 1021 is provided and a part of the processing performed by the statistics data processing device 1011 is different.
The unit statistic data generation apparatus 1021 has a function of performing all or part of processing that can be executed by the t-digest technique (for example, see Non-patent Document 1). In the present embodiment, the unit statistic data generation apparatus 1021 uses the t-digest technique to determine the unit based on the data to be analyzed, the information for specifying the viewpoint, and the information for specifying the scale of each viewpoint. It has a function to generate statistical data.
The unit statistic data generation device 1021 may be managed by a person who manages the statistic data processing device 1011 or the unit statistic data generation device 1021 provided by another person is used. A configuration may be used.

The statistic data processing device 1011 is different from the statistic data processing device 12 shown in FIG. 1 in that the unit statistic data generating device 1021 performs the processing for the processing performed by the unit statistic data generating device 1021. It is configured to request and receive processing results.
In the present embodiment, the statistic data group generation unit 153 transmits a signal requesting processing for generating unit statistic data to the unit statistic data generation apparatus 1021 via the network 21 by the communication unit 114. The signal includes information necessary for generating unit statistic data, for example, data to be analyzed (or information for specifying it), information for specifying one or more viewpoints, and each Contains information that identifies the scale of the viewpoint.
When the unit statistic data generation apparatus 1021 receives such a request signal, the unit statistic data generation apparatus 1021 generates unit statistic data in response to the request, and sends a signal including the generated unit statistic data via the network 21. And sent to the statistics data processing apparatus 1011.
The statistic data group generation unit 153 uses (uses) the unit statistic data received by the communication unit 114 from the unit statistic data generation apparatus 1021 to generate a statistic data group.

(Summary of the above embodiments)
As one configuration example, for at least one viewpoint and a plurality of scales for each viewpoint, statistical data for comparing reference data, which is statistical data based on data included for each combination of viewpoints and scales, and comparison target data An abnormality for determining a range that is considered to be abnormal among a plurality of ranges of viewpoints based on a comparison result by the comparison unit (statistical data comparison unit 154 in the example of FIG. 2) and the statistical data comparison unit A statistic data processing device (in the example of FIGS. 1 and 11, the statistic data processing devices 12 and 1011) including a range determination unit (in the example of FIG. 2, an abnormal range determination unit 155).
As an example of the configuration, the abnormal range determination unit determines range candidates (abnormal range candidates) of viewpoints considered to be abnormal based on the result of comparison by the statistic data comparison unit, and the abnormal range determination unit Among the plurality of ranges included in the viewpoint range candidates, a range in which the LOF representing the difference between the reference data and the comparison target data is maximized is determined as a range that is considered to be abnormal.
As one configuration example, the statistic data comparison unit compares a plurality of reference data with comparison target data.
As one configuration example, one or both of the reference data and the comparison target data is statistical data based on data included in each combination of viewpoints and scales for different viewpoints and different scales for each viewpoint. Is obtained from the statistic data group (in the example of FIG. 3, the statistic data group 201).
As one configuration example, the statistical data comparison unit includes, for at least one viewpoint and a plurality of scales for each viewpoint, reference data and comparison target data that are statistics data based on data included for each combination of viewpoints and scales. And a statistical data processing method in which an abnormal range determination unit determines a range considered to be abnormal among a plurality of ranges of viewpoints based on a result of comparison by the statistical data comparison unit (see FIG. 1, in the example of FIG. 11, a method of processing performed by the statistic data processing devices 12 and 1011).
As one configuration example, the statistical data comparison unit includes, for at least one viewpoint and a plurality of scales for each viewpoint, reference data and comparison target data that are statistics data based on data included for each combination of viewpoints and scales. And a step in which an abnormal range determination unit determines a range that is considered to be abnormal among a plurality of ranges of viewpoints based on a result of comparison by the statistical data comparison unit. A program to be executed (in the example of FIGS. 1 and 11, a computer configuring the statistic data processing devices 12 and 1011).

A program for realizing the function of each device (for example, the statistic data processing device 12, 1011, etc.) according to the embodiment described above is recorded on a computer-readable recording medium (storage medium), and this recording medium Processing may be performed by causing the computer system to read and execute the program recorded in the above.
Here, the “computer system” may include hardware such as an operating system or peripheral devices.
The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM (Read Only Memory), a writable non-volatile memory such as a flash memory, a portable medium such as a DVD (Digital Versatile Disc), A storage device such as a hard disk built in a computer system.
Furthermore, the “computer-readable recording medium” refers to a volatile memory (for example, DRAM (DRAM) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Dynamic Random Access Memory)) that holds a program for a certain period of time is also included.
The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
Further, the above program may be for realizing a part of the functions described above. Further, the above program may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

  As mentioned above, although embodiment of this invention was explained in full detail with reference to drawings, the concrete structure is not restricted to this embodiment, The design change etc. of the range which does not deviate from the summary of this invention are included.

DESCRIPTION OF SYMBOLS 1,1001 ... Data processing system, 11-1 to 11-n ... Terminal device, 12, 1011 ... Statistics data processing device, 13, 1012 ... Database, 21 ... Network, 111 ... Input part, 112 ... Output part, 113 ... storage unit 114 ... communication unit 115 ... control unit 131 ... data acquisition unit 132 ... statistic data processing unit 133 ... data output control unit 151 ... viewpoint setting unit 152 ... scale setting unit 153 ... statistics Quantity data group generation unit, 154... Statistics data comparison part, 155... Abnormal range determination part, 201, 511-1 to 511-30, 522... Statistics data group, 211 to 213, 221 to 223, 311 to 313, 321 to 323, 411 to 413, 421 to 423... Unit statistics data, 511..., Reference data, 521. Comparison processing, 524 ... abnormal detection processing, 525 ... abnormal range, 611-615, 621-625, 631-635, 811-814 ... characteristics, 711 ... reference value, 721-726 ... target value, 1021 ... unit statistic data Generator, Q1 ... threshold

Claims (6)

A statistic data comparison unit that compares reference data, which is statistical data based on data included in each combination of the viewpoint and the scale, and comparison target data with respect to at least one viewpoint and a plurality of scales for each viewpoint. When,
An abnormal range determination unit that determines a range that is considered to be abnormal among a plurality of ranges of the viewpoint, based on a result of comparison by the statistical data comparison unit;
A statistics data processing apparatus comprising: The abnormal range determination unit determines a range candidate of the viewpoint that is considered to be abnormal based on a result of the comparison by the statistical data comparison unit,
The abnormal range determination unit is considered to be abnormal in a range in which a LOF representing a difference between the reference data and the comparison target data is maximum among a plurality of ranges included in the range candidates of the viewpoint. Judging as a range,
The statistic data processing apparatus according to claim 1. The statistic data comparison unit compares a plurality of the reference data with the comparison target data.
The statistic data processing apparatus according to claim 1. One or both of the reference data and the comparison target data are statistics based on data included for each combination of the viewpoint and the scale with respect to a plurality of different viewpoints and a plurality of different scales for each viewpoint. Obtained from a statistic data group having quantity data,
The statistics data processing apparatus according to any one of claims 1 to 3. The statistic data comparison unit includes, for at least one viewpoint and a plurality of scales for each viewpoint, reference data and comparison target data that are statistics data based on data included for each combination of the viewpoint and the scale. Compare and
An abnormal range determination unit determines a range that is considered to be abnormal among a plurality of ranges of the viewpoint based on a result of comparison by the statistical data comparison unit,
Statistics data processing method. The statistic data comparison unit includes, for at least one viewpoint and a plurality of scales for each viewpoint, reference data and comparison target data that are statistics data based on data included for each combination of the viewpoint and the scale. A step of comparing;
An abnormal range determining unit, based on a result of comparison by the statistical data comparing unit, determining a range that is considered to be abnormal among a plurality of ranges of the viewpoint;
A program that causes a computer to execute.