JP6795448B2 - Data processing equipment, data processing methods and programs - Google Patents

Description

The present invention relates to data processing devices, data processing methods and programs.

In various fields such as transportation and healthcare, time-series data of a predetermined target is detected. It may be important to immediately detect the abnormality to be detected from such time series data.
In addition, in many cases, stream data in which the number of samples of time-series data becomes enormous and data is constantly generated one after another is processed.

The method of detecting an abnormality is roughly divided into a method of detecting an abnormality using prior knowledge and a method of detecting an outlier as an abnormality.
In the method of detecting an abnormality using prior knowledge, when time-series data satisfying a specific condition is generated, the abnormality is detected as the occurrence of the abnormality, but it is not possible to deal with an unknown event. In addition, in the method of detecting an abnormality using prior knowledge, preconditions are often assumed, so it is necessary to modify a specific condition each time the precondition changes. In addition, this method requires learning prior knowledge. Machine learning may be used to make up for this, but sufficiently high-speed processing has not been established.

Under these circumstances, it is important to establish and improve the method for detecting outliers in stream data.
For example, although techniques for detecting outliers have existed for some time, there are cases where sufficiently high-speed processing is not performed, especially when targeting stream data. In situations where it is difficult to predict when an abnormality will occur, it is necessary to constantly monitor the presence or absence of an abnormality, and the processing speed is sufficiently high compared to the speed at which the data that makes up the time series data occurs one after another. It had to be realized.

As an example, Non-Patent Document 1 proposes a technique called t-digest. The t-digest has a data structure for estimating the distribution of a set of values, and for example, it is possible to estimate the distribution of stream data.
However, it may be difficult to realize sufficiently high-speed processing only by t-digest.

TED DUNNING AND HTTPS ERTL, "COMPUTING EXTREMELY ACCURATE QUANTILES USING t-DIGESTS", [online], [Search April 12, 2017], Internet <URL: https://github.com/tdunning/t-di blob / master / docs / t-digest-paper / histo.pdf ＞

In the past, it has sometimes been difficult to perform multidimensional and multiscale data analysis. In addition, in t-digest, multidimensional and multiscale were not assumed.

The present invention has been made in consideration of such circumstances, and provides a data processing apparatus, a data processing method, and a program capable of detecting an abnormality by performing multidimensional and multiscale data analysis. That is the issue.

As a configuration example, for at least one viewpoint and a plurality of scales for each viewpoint, reference data, which is statistical data based on data included in each combination of the viewpoint and the scale, and comparison target data are compared. A statistic data comparison unit and an abnormality range determination unit for determining a range deemed to be abnormal among a plurality of ranges of the viewpoint based on the result of comparison by the statistic data comparison unit are provided . The abnormality range determination unit determines a range candidate of the viewpoint considered to have an abnormality based on the result of comparison by the statistics data comparison unit, and the abnormality range determination unit is included in the range candidate of the viewpoint. It is a statistic data processing apparatus that determines the range in which the LOF representing the difference between the reference data and the comparison target data is maximum among the plurality of ranges to be determined as a range considered to be abnormal .

According to the present invention, it is possible to detect anomalies by performing multidimensional and multiscale data analysis.

It is a block diagram which shows the schematic structure of the data processing system which concerns on one Embodiment (1st Embodiment) of this invention. It is a block diagram which shows the schematic structure of the statistic data processing apparatus which concerns on one Embodiment of this invention. It is a figure which shows typically the data structure of an example of the statistic data group which concerns on one Embodiment of this invention. It is a figure which shows the outline of the abnormality detection processing based on the statistical data which concerns on one Embodiment of this invention. It is a figure which shows an example of the statistical data which becomes the reference which concerns on one Embodiment of this invention. It is a figure which shows another example of the statistic data which becomes the reference which concerns on one Embodiment of this invention. It is a figure which shows an example of the statistical data to be analyzed which concerns on one Embodiment of this invention. It is a figure which shows an example of the comparison result of the statistic data for each time range which concerns on one Embodiment of this invention. It is a figure for demonstrating an example of the determination process of the abnormal period which concerns on one Embodiment of this invention. It is a figure which shows an example of the process procedure for detecting the presence or absence of an abnormality performed in the statistic data processing apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows the schematic structure of the data processing system which concerns on one Embodiment (second Embodiment) of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First Embodiment)
[Data processing system]
FIG. 1 is a block diagram showing a schematic configuration of a data processing system 1 according to an embodiment (first embodiment) of the present invention.
The data processing system 1 includes n (n is an integer of 2 or more) terminal devices 11-1 to 11-n, a statistic data processing device 12, a database 13, and a network 21.
As the network 21, any wired or wireless network may be used, and for example, the Internet or a Wi-Fi (registered trademark) network may be used.
In the present embodiment, the database 13 is provided separately from the statistic data processing device 12, but as another example, the database 13 may be integrated with the statistic data processing device 12.

The statistic data processing device 12 acquires and analyzes predetermined target data regarding the terminal devices 11-1 to 11-n. In the present embodiment, it is assumed that the data is stream data in which the number of samples becomes enormous (big data) and data is always generated one after another for a predetermined target time-series data.

As the predetermined target data, arbitrary data may be used, for example, data relating to IoT (Internet of Things) may be used, or other data may be used.
As a specific example, data in an arbitrary system may be used as the data of a predetermined target, for example, data related to a transportation system such as a vehicle, data related to a healthcare system such as a person, a factory system for producing a product, or the like. Data about financial systems such as securities, data about wired or wireless communication systems, and the like may be used. Further, in an arbitrary system, various data may be used as the predetermined target data, for example, temperature-related data, humidity-related data, velocity-related data, acceleration-related data, image-related data, oxygen, and the like. Data on the concentration of substances, data on quality, data on stock prices, data on communication signals, data on the position of terminal devices (terminal devices 11-1 to 11-n in this embodiment), data on areas where terminal devices exist. Etc. may be used.

In addition, various methods may be used as a method for detecting data of a predetermined target.
As an example, in each terminal device 11-1 to 11-n, a configuration for detecting predetermined target data regarding the respective terminal device 11-11 to 11-n may be used. In this configuration, each terminal device 11-1 to 11-n includes a detection unit that detects data of a predetermined target, and processes the data detected by the detection unit as statistical data via the network 21. It is transmitted to the device 12. The detection unit may be, for example, a sensor or an imaging device (camera). Here, the image pickup device (camera) may also be regarded as an example of the sensor.
In addition, each terminal device 11-1 to 11-n may be an IoT terminal device, or may be another terminal device.

As another example, the data processing system 1 may include a detection device (not shown) different from the terminal devices 11-11 to 11-n. In this configuration, the detection device has a detection unit that detects predetermined target data regarding each terminal device 11-1 to 11-n, and the data detected by the detection unit is transmitted to the network 21. It is transmitted to the statistic data processing device 12 via.

The detection device may include, for example, an imaging device (camera) that captures an image of a predetermined area, and the number of terminal devices 11-1 to 11-n existing in the predetermined area based on the image. Data and the like may be detected as predetermined target data.
The detection device may include, for example, a communication unit that communicates wirelessly or by wire with terminal devices 11-1 to 11-n existing in a predetermined area, and based on the result of this communication, the detection device may be located in the predetermined area. Data of the number of existing terminal devices 11-1 to 11-n may be detected as predetermined target data.
The detection device may include, for example, a signal acquisition unit that acquires a signal transmitted from the terminal devices 11-1 to 11-n, and based on the status of the signal, the frequency of occurrence or the degree of delay of the signal, etc. Data representing the above may be detected as predetermined target data.

In the present embodiment, each terminal device 11-11 to 11-n can communicate with another device (for example, a statistic data processing device 12) via the network 21. Each terminal device 11-1 to 11-n communicates with the network 21 by wire or wirelessly.
As another example, each terminal device 11-1 to 11-n does not have to have a communication function.

Further, as the respective terminal devices 11-1 to 11-n, for example, terminal devices having the same configuration may be used, or terminal devices having different configurations may be included.
Further, each of the terminal devices 11-1 to 11-n may be added to or attached to an object, or may be carried or attached by a person. The thing may be arbitrary, for example, a vehicle such as an automobile, an electric appliance, or the like.

Further, a detection device other than the terminal devices 11-1 to 11-n may be provided in, for example, the statistic data processing device 12. In this configuration, the statistic data processing device 12 acquires and analyzes the data detected by the detection device.

[Statistical data processing device]
FIG. 2 is a block diagram showing a schematic configuration of a statistic data processing device 12 according to an embodiment of the present invention.
The statistic data processing device 12 includes an input unit 111, an output unit 112, a storage unit 113, a communication unit 114, and a control unit 115.
The control unit 115 includes a data acquisition unit 131, a statistic data processing unit 132, and a data output control unit 133.
The statistic data processing unit 132 includes a viewpoint setting unit 151, a scale setting unit 152, a statistic data group generation unit 153, a statistic data comparison unit 154, and an abnormality range determination unit 155.

The input unit 111 inputs information from the outside. The input unit 111 has, for example, an operation unit that accepts an operation performed by a user (person), and inputs information according to the operation received by the operation unit. The input unit 111 is connected to, for example, an external device (for example, a recording medium) and inputs information output from the external device.
The output unit 112 outputs information. The output unit 112 has, for example, a screen, and displays (outputs) information on the screen. The output unit 112 is connected to, for example, an external device (for example, a recording medium) and outputs information to the external device.
The storage unit 113 stores information. In this embodiment, the storage unit 113 and the database 13 may be used arbitrarily.
The communication unit 114 communicates information. In the present embodiment, the communication unit 114 communicates information with another device (for example, terminal device 11-1 to 11-n or another detection device) via the network 21.

The control unit 115 performs various controls on the statistic data processing device 12.
In the present embodiment, the storage unit 113 stores information on a predetermined control program and its parameters. Further, the control unit 115 is configured by using a CPU (Central Processing Unit). Then, in the control unit 115, the CPU executes various controls by executing the control program stored in the storage unit 113 using the parameters stored in the storage unit 113.

The configuration of the statistic data processing device 12 including the processing units 111 to 115 shown in FIG. 2 is an example, and other configurations may be used. For example, the functional divisions of the processing units 111 to 115 are for convenience of explanation and are not necessarily limited to the configuration shown in FIG.

The function of the control unit 115 will be described.
The data acquisition unit 131 acquires a predetermined target data as the data to be analyzed.
As an example, the data acquisition unit 131 acquires data received from another device (for example, terminal device 11-1 to 11-n or another detection device) by the communication unit 114 as data to be analyzed. May be good.
As another example, the data acquisition unit 131 sequentially stores the data acquired in the past in the database 13, and when processing the data, acquires the data from the database 13 as the data to be analyzed. You may.
Further, as another example, a configuration may be used in which the data of a predetermined target is stored in the database 13 without going through the statistics data processing device 12, and in this case, the data acquisition unit 131 of the data At the time of processing, the data may be acquired from the database 13 as data to be analyzed.

The statistic data processing unit 132 performs statistical processing on the data acquired by the data acquisition unit 131, and acquires the resulting data (also referred to as “statistical data” in the present embodiment).
In addition, the statistic data processing unit 132 may process, for example, data acquired one after another in real time by the data acquisition unit 131 (a group of newly increasing data), and data acquisition. There may be a case where processing is performed on the past data (a group of data that does not newly increase) acquired by the unit 131.
In addition, the statistic data processing unit 132 detects (determines) the presence or absence of an abnormality based on the statistic data.

The data output control unit 133 controls the output unit 112 to output the data to be output. As this output, for example, a display output such as a character, a figure, or a graph is used.
Arbitrary data may be used as the data to be output. For example, the data acquired by the data acquisition unit 131 or the result data obtained by the processing by the statistic data processing unit 132 is used. May be done. The result data obtained by the processing by the statistic data processing unit 132 includes, for example, the result data obtained by the processing by the statistic data group generation unit 153 and the result data obtained by the processing by the statistic data comparison unit 154. One or more of the resulting data or the result data obtained by processing by the abnormal range determination unit 155 may be used.

The statistic data processing unit 132 will be described.
The viewpoint setting unit 151 sets a viewpoint (item) for the data to be analyzed. The viewpoint setting unit 151 may set two or more viewpoints for the data to be analyzed.
Here, as the viewpoint, any viewpoint may be used, and for example, time (time), area (region), device type, and the like may be used.
Further, the viewpoint setting unit 151 may set, for example, a predetermined viewpoint, or may set a viewpoint instructed by a user or the like. When the viewpoint is predetermined, for example, information that identifies the viewpoint is stored in the storage unit 113.

The scale setting unit 152 sets the scale (grain size) for each viewpoint. The scale setting unit 152 may set two or more scales for each viewpoint.
Here, as the scale, a scale having an arbitrary size may be used.
For example, as the time scale, a 1-second scale, a 1-minute scale, a 1-hour scale, a 1-day scale, or any other scale of any size may be used.
For example, the area scale may be a district scale, a city scale, a prefecture scale, a national scale, or any other scale of any size.
For example, as the scale of the device type, the scale of the model, the scale of the manufacturer, the scale of the OS (Operating System), or the scale of any other type (attribute) may be used.
For example, the scale setting unit 152 may set a predetermined scale for each viewpoint, or may set a scale instructed by a user or the like. When the scale is predetermined, for example, information for identifying the scale is stored in the storage unit 113.

The statistic data group generation unit 153 performs statistical processing on the data acquired by the data acquisition unit 131 based on the viewpoint set by the viewpoint setting unit 151 and the scale set by the scale setting unit 152. Then, a set of a plurality of statistical data (also referred to as a "statistical data group" in the present embodiment) is generated.
In the present embodiment, the statistic data group generation unit 153 collects the data resulting from the multidimensional and multiscale data analysis into the statistic data group (in the present embodiment, the “multidimensional multiscale statistic data group””. Also called.).
Here, in the present embodiment, the dimension represents a viewpoint, and the multidimensional represents a plurality of viewpoints.
Further, in the present embodiment, the multi-scale indicates that there are a plurality of scales.

The statistic data comparison unit 154 compares the reference statistic data with the statistic data to be analyzed. In the present embodiment, the statistic data comparison unit 154 compares the reference statistic data with the statistic data to be analyzed for a plurality of different scales.
Here, as the reference statistic data, for example, the statistic data may be acquired by the data acquisition unit 131, or the statistic data included in the statistic data group generated by the statistic data group generation unit 153 is used. You may.
Further, the statistic data to be analyzed may be acquired by, for example, the data acquisition unit 131, or the statistic data processing unit 132 (for example, statistic data) based on the data acquired by the data acquisition unit 131. The statistic data generated by the comparison unit 154) may be used, or the statistic data included in the statistic data group generated by the statistic data group generation unit 153 may be used.

The abnormality range determination unit 155 is a range in which an abnormality has occurred (in the present embodiment, a time (time) range) for the statistic data to be analyzed based on the result of comparison by the statistic data comparison unit 154. Also referred to as "abnormal period"). It should be noted that this determination may be, for example, a presumptive determination.
In the present embodiment, the existence of an abnormality is detected by determining the abnormality period by the abnormality range determination unit 155 for the statistical data to be analyzed.

[Example of data structure of statistic data group]
FIG. 3 is a diagram schematically showing a data structure of an example of the statistic data group 201 according to the embodiment of the present invention.
In the example of FIG. 3, the statistic data group 201 is a multidimensional multi-scale statistic data group which is the result of processing time series data by the statistic data processing unit 132. The statistic data group 201 has a structure of distribution data (statistical data in the present embodiment) extended in multiple dimensions and multiscales for time series data.
In the example of FIG. 3, time, area, and device type are used as a plurality of viewpoints. In addition, a plurality of scales (multi-scales) are used for each viewpoint.

In the example of FIG. 3, a time scale s0, a time scale s1, and a time scale s2 are used as a plurality of scales related to time. As an example, the time scale s0 is "10 seconds", the time scale s1 is "20 seconds", and the time scale s2 is "40 seconds".
Further, as a plurality of scales related to the region, the region scale A1, the region scale A2, and the region scale A3 are used. As an example, the area scale A1 is "Tokyo", the area scale A2 is "Kanto", and the area scale A3 is "Japan (nationwide)".
Further, as a plurality of scales related to the device type, the device type scale D1 and the device type scale D2 are used. As an example, the device type scale D1 is a "specific model A1", and the device type scale D2 is a "specific manufacturer B1".

Further, in the example of FIG. 3, the direction (arrow) of the axis representing the time is shown. As the time range (time division), the time range t0, the time range t1, the time range t2, and the time range t3 are used. The time range t0 is "0 seconds or more and less than 10 seconds", the time range t1 is "10 seconds or more and less than 20 seconds", the time range t2 is "20 seconds or more and less than 30 seconds", and the time range t3 is " 30 seconds or more and less than 40 seconds. "
Any timing may be used as the initial value of the time (0 seconds in this embodiment).

Here, when one time scale, one area scale, and one device type scale are specified and one time range corresponding to the time scale is specified, the statistic becomes a unit. Data (also referred to as "unit statistic data" in this embodiment) is specified.
The unit statistic data specified by one time scale, one area scale, one device type scale, and one time range according to the time scale is the maximum time in the time range. By performing a statistical calculation by the statistic data processing unit 132 for data belonging to the time scale from to the time retroactive to the time, belonging to the region scale, and belonging to the device type scale. Corresponds to the obtained statistic data.

In the example of FIG. 3, six unit statistic data 211 to 213 and 221 to 223 corresponding to the time scale s0 and the time range t3 and six unit statistic data 311 corresponding to the time scale s1 and the time range t3. Only 313 and 321 to 223 and the six unit statistic data 411 to 413 and 421 to 423 corresponding to the time scale s2 and the time range t3 are coded, and the other unit statistic data are coded. Is omitted.

As an example, the unit statistic data 211 corresponds to the time scale S0 (10 seconds), the area scale A1 (Tokyo), the device type scale D1 (specific model A1), and the time range t3 (30 seconds or more and less than 40 seconds). .. The unit statistic data 211 is statistic data obtained based on data in which the time belongs to 30 seconds or more and less than 40 seconds, the area belongs to Tokyo, and the device type belongs to a specific model A1. Is. That is, the data is generated in the terminal devices 11-1 to 11-n existing in Tokyo, the time is 30 seconds or more and less than 40 seconds, and the device type is the specific model A1. It means that the data is generated in ~ 11-n. The unit statistic data is statistic data obtained by using a set of such data.

Here, as the unit statistic data, data of an arbitrary statistic may be used, for example, data of an arbitrary value related to order statistics may be used, or data of an average value may be used. May be good.
As the value related to the order statistic, for example, the median value may be used. In general, when a plurality of data to be processed are the same, it is considered that the processing time for acquiring the median value is shorter than the processing for acquiring (calculating) the average value.
Further, as the value related to the order statistics, for example, the value of the cumulative distribution function (CDF: Cumulative Distribution Function) may be used, or the value of the probability distribution function (PDF: Probability Density Function) may be used. ..

In the example of FIG. 3, the unit of each square (rectangular parallelepiped or cube) is unit statistic data (unit statistic data 211-213, 221-223, 31-13, 321-23, 411-413, 421-423, etc. ) Corresponds to.
The statistic data group generation unit 153 may acquire unit statistic data by using an arbitrary method. For example, refer to an existing technique of t-digest (for example, Non-Patent Document 1). ) May be used to calculate and obtain unit statistic data. In the present embodiment, the statistic data group is configured to include a plurality of unit statistic data.

Further, as the plurality of scales used for one viewpoint, for example, a plurality of scales having an inclusive relationship with each other may be used, or a plurality of scales having an inclusive relationship with each other may be used. Alternatively, a plurality of scales that are inclusively related to only a part may be used.
As an example of the scale of the area, there are "Tokyo", "Kanto", and "Japan" as a plurality of scales having an inclusive relationship with each other.
As a plurality of scales that are not inclusive of each other, for example, "Tokyo", "Chiba", and "Ibaraki" are examples of the area scale.
As an example of the scale of the area, there are "Tokyo", "Kanto" (including Tokyo), and "Osaka" as a plurality of scales that are inclusively related to only a part.

[Abnormality detection processing performed in the statistic data processing unit]
FIG. 4 is a diagram showing an outline of an abnormality detection process based on statistical data according to an embodiment of the present invention.
In the present embodiment, data relating to latency is used for a plurality of terminal devices 11-1 to 11-n as data to be analyzed (comparison target) and reference data. As the latency, the time (delay time) until a response arrives in response to the request transmitted from each terminal device 11-1 to 11-n is used.
Here, it is assumed that each terminal device 11-1 to 11-n is possessed by a different person. For convenience of explanation, it is assumed that there is a commuting rush at a fixed time on weekdays and no commuting rush on holidays. In addition, there is usually a morning rush hour and an evening commuting rush on weekdays, but in this example, only the morning commuting rush is shown and the evening commuting rush is omitted.

In the example of FIG. 4, the statistic data group 522 to be analyzed is generated based on the data 521 to be analyzed. In the present embodiment, the statistic data group 522 to be analyzed is generated based on the data for one day to be detected for the presence or absence of abnormality.
Further, in the example of FIG. 4, as the reference data 511, the statistic data group 511-1 to 511-30 generated based on the data for one day is used for each day of the past 30 days. Has been done. Here, it is assumed that no abnormality has occurred on the 30th day when the reference data 511 is acquired, and it is assumed that the normal reference data is acquired.

The statistic data comparison unit 154 performs a process (comparison process 523) of comparing the reference data 511 with the statistic data group 522 to be analyzed.
Based on the result of such comparison, the abnormality range determination unit 155 performs a process of determining the abnormality range of the statistical data (in this example, the abnormality detection process 524 for detecting an abnormal distribution). As a result, the abnormality range determination unit 155 determines the abnormality range 525 for the statistical data group to be analyzed.
In the present embodiment, when the abnormality range 525 exists, it is detected (determined) that there is an abnormality, and when the abnormality range 525 does not exist, it is detected (determined) that there is no abnormality.
In this example, a case where an abnormal period within a time (time) range is detected (determined) as an abnormal range 525 will be described.

An example of statistical data is shown with reference to FIGS. 5 to 7. The statistical data shown in FIGS. 5 to 7 are artificially created for convenience of explanation, and represent the time transition of the data distribution.
FIG. 5 is a diagram showing an example of statistical data as a reference according to an embodiment of the present invention.
In the example of FIG. 5, statistical data on weekdays (Weekday) is shown.
In the graph shown in FIG. 5, the horizontal axis represents the time for 24 hours per day (from time 0 am to time 12 pm), and the vertical axis represents the latency [ms] of each terminal device 11-11 to 11-n. ] Is represented.
This graph shows the latency detected for each terminal device 11-1 to 11-n.
In addition, this graph shows percentile characteristics in order statistics for the latencies of a plurality of terminal devices 11-1 to 11-n. Specifically, the characteristics of the 10th percentile 611, the characteristics of the 25th percentile 612, the characteristics of the 50th percentile 613, the characteristics of the 75th percentile 614, and the characteristics of the 90th percentile 615 are shown.
Further, in this example, it is assumed that there is a commuting rush around 8 am on weekdays, and the latency tends to be the maximum in the day.

FIG. 6 is a diagram showing another example of statistical data as a reference according to an embodiment of the present invention.
In the example of FIG. 6, statistical data on holidays (Weekend) is shown.
In the graph shown in FIG. 6, the horizontal axis represents the time for 24 hours per day (from time 0 am to time 12 pm), and the vertical axis represents the latency [ms] of each terminal device 11-11 to 11-n. ] Is represented.
This graph shows the latency detected for each terminal device 11-1 to 11-n.
In addition, this graph shows percentile characteristics in order statistics for the latencies of a plurality of terminal devices 11-1 to 11-n. Specifically, the characteristics 621 of the 10th percentile, the characteristic 622 of the 25th percentile, the characteristic 623 of the 50th percentile, the characteristic 624 of the 75th percentile, and the characteristic 625 of the 90th percentile are shown.
Further, in this example, it is assumed that there is no commuting rush on holidays and a period in which the latency is particularly large is not observed as a tendency.

FIG. 7 is a diagram showing an example of statistical data to be analyzed according to an embodiment of the present invention.
In the example of FIG. 7, statistical data on weekdays (Weekday) is shown.
In the graph shown in FIG. 7, the horizontal axis represents the time for 24 hours per day (from time 0 am to time 12 pm), and the vertical axis represents the latency [ms] of each terminal device 11-11 to 11-n. ] Is represented.
This graph shows the latency detected for each terminal device 11-1 to 11-n.
In addition, this graph shows percentile characteristics in order statistics for the latencies of a plurality of terminal devices 11-1 to 11-n. Specifically, the characteristics 631 of the 10th percentile, the characteristic 632 of the 25th percentile, the characteristic 633 of the 50th percentile, the characteristic 634 of the 75th percentile, and the characteristic 635 of the 90th percentile are shown.
In addition, in this example, there is a commuting rush around 8 am in this graph, and the latency tends to be the maximum in one day, but it is abnormal compared to the reference data (here, the example in FIG. 5). Data distribution is assumed. Specifically, in the example of FIG. 7, the characteristic 631 of the 10th percentile and the characteristic 632 of the 25th percentile are closer than those in the normal state (reference time).

In this example, as the reference 30-day statistic data, 22 days of weekday (here, Monday-Friday) statistic data as shown in FIG. 5 are acquired, and are shown in FIG. Statistic data for such holidays (here, Saturday-Sunday) has been acquired for eight days.
Then, in this example, the statistic data processing unit 132 uses the statistic data on weekdays as a reference and the statistic data on holidays as another reference. Therefore, the statistic data comparison unit 154 compares the statistic data for one day to be analyzed with the statistic data for 22 days as a reference on weekdays, and the statistic data comparison unit 154 for one day to be analyzed. Perform one or more of the processes of comparing the statistic data with the reference statistic data of eight days' holidays.

FIG. 8 is a diagram showing an example of a comparison result of statistical data for each time range according to an embodiment of the present invention.
In the example of FIG. 8, the reference weekday statistic data shown in FIG. 5 and the statistic data to be analyzed shown in FIG. 7 are compared. In addition, as the reference weekday statistic data, 22 days worth of data is used, and one day's worth of statistic data is used for each day. Such 22 days' worth of data is in a cluster. Further, as the statistic data to be analyzed, the data for one day is used, and the statistic data for one day on that day is used.

In the example of FIG. 8, the time range is a range of every 30 minutes, specifically, a range from 6:00 to 6:30 (6:00 to 6:30), and from 6:30 to 7. Range from time (6:30 to 7:00), range from 7:00 to 7:30 (7:00 to 7:30), from 7:30 to 8:00 (7:30 to 8:00) The range is from 8:00 to 8:30 (8:00 to 8:30), and from 8:30 to 9:00 (8:30 to 9:00).

Further, in the example of FIG. 8, a scale of 30 minutes is used as a scale (time scale) of the viewpoint "time". The period of this 30-minute time scale is combined with the period of the time range every 30 minutes.
Further, in the example of FIG. 8, although it is shown in a plane, any number of viewpoints other than time may be used. In this case, the data in each time range shown in FIG. 8 is the data when a scale of a predetermined size is set for the scale of each viewpoint other than time.

In the example of FIG. 8, 22 values (also referred to as “reference values” in the present embodiment) corresponding to the reference weekday statistical data are shown in each time range to form a cluster. ing. In addition, one value (also referred to as "target value" in the present embodiment) corresponding to the statistical data to be analyzed is shown.
In the example of FIG. 8, one reference value 711 in the time range (6:00 to 6:30) is coded, and the reference values are omitted for the other reference values. Further, one target value 721 to 726 in each time range is coded.

The statistic data comparison unit 154 is a value representing the distance between a plurality of (22 in this example) reference values and one target value (target values 721 to 726 in each time range) in each time range. Is calculated.
Here, as the distance, an arbitrary distance may be used, for example, a CDF-based Jensen-Shannon distance or a PDF-based Jensen-Shannon distance may be used, and as another example, respectively. The Euclidean distance may be used by quantifying the percentile value of the day.

In the time range where the statistic data to be analyzed does not include anomalies, the statistic data is included in the reference cluster (in this example, the reference weekday cluster or the reference holiday cluster). , In the time range where the statistic data to be analyzed contains anomalies, the statistic data is not included in the reference cluster.

FIG. 9 is a diagram for explaining an example of an abnormal period determination process according to an embodiment of the present invention.
In the example of FIG. 8, an example when the time scale is 30 minutes is shown. The abnormality range determination unit 155 also performs comparison processing by the statistic data comparison unit 154 for the case where the time scale is 30 minutes and for the time scale having an arbitrary number of different periods (different time lengths). To do. In this example, the abnormality range determination unit 155 causes the statistic data comparison unit 154 to perform comparison processing when the time scales are 1 minute, 10 minutes, 30 minutes, and 60 minutes (= 1 hour), respectively.

In the graph shown in FIG. 9, the horizontal axis represents the time from 6:00 (6:00) to 10:00 (10:00), and the vertical axis represents the value of LOF (Local Outlier Factor). Here, as the LOF, a LOF of 22 reference value clusters and one target value 721 to 726 in each time range is used. The LOF represents the degree to which the target values 721 to 726 are out of the cluster.
In the example of FIG. 9, the LOF characteristic 811 when the time scale "1 minute" was used, the LOF characteristic 812 when the time scale "10 minutes" was used, and the time scale "30 minutes" were used. The characteristic 813 of the LOF at the time and the characteristic 814 of the LOF when the time scale "60 minutes" is used are shown. The timing of the start point (in the example of FIG. 3, the start point of the time range t0) is matched to the periods of the plurality of different time scales.

In the statistic data processing device 12, for example, the storage unit 113 is set with a predetermined threshold value Q1 regarding the LOF.
First, in the abnormality range determination unit 155, the calculated LOF exceeds a predetermined threshold value Q1 for each of a plurality of predetermined different time scales (1 minute, 10 minutes, 30 minutes, 60 minutes in this example). Judge whether or not. As a result of this determination, the abnormality range determination unit 155 acquires the entire period in which the LOF exceeds a predetermined threshold value Q1 on one or more time scales as candidates for the abnormality period. In the example of FIG. 9, the candidates for the abnormal period are from 7:00 to 9:00. As the candidate for the abnormal period, for example, a strict period in which the LOF exceeds a predetermined threshold value Q1 may be used, or a predetermined division (for example, a 1-minute unit division or a 5-minute unit) may be used. A period (such as a delimiter) may be used to deviate from the exact period.

Next, the abnormality range determination unit 155 sets a plurality of time ranges for the candidates of the abnormality period, and causes the statistic data comparison unit 154 to perform comparison processing.
In this example, the anomaly range determination unit 155 sets the start time and end time for the period from 7:00 to 9:00, which is a candidate for the anomaly period, by using 1 minute, which is the smallest time scale already used. Set a time range in which each of the above is shifted by 1 minute. Specifically, the abnormality range determination unit 155 takes the start time from 7:00 to 8:59 every minute and the end time from 7:01 to 9:00 every minute, and the start time and the end time. Set the time range for all combinations with. That is, from 7:00 to 7:01, from 7:00 to 7:02, from 7:00 to 7:03, ..., from 7:00 to 8:59, from 7:01 to 7:02. The time range is set from 7:01 to 7:03, ..., From 8:58 to 8:59, from 8:58 to 9:00, and from 8:59 to 9:00. Only combinations in which the start time is earlier than the end time are used. In this example, the number of combinations of (start time, end time) is {120 × 119/2}.

In the example of FIG. 9, the abnormality range determination unit 155 determines that the LOF calculated for the time range from 7:00 to 8:30 is the maximum among the plurality of set time ranges. Then, the abnormality range determination unit 155 determines that the period corresponding to the time range (the range from 7:00 to 8:30) is the abnormality range 815 (ANORMALLY FACTOR).
In the present embodiment, the abnormality range determination unit 155 always (for example, at regular intervals) has a LOF at each of a plurality of time scales (1 minute, 10 minutes, 30 minutes, 60 minutes in this example). It is determined whether or not the predetermined threshold value Q1 is exceeded, and when the LOF exceeds the predetermined threshold value Q1, as a subsequent process, the process of determining the maximum time range for the LOF in a plurality of time ranges is performed. Transition.

In this way, the abnormal range determination unit 155 is outliers (outliers) based on the degree to which the value to be analyzed (target value) deviates from the reference value (normal value) acquired in advance. It is determined whether or not it is an abnormal value).
The reference value (normal value) may be learned by using, for example, machine learning.

Here, the size or the number of the time scales initially set to detect the candidates for the abnormal period may be arbitrary.
Further, when it is determined that there is no abnormality in the time scale initially set for detecting the candidate of the abnormality period, the abnormality range determination unit 155 ends, for example, the abnormality detection process.
In addition, after a candidate for an abnormal period is detected, the time range when a time range is further set for the detected candidate for the abnormal period to determine the abnormal period is defined as the range or the number thereof. May be arbitrary.

Further, in the present embodiment, the case where the abnormal period is determined for the index (time scale, time range) related to the viewpoint of time (or time) is shown, but as another example, the viewpoint other than time (or time) Anomalous range may be determined for an index relating to (eg, scale of the viewpoint, range of the viewpoint). As a viewpoint other than the time (or time), for example, a viewpoint of an area, a viewpoint of a device type, or the like may be used.
Further, in the present embodiment, the case where the abnormal range is determined for one viewpoint (time (or time) viewpoint) is shown, but as another example, the abnormal range is determined for a combination of two or more different viewpoints. Judgment may be made.
Further, in the present embodiment, the distances between the clusters of a plurality of reference values and the target values are calculated and used, but as another configuration example, one representative in the cluster is used instead of the clusters of the plurality of reference values. Values may be used. As one representative value in the cluster, for example, the average value (or the center of gravity value) of the values included in the cluster may be used.

As the reference statistic data, the reference holiday statistic data shown in FIG. 6 may be used instead of the reference weekday statistic data.
Further, as another configuration example, a plurality of types (for example, weekdays and holidays) of reference statistic data and analysis target statistic data may be compared in parallel. In this case, for example, among a plurality of types of reference statistic data (in the case of the present embodiment, a plurality of types of clusters), the one closest to the statistic data to be analyzed (in this embodiment) at each timing when the comparison process is performed In the case of the present embodiment, a method of comparing (one cluster having the smallest distance) may be used.

[Example of processing performed in the statistic data processing device]
FIG. 10 is a diagram showing an example of a processing procedure for detecting the presence or absence of an abnormality performed in the statistic data processing device 12 according to the embodiment of the present invention.
In the present embodiment, the abnormality range determination unit 155 causes the statistic data comparison unit 154, the viewpoint setting unit 151, and the scale setting unit 152 to perform their respective processes to determine the abnormality range.
Note that this example is an example, and any other processing procedure may be used.

(Step S1)
The statistic data comparison unit 154 acquires data (reference data) as a reference for determining the abnormal range.
(Step S2)
The statistic data comparison unit 154 acquires the target data (target data) for determining the abnormal range.
(Step S3)
The viewpoint setting unit 151 sets the viewpoint used for the processing of determining the abnormal range.
(Step S4)
The scale setting unit 152 sets the scale used for the processing of determining the abnormal range.
(Step S5)
The statistic data comparison unit 154 compares the reference data with the target data using the set viewpoint and the set scale.
(Step S6)
The abnormality range determination unit 155 determines the abnormality range based on the result of the comparison process by the statistic data comparison unit 154, and thereby detects the presence or absence of an abnormality. The abnormality range determination unit 155 may change the conditions of the comparison processing to a plurality of types and perform the comparison processing by the statistic data comparison unit 154 to determine the abnormality range. The condition of the comparison process may be, for example, a condition relating to one or more of the viewpoint, scale, and range (time range, etc.).

[Summary of the first embodiment]
As described above, in the data processing system 1 according to the present embodiment, it is possible to detect an abnormality by performing multidimensional and multiscale data analysis in the statistic data processing device 12.
In the data structure of the statistic data group according to the present embodiment, for example, it is possible to monitor and detect an event such as the occurrence of an abnormality from a plurality of viewpoints and a plurality of scales, and various viewpoints and various scales. It is possible to monitor and detect scale events in parallel. In this case, in the data structure of the statistic data group according to the present embodiment, for example, it is possible to determine which viewpoint and scale the event has occurred. As a specific example, when an event occurs on a wide area scale, it can be estimated that the event is caused by a wide area, and when an event occurs on a specific area scale, the specific event occurs. It can be presumed that the cause is limited to the area of.
Here, in the present embodiment, the configuration for processing the statistic data for the values related to the terminal devices 11-1 to 11-n is set, but the configuration for processing the statistic data for other arbitrary values may be implemented. ..

(Second Embodiment)
[Data processing system]
FIG. 11 is a block diagram showing a schematic configuration of a data processing system 1001 according to an embodiment (second embodiment) of the present invention.
The data processing system 1001 includes n terminal devices 11-11 to 11-n, a statistic data processing device 1011, a database 1012, a unit statistic data generation device 1021, and a network 21. Here, the terminal devices 11-11 to 11-n and the network 21 are the same as those shown in FIG. 1, and are designated by the same reference numerals for convenience of explanation.
Further, the database 1012 has a function of storing data in the same manner as the database 13 shown in FIG.

The difference between the data processing system 1001 according to the present embodiment and the data processing system 1 shown in FIG. 1 will be described. In the example of FIG. 11, the point that the unit statistic data generation device 1021 is provided and a part of the processing performed by the statistic data processing device 1011 are different from the example of FIG.
The unit statistic data generation device 1021 has a function of performing all or a part of the processing that can be executed by the t-digest technique (see, for example, Non-Patent Document 1 and the like). In the present embodiment, the unit statistic data generation device 1021 uses the t-digest technique to unitize the data based on the data to be analyzed, the information for specifying the viewpoint, and the information for specifying the scale of each viewpoint. It has a function to generate statistic data.
As the unit statistic data generation device 1021, for example, the unit statistic data generation device 1021 may be managed by a person who manages the statistic data processing device 1011 or is provided by another person. The configuration may be used.

The statistic data processing device 1011 differs from the statistic data processing device 12 shown in FIG. 1 in that the unit statistic data generation device 1021 performs the processing performed by the unit statistic data generation device 1021. It is configured to request and receive the processing result.
In the present embodiment, the statistic data group generation unit 153 transmits a signal requesting a process for generating unit statistic data to the unit statistic data generation device 1021 by the communication unit 114 via the network 21. The signal contains information necessary to generate unit statistic data, such as data to be analyzed (or information that identifies it), information that identifies one or more viewpoints, and each. Contains information that identifies the scale of the viewpoint.
When the unit statistic data generation device 1021 receives the signal of such a request, the unit statistic data generation device 1021 generates the unit statistic data in response to the request, and transmits the signal including the generated unit statistic data via the network 21. , Transmit to the statistic data processing device 1011.
The statistic data group generation unit 153 uses (uses) the unit statistic data received from the unit statistic data generation device 1021 by the communication unit 114 to generate a statistic data group.

(Summary of the above embodiments)
As a configuration example, for at least one viewpoint and a plurality of scales for each viewpoint, statistic data for comparing reference data and comparison target data, which are statistic data based on data included in each combination of viewpoint and scale. Anomalies that determine a range that is considered to be abnormal among multiple ranges of viewpoints based on the results of comparison between the comparison unit (statistical data comparison unit 154 in the example of FIG. 2) and the statistic data comparison unit. It is a statistic data processing device (statistical data processing device 12, 1011 in the examples of FIGS. 1 and 11) including a range determination unit (abnormal range determination unit 155 in the example of FIG. 2).
As a configuration example, the abnormality range determination unit determines a range candidate (candidate for an abnormality range) from a viewpoint considered to have an abnormality based on the result of comparison by the statistic data comparison unit, and the abnormality range determination unit determines. Among the plurality of ranges included in the range candidates of the viewpoint, the range in which the LOF representing the difference between the reference data and the comparison target data is maximized is determined as the range considered to be abnormal.
As a configuration example, the statistic data comparison unit compares a plurality of reference data with the comparison target data.
As a configuration example, one or both of the reference data and the comparison target data are statistic data based on the data included in each combination of viewpoints and scales for a plurality of different viewpoints and a plurality of different scales for each viewpoint. Is obtained from the statistic data group (statistical data group 201 in the example of FIG. 3).
As a configuration example, the statistic data comparison unit performs reference data and comparison target data, which are statistic data based on data included in each combination of viewpoints and scales for at least one viewpoint and a plurality of scales for each viewpoint. A statistic data processing method (Fig.) In which the abnormality range determination unit determines a range considered to be abnormal among a plurality of ranges of viewpoints based on the result of comparison by the statistic data comparison unit. 1. In the example of FIG. 11, the processing method performed by the statistic data processing devices 12 and 1011).
As a configuration example, the statistic data comparison unit performs reference data and comparison target data, which are statistic data based on data included in each combination of viewpoints and scales for at least one viewpoint and a plurality of scales for each viewpoint. The computer is provided with a step of comparing with and a step of determining the range in which the abnormality range determination unit is considered to be abnormal among a plurality of ranges of the viewpoint based on the result of comparison by the statistics data comparison unit. It is a program to be executed (in the example of FIGS. 1 and 11, the computer constituting the statistical data processing devices 12 and 1011).

A program for realizing the functions of the devices (for example, the statistic data processing devices 12, 1011 and the like) according to the above-described embodiment is recorded on a computer-readable recording medium (storage medium), and the recording medium is recorded. The processing may be performed by loading the program recorded in the computer system into a computer system and executing the program.
The term "computer system" as used herein may include hardware such as an operating system or peripheral devices.
The "computer-readable recording medium" includes a flexible disk, a magneto-optical disk, a ROM (Read Only Memory), a writable non-volatile memory such as a flash memory, and a portable medium such as a DVD (Digital Versaille Disc). A storage device such as a hard disk built into a computer system.
Further, the "computer-readable recording medium" is a volatile memory inside a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line (for example, DRAM (for example, DRAM). It also includes those that hold the program for a certain period of time, such as Dynamic Random Access Memory)).
Further, the above program may be transmitted from a computer system in which this program is stored in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting a program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
Further, the above program may be for realizing a part of the above-mentioned functions. Further, the above program may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like within a range not deviating from the gist of the present invention are also included.

1,1001 ... Data processing system, 11-1 to 11-n ... Terminal device, 12, 1011 ... Statistics data processing device, 13, 1012 ... Database, 21 ... Network, 111 ... Input unit, 112 ... Output unit, 113 ... Storage unit, 114 ... Communication unit, 115 ... Control unit, 131 ... Data acquisition unit, 132 ... Statistics data processing unit, 133 ... Data output control unit, 151 ... Viewpoint setting unit, 152 ... Scale setting unit, 153 ... Statistics Quantitative data group generation unit, 154 ... Statistics data comparison unit, 155 ... Abnormal range determination unit, 201, 511-1 to 511-30, 522 ... Statistics data group, 211-213, 221-223, 313-1313, 321-23, 411-413, 421-423 ... Unit statistics data, 511 ... Reference data, 521 ... Data to be analyzed, 523 ... Comparison processing, 524 ... Abnormality detection processing, 525 ... Abnormal range, 611-615 , 621-625, 631-635, 811-814 ... Characteristics, 711 ... Reference value, 721-726 ... Target value, 1021 ... Unit statistics data generator, Q1 ... Threshold

Claims (6)

A statistic data comparison unit that compares reference data, which is statistic data based on data included in each combination of the viewpoint and the scale, with data to be compared with respect to at least one viewpoint and a plurality of scales for each viewpoint. When,
Based on the result of comparison by the statistic data comparison unit, an abnormality range determination unit that determines a range considered to have an abnormality among a plurality of ranges of the viewpoint, and an abnormality range determination unit.
Equipped with a,
The abnormality range determination unit determines a range candidate of the viewpoint considered to have an abnormality based on the result of comparison by the statistic data comparison unit.
The abnormality range determination unit considers that there is an abnormality in the range in which the LOF representing the difference between the reference data and the comparison target data is maximum among the plurality of ranges included in the range candidates of the viewpoint. Judge as a range,
Statistic data processor. The abnormal range determination unit
As a process of determining the range candidate of the viewpoint, which is considered to be abnormal, based on the result of comparison by the statistic data comparison unit.
Based on the result of comparing the reference data and the comparison target data for a plurality of different scales by the statistic data comparison unit, an abnormality is found in the entire range determined to be abnormal in one or more scales. Judging that it is the range candidate of the viewpoint that is considered to be
The statistic data processing apparatus according to claim 1. The statistic data comparison unit compares a plurality of the reference data with the comparison target data.
The statistic data processing apparatus according to any one of claims 1 and 2. One or both of the reference data and the comparison target data are statistics based on data included in each combination of the viewpoint and the scale for a plurality of different viewpoints and a plurality of different scales for each viewpoint. Obtained from a statistic data group that has quantitative data,
The statistic data processing apparatus according to any one of claims 1 to 3. The statistic data comparison unit selects reference data and comparison target data, which are statistic data based on the data included in each combination of the viewpoint and the scale, for at least one viewpoint and a plurality of scales for each viewpoint. Compare and
The abnormality range determination unit determines a range considered to be abnormal among the plurality of ranges of the viewpoint based on the result of comparison by the statistic data comparison unit.
It is a statistic data processing method
The abnormality range determination unit determines a range candidate of the viewpoint that is considered to have an abnormality based on the result of comparison by the statistic data comparison unit.
The abnormality range determination unit considers that there is an abnormality in the range in which the LOF representing the difference between the reference data and the comparison target data is maximum among the plurality of ranges included in the range candidates of the viewpoint. Judge as a range,
Statistic data processing method. The statistic data comparison unit selects reference data and comparison target data, which are statistic data based on the data included in each combination of the viewpoint and the scale, for at least one viewpoint and a plurality of scales for each viewpoint. Steps to compare and
A step in which the abnormality range determination unit determines a range considered to be abnormal among the plurality of ranges of the viewpoint based on the result of comparison by the statistic data comparison unit.
Is a program that allows a computer to execute
The abnormality range determination unit determines a range candidate of the viewpoint that is considered to have an abnormality based on the result of comparison by the statistic data comparison unit.
The abnormality range determination unit considers that there is an abnormality in the range in which the LOF representing the difference between the reference data and the comparison target data is maximum among the plurality of ranges included in the range candidates of the viewpoint. Judge as a range,
program.