JP2009009304A - Processing method for stream data, and stream data processing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To materialize processing for cutting and dividing a tuple on a stream into a partial tuple string of a phenomenon unit without taking trouble and cost in a stream data processing. <P>SOLUTION: In this stream data processing system 110, a tuple string on the stream is divided into the partial tuple strings based on a proximity relation system of time of a time-stamp imparted to each stream tuple on the stream, and a tuple set belonging to the latest partial tuple string in each the time is extracted as an analysis target in the time and is processed in a data analysis part 130 of the stream data processing system 110. Thereby, in the stream data processing, a set of observation data corresponding to each phenomenon can be extracted in just proportion. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a technique for processing time-series data continuously generated at a high rate in real time, and in particular, from a series of observation data continuously generated by monitoring the occurrence of a certain phenomenon in stream data processing. The present invention relates to a technique effective in processing for extracting a data set corresponding to a phenomenon based on proximity relations on a time axis.

  With the advent of the ubiquitous information society, time-series data that is continuously generated at a high rate, such as RFID (Radio Frequency IDentification) reading information, sensor network observation data, network monitoring, system monitoring, stock price monitoring, and traffic information, will increase. Therefore, a new service is desired that analyzes these time series data instantaneously and feeds back to the user.

  As a data processing method for constructing such a service, a technique called stream data processing has been proposed by a database-related academic society. If stream data processing is used, the data processing model based on the relational algebra used in the conventional relational database is also applied to the analysis of time series data, and the definition of data processing by the user is defined by SQL (Structured Query). (Language) can be described in a declarative search language. This greatly improves the efficiency of the service development work as described above. The principle and implementation method of stream data processing are disclosed in Non-Patent Document 1 (see Non-Patent Document 1).

  In the following, as a term representing time series data, a data series that continues infinitely is called a stream, and each data on the stream is called a stream tuple or simply a tuple. The stream tuple is always given a time stamp, and the stream tuple is arranged in ascending order of the time stamp on the stream.

  If stream data processing is used, various operations on a data set defined by relational algebra such as join operation, sum / difference / product set operation, aggregation operation, grouping processing, and deduplication are performed on the stream. It is possible.

  However, since these operations target a data set with a finite number of elements, in order to apply these operations to streams where tuples continue to arrive (that is, the elements of the set continue to increase infinitely). Therefore, it is necessary to always limit the set of tuples to be analyzed at each time when the processing is performed.

  As a method of limiting the tuple set to be analyzed from the stream, a sliding window method is proposed that always cuts out the tuple set to be analyzed at a certain point of time based on the order relation of tuples on the time axis. Has been.

  This method is disclosed in Non-Patent Document 1. Existing sliding windows are classified into two types: time specification windows and number specification windows.

  The time designation window limits the period during which each tuple is analyzed to a period starting from the time stamp of the tuple and ending at a time after a certain fixed time from the time stamp. The constant time can be specified at any time by the user. The number designation window simultaneously limits the tuples to be analyzed to the latest constant number. The number can be specified as an arbitrary number by the user.

  As another method for limiting the set of tuples to be analyzed from the stream, a method of inserting a delimiter signal in the storm to indicate the end of the set has been proposed. This method is disclosed in Non-Patent Document 2 (see Non-Patent Document 2).

As described above, in stream data processing, by providing a function to limit the set to be analyzed from time-series data, the amount of data change, calculation of moving average value, detection of data generation order, etc. It is possible to flexibly define important processes in the analysis of time series data.
B. Babcock, S.M. Babu, M.M. Data, R.A. Motwani and J.M. Widom, “Models and issues in data stream systems”, In Proc. of PODS 2002, pp. 1-16. (2002) T.A. Johnson, S.M. Muthukrishnan, V.M. Shkapenyuk and O.K. Spatcheck, "A Heartbeat Mechanism and its Application in Gigascope", In Proc. of VLDB 2005, pp. 1079-1088. (2005)

  In the future, with the development of technology for ubiquitous information devices, services such as efficient inspection work by reading RFID congestion, environmental monitoring by sensor networks, and advancement of telematics by increasing the functionality of in-vehicle sensors can be realized.

  In order to realize these services, data processing technology that analyzes phenomena occurring in the real world from observation data generated by various devices is essential. Stream data processing is considered a technique suitable for realizing such services.

  In order to realize the phenomenon analysis from the observation data generated by the device, it is essential to extract a partial data string corresponding to each phenomenon from the time series of observation data.

  The necessity of this processing will be described using an example of an incoming inspection using RFID congestion reading in FIG. In this case, when the corrugated cardboards 1303 and 1304 pass through the reading gate 1302, data as shown in Table 1310 representing information on each product packed in the corrugated cardboard is generated.

  Each row 1311 to 1317 is data relating to each product, and includes an id column indicating a product ID, a weight column indicating a product weight, and a date column indicating a manufacturing date. Data 1311 to 1313 are generated by reading the cardboard 1303, and data 1314-1317 are generated by reading the cardboard 1304. The stream data processing system 110 receives the data, calculates the number of packed products, the total weight, and the production period for each cardboard, and notifies an inspector or the like.

  The timing of work by the inspector and the timing at which the read data is generated are shown in a timing window 1320 below the table 1310. When the inspector reads the cardboard 1303 that becomes the cardboard 1 at 9: 15′02 and passes it through the gate 1302, the data 1321 of each product packed in the cardboard is 9: 15′02 to 9: 15′04. Generated in between.

  The inspector confirms the result of totaling the data by the stream data processing system 110 at 9: 15'06. The same applies to cardboard 2 (1304). In this process, the stream data processing system needs to generate a total result for each cardboard.

  Here, when the reading of each cardboard is regarded as one phenomenon, the stream data processing needs to extract a partial data string corresponding to each cardboard (that is, one phenomenon) from the time series product data.

  As another example, an example of environmental monitoring by the sensor network of FIG. 7 is shown. Physical quantities such as voice, light quantity, temperature, humidity, atmospheric pressure, wind pressure, wind speed, and vibration are measured by sensors 1401 (IDs S01 to S23) arranged in a space, and an earthquake, a fire is detected based on the measurement data. Analyzes phenomena such as explosions and accidents.

  Regardless of the physical quantity to be observed, the type of phenomenon, and the combination of both, a sensor network can be applied to such a phenomenon analysis. Here, phenomena 1402, 1403, and 1404 occur at times of 10:25, 10:28, and 10:32, and physical quantities resulting from these phenomena are surrounded by lines 1405, 1406, and 1407, respectively. The detected sensor group detects and measures.

  Table 1410 shows the measurement data. Each row 1411 to 1422 is each measurement data, and includes an id column indicating the ID of the measured sensor and a value column indicating the measurement value. Data 1411 to 1414 is generated by the phenomenon 1402, data 1415 to 1417 is generated by the phenomenon 1403, and data 1418 to 1422 is generated by the phenomenon 1404. The closer the sensor is to the place where the phenomenon occurs, the larger the measured value.

  Window 1430 is a diagram showing the occurrence timing of the phenomenon and the timing at which measurement data is generated by the sensor. In order to realize analysis for each phenomenon from such a data series, the stream data processing needs to extract a partial data string corresponding to each phenomenon from time-series measurement data.

  As described above, in order to apply the stream data processing to the phenomenon analysis based on the observation data generated by the ubiquitous device, it is essential to process the tuples on the stream in units of phenomena.

  On the other hand, in the example of FIG. 4, it is impossible to realize such processing in the time designation window and the number designation window, which are the existing sliding window methods disclosed in Non-Patent Document 1. Show.

  The time designation window 1330 indicates a case where the time designation window is applied to the stream indicated in the timing window 1320. The window 1331 represents a period in which each product data 1321 is an analysis target by a line segment extending from the black circle to the right direction (future direction on the time axis).

  All line segments have the same length as the analysis target period 1333 specified by the user. Window 1332 shows the result of calculating the total number of products from the stream tuple having the analysis target period shown in window 1331. At 9: 15'06, when the inspector confirms the result, the product number 3 of the cardboard 1303 is correctly confirmed.

  On the other hand, when the inspector confirms the result at 9: 15'18, since the reading data of the cardboard 1303 still remains as the analysis target period, an incorrect value 7 different from the product number 4 of the cardboard 1304 is obtained. Will be confirmed.

  A window 1340 shows a case where a time designation window is applied to the stream shown in the timing window 1320 by shortening the analysis target period. 1343 indicates the length of the analysis target period. A window 1341 shows a period in which each data is an analysis target, and a window 1342 shows a result of calculating the total number of products.

  When the data analysis target period is short, the duration of the result calculated therefrom is also short, so that the inspector confirms the wrong value at either 9: 15'06 or 9: 15'18. Become.

  Since the work speed of the inspector is not constant, it is impossible to completely avoid the above-described problems even if the analysis target period is long or short.

  The number designation window 1350 shows a case where three simultaneous analysis target numbers are applied to the stream shown in the timing window 1320 as the number designation window. A window 1351 shows a period in which each data is an analysis target, and a window 1352 shows a result of calculating the total number of products. In the number designation window, since only the latest three pieces of data always remain as analysis targets, the total number of products is 3, regardless of the number of products included in the cardboard, and a correct result cannot be obtained.

  Similarly, in the example of the sensor network of FIG. 7, the occurrence interval of phenomena and the number of measurement data corresponding to each phenomenon are indefinite. Therefore, the measurement data is divided into phenomenon units in the time designation window and the number designation window. Is impossible.

  In addition, the method disclosed in Non-Patent Document 2 for inserting a delimiter signal into a stream has a problem in the process of dividing time-series data into phenomenon units using the examples of FIGS. 4 and 7. Show.

  In the example of FIG. 4, when a method for inserting a delimiter signal is applied, a partition bar is arranged between cardboards, and when the partition bar read data is inserted, a delimiter signal is inserted into the stream so that RFID read data is in cardboard units. It is possible to carve out with.

  Alternatively, a cardboard break can be detected by adding an infrared sensor to the reading gate 1302. Alternatively, each time an inspector passes one cardboard through the reading gate 1302, it is possible to clearly indicate the separation of the cardboard by an operation such as pressing a reading end button.

  However, in any of these methods, extra labor and cost such as arrangement of partition bars, addition of an infrared sensor, and operation of a reading end button are required.

  In addition, in the example of FIG. 7, in order to insert a delimiter signal indicating the end of the measurement data string corresponding to one phenomenon into the stream, when one phenomenon occurs, all the sensors that detect the phenomenon A process for determining that the measurement data has been generated is necessary.

  However, in the sensor network, since each sensor arranged in a distributed manner generates measurement data independently, it is difficult to realize such a determination process.

  As described above, in order to apply stream data processing to phenomenon analysis based on observation data generated by a ubiquitous device, processing for separating tuples on a stream by phenomenon unit is indispensable. Such a process cannot be realized in the specified window or number specified window, and even when the delimiter signal insertion method is used, it takes extra time for the user to implement, it is costly, or the implementation itself is difficult There was a problem that there was.

  An object of the present invention is to provide a technique capable of realizing, in stream data processing, processing for dividing tuples on a stream into partial tuple sequences in units of phenomena without requiring user effort and cost. is there.

  The above and other objects and novel features of the present invention will be apparent from the description of this specification and the accompanying drawings.

  Of the inventions disclosed in the present application, the outline of typical ones will be briefly described as follows.

  The outline of the present invention will be briefly described below.

  The present invention divides a tuple sequence on a stream into partial tuple sequences based on a time proximity relation system of time stamps assigned to each stream tuple on the stream, and belongs to the latest partial tuple sequence at each time. The tuple set is passed to the data analysis unit of the stream data processing system as an analysis target at the time.

  Hereinafter, the partial tuple sequence is referred to as a data cluster, and a condition related to the time stamp of the tuple referred to when configuring the data cluster is referred to as a cluster configuration condition.

More specifically, a stream data processing system is provided that includes an analysis target data cutout unit configured by the following processing at a data input port of the system.
(A) Processing for extracting a cluster configuration condition from a data processing definition by a user.
(B) When a new tuple is input to the system, the time stamp of the new tuple is checked based on the cluster configuration condition, and the new tuple is added to the data cluster currently being analyzed. The process of determining whether or not to be added.
(C) If the new tuple is not added to the data cluster that is the analysis target, the data analysis unit is notified so that all tuples belonging to the data cluster are excluded from the analysis target. A process of defining a data cluster as an analysis target and adding the new tuple to the data cluster.
(D) Processing for passing the new tuple as an analysis target to the data analysis unit.

  The determination based on the cluster configuration condition is specifically one of the following two types.

  (1) When a new tuple arrives, the time stamp of the new tuple is compared with the time stamp of the oldest tuple among the tuples belonging to the data cluster being analyzed at that time. If the stamp difference is within the time threshold specified by the user, the new tuple is added to the data cluster; otherwise, the process of (c) is performed.

  (2) When a new tuple arrives, the time stamp of the new tuple is compared with the time stamp of the latest tuple among the tuples belonging to the data cluster being analyzed at that time. If the difference is within the time threshold specified by the user, the new tuple is added to the data cluster; otherwise, the process of (c) is performed.

  As described above, in the time-series data, a group of data sets whose generation times are close to each other is cut out as an analysis target of stream data processing.

  Among the inventions disclosed in the present application, effects obtained by typical ones will be briefly described as follows.

  (1) In stream data processing, a set of observation data corresponding to each phenomenon can be extracted without excess or deficiency.

  (2) Further, since the extraction process can be performed based only on the data generation time, it is possible to eliminate the labor and cost for generating the data delimiter signal.

  (3) Furthermore, since the data generation sources are distributed in plural, the extraction process can be realized even when it is difficult to generate a data delimiter signal.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that components having the same function are denoted by the same reference symbols throughout the drawings for describing the embodiment, and the repetitive description thereof will be omitted.

  FIG. 1 is a block diagram showing a configuration example of a stream data processing system according to an embodiment of the present invention, FIG. 2 is a flowchart showing an example of data cluster cutout processing by the stream data processing system of FIG. 1, and FIG. 4 is an explanatory diagram of stream data processing by the stream data processing system of FIG. 1, FIG. 4 is an explanatory diagram showing an example of an incoming inspection work process by RFID congestion reading using the stream data processing system of FIG. 1, and FIG. FIG. 6 is an explanatory diagram showing another example of stream data processing by the stream data processing system of FIG. 1, and FIG. 7 is a diagram illustrating one example of the present invention. FIG. 8 is an explanatory diagram showing an example of data processing by the sensor network according to the embodiment. FIG. 9 is a block diagram showing a configuration example of a stream data processing system that implements data cutout processing according to the form of FIG. 8, FIG. 9 is a flowchart showing an example of data cluster cutout processing by the stream data processing system of FIG. 8, and FIG. FIG. 11 is an explanatory diagram illustrating an example of network monitoring according to an embodiment of the present invention, and FIG. 12 is a diagram illustrating the network monitoring of FIG. 11 using the sensor network using the stream data processing system. FIG. 13 is a flowchart showing an example of the data cutout process by the network monitoring in FIG. 12, FIG. 14 is a block diagram showing an example of the configuration of the stream data dividing system according to the embodiment of the present invention, FIG. 15 shows the network of FIG. Is an explanatory diagram showing a stream data processing example by click monitoring.

  In this embodiment, as shown in FIG. 1, the stream data processing system 110 includes a data processing definition registration user interface 111 that accepts a data processing definition 101 defined by a user, an analysis target data cutout unit 120, and a data analysis unit 130. Consists of

  The analysis target data cutout unit 120 executes processing for extracting the analysis target data cluster from the input stream 141 according to the cluster configuration condition included in the definition 101. The data analysis unit 130 executes analysis processing on the data cluster in accordance with the definition 101, and outputs the result streams 143 and 144 to a client or the like.

  The data processing definition registration user interface 111 interprets the received definition 101, extracts the definition of the cluster configuration condition from the definition 101, and notifies the analysis target data cutout unit 120 of it. Further, the data analysis processing execution tree 131 is configured according to the contents of the definition 101 and arranged on the analysis unit 130.

  In the present embodiment, the description “[CLUSTER WITHIN 3 seconds]” in the definition 101 is the definition of the cluster configuration condition. “WITHIN” means the type of the cluster configuration condition (1).

  When the character string in the same part is “INTERVAL”, it means that the type is the cluster configuration condition (2). Hereinafter, (1) is referred to as a WITHIN condition, and (2) is referred to as an INTERVAL condition. “3 seconds” means that the time threshold is 3 seconds. Note that any time can be defined as the time threshold.

  Hereinafter, the configuration and operation of the analysis target data cutout unit 120 will be described with reference to FIGS. 1 and 2. However, FIG. 2 shows an operation of the analysis target data determination unit 121 that realizes the WITHIN condition.

  The analysis target data cutout unit 120 first stores the time threshold value notified from the data processing definition registration user interface 111 in the time threshold value storage area 128. Furthermore, according to the initialization process S501, the time Time. MIN is stored, and -1 is stored in the cluster number storage area 127.

  Time. MIN is a time determined to be in the past even if compared with any actual time. After the initialization S501, the process proceeds to a new input data reception waiting process S502.

  When receiving the input stream 141 that is new input data from outside the system 110, the analysis target data determination unit 121 stores the time stamp of the data in the input data time storage area 122, and compares the data time according to processing S503. Using the unit 123, the time stamp of the new input data stored in the area 122 is compared with the cluster time stored in the cluster time storage area 126.

  If the comparison shows that the time stamp of the new input data exceeds the cluster time, the time obtained by adding the time threshold to the time stamp of the new input data is set to the update value of the cluster time in accordance with step S504. Is stored in the cluster time storage area 126.

  Further, according to the process S505, the cluster number currently stored in the cluster number storage area 127 is notified to the non-analysis target data deletion unit 125. Subsequently, in accordance with the processing S506, the cluster number stored in the cluster number storage area 127 is incremented. When input data is first received after the initialization process S501, the processes S504 to S506 are always executed. After these processes, the process proceeds to process S507.

  On the other hand, if the time stamp of the new input data is earlier than the cluster time based on the comparison in step S503, the process directly proceeds to step S507. In this process S507, the cluster number currently stored in the cluster number storage area 127 is assigned to the new input data and passed to the analysis target data insertion unit 124. When the process S507 ends, the process proceeds to the new input data reception wait process S502 again.

  The data insertion unit 124 inserts the new input data into the analysis target data buffer 132 held by the data analysis unit 130. On the other hand, the data deletion unit 125 deletes all data to which the notified cluster number is assigned from the buffer 132.

  Each time data insertion / deletion occurs in the buffer 132, the data analysis unit 130 performs an analysis process on a set of data 142 existing on the buffer 132 based on the data analysis process execution tree 131. Send the result to the outside. A method for realizing the data analysis unit 130 is disclosed in Non-Patent Document 1, for example.

  Hereinafter, the results when the stream data processing method realized by the configuration of FIG. 1 and the operation of FIG. 2 is applied to the incoming inspection work by RFID congestion reading shown in FIG. 4 will be described with reference to FIG.

  FIG. 3 shows the result of applying the processing method when the stream data processing system 110 receives the data shown in Table 1310 (FIG. 4) at the timing shown in the timing window 1320 and processes it according to the data processing definition 101. Is shown.

  Definition 101 means that the data (rfid) in Table 1310 is divided into data clusters according to the WITHIN condition with a time threshold of 3 seconds, and the number of packed products, the total weight, and the production period are calculated for each data cluster. .

  In FIG. 3, a window 601 indicates the analysis target period of each data, a window 603 indicates the length of the time threshold, and the length of the time threshold is, for example, about 3 seconds. is there.

  When data a (1311) is read at 9: 15'02, a data cluster is newly generated. The cluster number is 0. At this time, the cluster time is determined to be 9: 15'05 obtained by adding the time threshold 3 seconds to the time stamp 9: 15'02 of the data a.

  Since the time stamps of the data a to c (1311 to 1313) are before the cluster time 9: 15'05, they are added to the same data cluster 0 as the data a (hereinafter, data to which the cluster number N is assigned). The data cluster consisting of is denoted as data cluster N).

  Thereafter, when the data d (1314) is received at 9: 15′12, the cluster time is exceeded, and therefore the analysis target period of the data a to c to which the cluster number 0 is given at this time is determined. finish.

  On the other hand, a new data cluster 1 is generated and data d is added to the cluster. The cluster time is determined to be 9: 15'15, which is obtained by adding the time threshold value 3 seconds to the time stamp 9: 15'12 of the data d. Since the time stamps of the data e to g (1315 to 1317) are before the cluster time 9: 15'15, they are added to the same data cluster 1 as the data d.

  A window 602 indicates the length of the time threshold, and shows the result of calculating the total number of products from the stream tuple having the analysis target period of each data. At 9: 15′06 and 9: 15′18, when the inspector confirms the result, the number of products of cardboard 1 and the number of products of cardboard 2 are both confirmed correctly. . Correct values are also confirmed for the total weight and the production period.

  Table 610 shows the processing result data. Data 611 to 617 are output in response to the input of data 1311 to 1317, respectively. The inspector confirms the data 613 at 9: 15'06 and the data 617 at 9: 15'18. These correctly represent the total values of cardboard 1 and cardboard 2, respectively.

  As described above, by using the configuration of FIG. 1 and the operation of FIG. 2, it is possible to realize a correct phenomenon analysis without imposing extra effort and cost on the user.

  Next, the operation of the analysis target data determination unit 121 that realizes the INTERVAL condition will be described with reference to FIG.

  5 is exactly the same as the operation of FIG. 2 except for the execution position of step S504. Since the processing S504 is in the position, the cluster time is always the time stamp of the new input data + the time threshold value regardless of whether or not the new input data is added to the current analysis target data cluster. Updated to value.

  As a result, when new input data arrives, a determination is always made as to whether it is within the time threshold compared with the time stamp of the latest data in the analysis target data cluster.

  Hereinafter, the results when the stream data processing method realized by the configuration of FIG. 1 and the operation of FIG. 5 is applied to an incoming inspection work by reading RFID congestion will be described with reference to FIG.

  In this case, when the stream data processing system 110 receives data at the timing indicated by the timing window 810 in FIG. 6 and processes the data according to the data processing definition 801, the result of applying the processing method is shown.

  The definition 801 means that the data (rfid) is divided into data clusters according to the INTERVAL condition with a time threshold value of 4 seconds, and the number of packed products, the total weight, and the production period are calculated for each data cluster.

  When data a is read at 9: 15'02, a data cluster is newly generated. The cluster number is 0. At this time, the cluster time is determined to be 9: 15'06, which is obtained by adding the time threshold value 823 (4 seconds) to the time stamp 9: 15'02 of the data a.

  Since the time stamp 9: 15'03 of the data b is before the cluster time 9: 15'06, it is added to the same data cluster 0 as the data a. At the same time, the cluster time is updated to 9: 15'07 by adding the time threshold 4 seconds to the time stamp 9: 15'03 of the data b.

  Similarly, since the data c to e are continuous within the time threshold of 4 seconds, they are all added to the data cluster 0. The cluster time after the input of data e is updated to 9: 15'11 by adding the time threshold of 4 seconds to the time stamp 9: 15'07 of data e.

  Thereafter, when the data f is received at 9: 15′13, the cluster time is exceeded, and therefore the analysis target period (analysis target period) of the data a to e to which the cluster number 0 at this time is given. Is completed).

  On the other hand, a new data cluster 1 is generated and data f is added to the cluster. The cluster time is determined to be 9: 15'17, which is obtained by adding 4 seconds of the time threshold value 823 to the time stamp 9: 15'13 of the data f. Since the data g to i are continuous within 4 seconds of the time threshold value 823, they are all added to the data cluster 1.

  At 9: 15′09 and 9: 15′19, when the inspector confirms the result 822 of calculating the total number of products from the stream tuple having the analysis target period 821, the number of products of the cardboard 1 is 5, And the product number 4 of the corrugated cardboard 2 is confirmed correctly. Correct values are also confirmed for the total weight and the production period.

  As described above, by using the configuration of FIG. 1 and the operation of FIG. 5, it is possible to realize a correct phenomenon analysis without imposing extra effort and cost on the user. In addition, since it is difficult for a large cardboard to finish passing through the gate within a certain time threshold, the data corresponding to one cardboard can be correctly collected under the WITHIN condition realized by the operation of FIG. Can not.

  Even in such a case, while the RFID read data is generated without interruption, it can be considered that one cardboard is passing, so by using the INTERVAL condition realized by the operation of FIG. The data set generated during the period can be extracted as data corresponding to one cardboard.

  As in this example, this method can be applied when data corresponding to one phenomenon does not fit in a certain period of time but is continuously generated.

  Next, the contents of data processing assumed in the example of the sensor network of FIG. 7 are shown.

  In the example of FIG. 7, the position where the occurrence frequency of the phenomenon is high is estimated based on the magnitude of the measurement value of each sensor. In the figure, it is shown that the occurrence frequency is high around the sensor S17.

  As the estimation processing, extracting the data indicating the largest value in the data cluster corresponding to each phenomenon and extracting the sensor ID that measured the value is the data processing assumed in this example.

  However, in such estimation processing, even if data processing is performed only on a measurement data set related to one phenomenon, an accurate value cannot be obtained, and analysis of data related to a plurality of phenomena is necessary. On the other hand, a data cut-out method for simultaneously analyzing a plurality of data clusters is provided.

  Hereinafter, a configuration for realizing the data cutout method will be described with reference to FIG.

  The description “[CLUSTER 2 WITHIN 3 seconds]” in the data processing definition 301 is the definition of the extraction method, and the description includes the latest two data clusters extracted under the WITHIN condition with a time threshold of 3 seconds. , Which means that it is subject to analysis at the same time.

  The data processing definition registration user interface 111 interprets the number following the “CLUSTER” character string as the number of data clusters to be analyzed simultaneously, and notifies the analysis target data cutout unit 120 of the number.

  The analysis target data cutout unit 120 stores the value in the simultaneous target cluster number storage area 311. This configuration is the same as that of the stream data processing system 110 shown in FIG. 1 except that the area 311 is added.

  The operation for realizing the data cutout method will be described below with reference to FIG.

  In this case, the process S901 is the same as the operation in FIG. 2 except that the process S901 is positioned instead of the process S505 in FIG. Process S505 is to notify the non-analysis data deletion unit 125 of the cluster number of the data cluster that was the analysis target at that time when it is determined that the new input data belongs to the new data cluster.

  On the other hand, the processing S901 in FIG. 9 notifies the data deletion unit 125 of a value obtained by adding 1 to the value obtained by subtracting the number of simultaneous target clusters from the current cluster number as the cluster number of the data cluster to be deleted. . As a result of the external processing, the number of simultaneous target cluster data clusters including newly generated data is held as the analysis target. If the number of simultaneous target clusters is 1, the process S901 is identical to the process S505.

  The above is the description regarding the data extraction method in which the designation of the number of simultaneous target clusters is added to the WITHIN condition.

  It is self-evident that a data cutout method in which designation of the number of simultaneous target clusters is added can also be realized in the same manner for the INTERVAL condition. The configuration for realizing the cutout method is the same as that in FIG. 8, and the operation can be realized by replacing the process S505 in FIG. 5 with a process S901.

  Hereinafter, the results when the stream data processing method realized by the configuration of FIG. 8 and the operation of FIG. 9 is applied to the environment monitoring by the sensor network shown in FIG. 7 will be described using FIG.

  FIG. 10 shows the result of applying the processing method when the stream data processing system 110 receives the data shown in Table 1410 at the timing of the window 1430 in FIG. 10 and processes the data according to the data processing definitions 301 and 1001. Show.

  Definition 301 means that data (sensor) 1410 is divided into data clusters according to a WITHIN condition with a time threshold of 3 seconds, and the latest two data clusters are simultaneously analyzed, and the measured values are added for each sensor. To do. The definition 1001 means that the ID of the sensor having the maximum added value is extracted. A window 1010 indicates an analysis target period of each data, and the window 1010 indicates the lengths of the time threshold values 1011, 1012, and 1013 (for example, 3 seconds).

  Data 1411 to 1414 belong to data cluster 0, data 1415 to 1417 belong to data cluster 1, and data 1418 to 1422 belong to data cluster 2, respectively.

  When data 1411 to 1414 are input, if data 1415 is input at 10:28, the condition branch of processing S503 falls to No, and processing S504, S901, and S506 are executed.

  However, in the process S901, nothing is executed because the cluster number −1 obtained by subtracting the simultaneous target cluster number 2 from the cluster number 0 at the time and adding 1 is negative. Therefore, data belonging to the data cluster 0 remains as an analysis target.

  On the other hand, if the data 1418 is input at the time of 10:32 in the state where the data 1411 to 1417 are input, the condition branch of the process S503 falls to No, and the processes S504, S901, and S506 are executed.

  At this time, the cluster number obtained by subtracting the number 2 of simultaneously targeted clusters from the cluster number 1 at that time and adding 1 is 0, so that the data 1411 to 1414 belonging to the data cluster 0 are excluded from the analysis target.

  On the other hand, a new data cluster 2 is generated and data 1418 is added. As a result, data clusters 1 and 2 having two simultaneous target clusters are extracted as analysis targets.

  A table 1020 shows processing result data according to the definition 301. Data 1021 to 1032 are calculated by inputting data 1411 to 1422, respectively. In the data 1026, a value obtained by adding the measurement values of the measurement data by the sensor S17 belonging to the data clusters 0 and 1 is the value in the value_sum column.

  Similarly, the data 1027 is obtained by adding the measurement values of the measurement data by the sensor S18 belonging to the data clusters 0 and 1, and the data 1032 is the measurement value of the measurement data by the sensor S17 belonging to the data clusters 1 and 2. Are added to the value_sum column.

  A table 1040 shows processing result data according to the definition 1001. By calculating data 1021 to 1032, data 1041 to 1052 are output, respectively. Data 1044, 1047, and 1052 are analysis results after occurrence of the phenomena 1402, 1403, and 1404, respectively. The data 1047 and 1052 that are the results of analyzing two phenomena include the sensor IDS 17 that is an assumed answer.

  As described above, by using the configuration of FIG. 8 and the operation of FIG. 9, a data extraction method for analyzing a plurality of data clusters is realized, and analysis for a plurality of phenomena is possible. The data cutout method can be used for the purpose of improving analysis accuracy.

  Next, an example of network monitoring will be described with reference to FIG.

  As shown in the figure, the network monitoring includes a network router 1501 and computers 1502, 1503 and 1504.

  As the packet passes through the router 1501, data as shown in Table 1510 is generated. Each row 1511 to 1522 is data relating to each packet, and includes a src column indicating a transmission source, a dst column indicating a transmission destination, and a length column indicating a data length.

  The stream data processing system 110 receives the data, extracts a plurality of packets whose generation times are close to each other as one message, and calculates a message length for each message.

  A timing window 1530 in FIG. 11 is a diagram showing a timing at which information about a packet is generated. As described above, since data related to packets sent from a plurality of computers 1502, 1503, and 1504 are mixed in one stream, it is possible to realize the extraction processing based only on the proximity relationship of the time stamps of the data. Can not.

  On the other hand, if attention is paid to an individual computer as a transmission source, a packet sequence constituting one message is continuously transmitted, and the transmission interval between different messages is sufficiently longer than that. By applying the condition, it is possible to cut out data related to the packet in units of messages.

  Windows 1540, 1550, and 1560 in FIG. 11 are diagrams in which only data related to packets having computers 1502, 1503, and 1504 as transmission sources are extracted from the data series shown in the timing window 1530, respectively.

  By classifying in this way, data relating to packets originating from the computer 1502 is sent to the data clusters 1541 and 1542, and data relating to packets originating from the computer 1503 is sent to the data clusters 1551 and 1552. Data relating to the original packet can be divided into data clusters 1561. The classification process can be realized based on the src column of the data 1511 to 1522.

  Hereinafter, a configuration for realizing the data cutout method will be described with reference to FIG.

  The description “[PARTITION BY src CLUSTER INTERVAL 1 second]” in the data processing definition 401 is the definition of the extraction method. The description groups data based on the value identity of the src column, This means that a data cluster cut out under the INTERVAL condition with a time threshold of 1 second is to be analyzed.

  The data processing definition registration user interface 111 interprets the character string that follows the “PARTITION BY” character string as the name of a column used for data grouping, and notifies the analysis target data cutout unit 120. Hereinafter, this column is referred to as a group identifier.

  This configuration is obtained by extending the configuration of the stream data processing system 110 shown in FIG. 1 so as to correspond to the data extraction processing for each group. The cluster time storage area 126 and the cluster number storage area 127 are necessary for each group. Both areas are collectively referred to as a group-specific storage area.

  There should be only one storage area for each group, and only one for each group. In order to realize this management, a group-specific storage area management table 411 that holds the correspondence between group identifier values and group-specific storage areas is provided.

  The group-specific storage areas 412 to 414 are used to separate data relating to packets of the transmission source computers 1502, 1503, and 1504 into data clusters. The management table 411 holds the correspondence between the group identifier values A to C and the group storage areas 412 to 414.

  Hereinafter, the operation for realizing the data cutout method will be described with reference to FIG.

  In the operation of FIG. 13, processing S1101 and S1102 are added to the operation of FIG. 2, and processing S501 and S505 are replaced with processing S501G and S505G, respectively. This operation starts from the new input data reception waiting process S502.

  When new input data is received, the group identifier corresponding to the value is extracted from the group-specific storage area management table 411 by referring to the group identifier value of the data in accordance with step S1101.

  Furthermore, if the group-specific storage area is not extracted in accordance with process S1102, the process proceeds to process S501G. The process S501G newly creates a storage area for each group, registers the correspondence between the group identifier value and the storage area for each group in the management table 411, and initializes the storage area for each group. The initialization process is the same as the process S501.

  Using the group-specific storage area obtained by the above processing, the processing in steps S503 to S507 is executed. However, in process S505, the non-analysis target data deletion unit is notified of the cluster number of the data cluster that has not been analyzed and the value of the group identifier.

  The above is the description regarding the data cutout method in which the grouping designation is added to the WITHIN condition. It is self-evident that a data extraction method with grouping designation added can be realized in the same manner for the INTERVAL condition.

  The configuration for realizing the clipping method is the same as that in FIG. 12, and the operation can be realized by moving the process S504 in FIG. 13 immediately before the process S507.

  Hereinafter, the results when the stream data processing method realized by the configuration of FIG. 12 and the operation of FIG. 13 is applied to the network monitoring shown in FIG. 11 will be described using FIG.

  FIG. 15 shows a result of applying the processing method when the stream data processing system 110 receives the data shown in Table 1510 at the timing shown in the timing window 1530 and processes the data according to the data processing definition 401. .

  Definition 401 divides data (nlog) 1510 into groups based on the value identity of the src column, and for each group, analyzes data clusters that are cut out under the INTERVAL condition with a time threshold value of 1 second, and is transmitted. This means that the packet length is added for each original computer.

  Windows 1210, 1220, and 1230 indicate analysis target periods of the data A, B, and C in the src column, respectively. The window 1210 shows the length of the time threshold value 1201 (for example, 1 second).

  A table 1240 shows processing result data according to the definition 401. Data 1241 to 1252 are calculated by inputting the data 1511 to 1522, respectively. Data 1244, 1251, 1245, 1252, and 1248 indicate message lengths calculated from the data clusters 1541, 1542, 1551, 1552, and 1561, respectively.

  As described above, by using the configuration of FIG. 12 and the operation of FIG. 13, even when a plurality of phenomena are mixed in one stream, it is possible to extract data in units of phenomena after separating the data strings. Become.

  Next, the configuration of a stream data division system that divides time-series data into data clusters, assigns a cluster number to each data, and stores the data in a database will be described with reference to FIG.

  The configuration shown in FIG. 14 is a modification of the analysis target data cutout unit 120 in the configuration of FIG. The stream data division system 210 includes a cluster division method definition registration user interface 211 that accepts a cluster division method definition 201 by a user, and a cluster determination unit 212.

  The configuration and operation of the cluster determination unit 212 are the same as the configuration of FIG. 1 and the operation of FIG. However, the data insertion unit 124C inserts a set of the data 221 and the cluster number 222 into the relation table on the database 220. Further, the non-analysis target data deletion unit 125 does not exist.

  The above is a description of the case where the WITHIN condition is specified as the cluster dividing method. However, it is obvious that the INTERVAL condition and the grouping can be specified by switching the configuration and operation of the cluster determining unit. It is.

  With the above configuration, time series data can be divided into phenomenon units and stored in a database. Even if the data is stored in the database together with the time stamp, it is difficult to define the processing for dividing into the phenomenon units by the post-mortem analysis based on the time stamp. On the other hand, by applying this configuration, such processing becomes unnecessary.

  Further, the configuration shown in FIG. 14 is arranged at the entrance of the delimiter signal system shown in Non-Patent Document 2, and a function for the data insertion unit 124C to send a delimiter signal together when the cluster number is incremented is added. Thus, the present invention can be applied as an extended function of the delimiter signal system.

  As mentioned above, the invention made by the present inventor has been specifically described based on the embodiment. However, the present invention is not limited to the embodiment, and various modifications can be made without departing from the scope of the invention. Needless to say.

  The present invention relates to a stream data processing technology for processing time-series data continuously generated at a high rate, such as RFID reading information, sensor network observation data, network monitoring, system monitoring, stock price monitoring, and traffic information, in real time. For sliding window processing that extracts data to be analyzed from series data at a certain point in time, in particular, from a series of observation data that is continuously generated by monitoring the occurrence of a phenomenon, a data set corresponding to each phenomenon is plotted on a time axis. It is suitable for the technique of extracting based on the proximity relationship above.

It is a block diagram which shows the structural example of the stream data processing system by one embodiment of this invention. It is a flowchart which shows the example of a data cluster cut-out process by the stream data processing system of FIG. It is explanatory drawing of the stream data processing by the stream data processing system of FIG. It is explanatory drawing which shows an example of the goods inspection work process by the congestion reading of RFID using the stream data processing system of FIG. 7 is a flowchart illustrating another example of a data cluster cutout process performed by the stream data processing system of FIG. 1. It is explanatory drawing which shows the other example of the stream data processing by the stream data processing system of FIG. It is explanatory drawing which shows an example of the data processing by the sensor network by one embodiment of this invention. It is a block diagram which shows the structural example of the stream data processing system which implement | achieves the data cut-out process by one embodiment of this invention. It is a flowchart which shows an example of the data cluster cut-out process by the stream data processing system of FIG. It is explanatory drawing of the stream data processing by the sensor network using the stream data processing system of FIG. It is explanatory drawing which shows an example of the network monitoring by one embodiment of this invention. It is explanatory drawing of the data cut-out process using the network monitoring of FIG. It is a flowchart which shows an example of the data cut-out process by the network monitoring of FIG. FIG. 14 is a block diagram showing a configuration example of a stream data division system according to an embodiment of the present invention. It is explanatory drawing which shows the example of stream data processing by the network monitoring of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 110 Stream data processing system 111 Data processing definition registration user interface 120 Analysis object data extraction part 121 Analysis object data determination part 122 Input data time storage area 123 Data time comparison part 124 Analysis object data insertion part 124C Data insertion part 125 Data not to be analyzed Deletion unit 126 Cluster time storage area 127 Cluster number storage area 128 Time threshold storage area 130 Data analysis unit 131 Data analysis processing execution tree 132 Analysis target data buffer 141 Input stream 143, 144 Result stream 210 Stream data division system 211 Cluster division Method definition registration user interface 212 Cluster determination unit 214C Data insertion unit 220 Database 411 Group-specific storage area management tables 412 to 414 groups Storage areas 601 to 603 window 810 timing window 1320 timing window 1501 network routers 1502,1503,1504 computer 1530 timing window 1541, 1542, correspondingly data cluster 1551,1552,1561 data cluster

Claims (11)

A stream data processing method for continuously analyzing a stream as a flow of time-series data, in which a plurality of data to which time stamps are attached continuously arrives in a line in ascending order of the time stamps. ,
When new data is input, it becomes the target of analysis processing at any point in the data cluster consisting of the time stamp of the new data and the partial sequence of data that arrives continuously on the time axis on the stream Calculating a time difference from the time stamp of the oldest data of the data belonging to the analysis target cluster;
When the calculated time difference is within an arbitrarily set time threshold, the new data is added to the analysis target cluster, and when the calculated time difference exceeds the time threshold, All data belonging to the analysis target cluster is deleted, a data cluster composed only of the new data is generated as the analysis target cluster, and each time the analysis target cluster is updated, it belongs to the analysis target cluster A stream data processing method comprising: analyzing the data set and processing the stream data. The stream data processing method according to claim 1,
When new data is input, the newest data cluster in the analysis target cluster group consisting of the time stamp of the new data and the number of data clusters as many as the number of simultaneous target clusters specified from the new one. A step of calculating a time difference from a time stamp of the oldest data among data belonging to a certain latest analysis target cluster;
When the calculated time difference is within the time threshold, the new data is added to the latest analysis target cluster, and when the calculated time difference exceeds the time threshold, the analysis target cluster Delete all data belonging to the oldest analysis target cluster, which is the oldest one data cluster in the group, generate a data cluster composed only of the new data and set it as the latest analysis target cluster, and the analysis target cluster group A stream data processing method comprising: performing an analysis on a data set belonging to the analysis target cluster group each time an update occurs, and processing the stream data. The stream data processing method according to claim 1,
When an arbitrary column is specified, each data is grouped in units of a data set having the same column value, and when new data is input, the data column value is referred to. The analysis target cluster for each group that is the analysis target cluster for each group to which the data belongs is extracted, and the time stamp of the new data and the time stamp of the oldest data among the data belonging to the group analysis target cluster Calculating a time difference;
When the calculated time difference is within the time threshold, the new data is added to the group-by-group analysis target cluster, and when the calculated time difference exceeds the time threshold, the group Delete all data belonging to another analysis target cluster, generate a data cluster consisting only of the new data, and make it a group analysis target cluster, and whenever the analysis target cluster group is updated, the analysis target cluster A stream data processing method comprising: analyzing a data set belonging to a cluster group and processing the stream data. A stream data processing method for continuously analyzing a stream as a flow of time-series data, in which a plurality of data to which time stamps are attached continuously arrives in a line in ascending order of the time stamps. ,
When an arbitrary time threshold value is specified, when new data is input, the data consists of a time stamp of the new data and a partial sequence of data that continuously arrives on the time axis on the stream. In the cluster, calculating a time difference with the time stamp of the oldest data of the data belonging to the analysis target cluster to be analyzed at a certain time point;
When the calculated time difference is within an arbitrarily set time threshold, the new data is added to the analysis target cluster, and when the calculated time difference is within the time threshold, The new data is added to the analysis target cluster, and when the calculated time difference exceeds the time threshold, all data belonging to the analysis target cluster is deleted, and only the new data is configured. Generating a data cluster to be an analysis target cluster, and performing an analysis on the data set belonging to the analysis target cluster and processing stream data each time the analysis target cluster is updated Stream data processing method. The stream data processing method according to claim 4, wherein:
When new data is input, if the time stamp of the new data and a specific number of simultaneous target clusters are specified by the user, an analysis consisting of the specified number of data clusters counted from the newer one A step of calculating a time difference from a time stamp of the newest data in the data belonging to the latest analysis target cluster which is the newest one data cluster among the target cluster group;
When the calculated time difference is within the time threshold, the new data is added to the latest analysis target cluster, and when the calculated time difference exceeds the time threshold, the analysis target Delete all data belonging to the oldest analysis target cluster to be the oldest one of the cluster group, generate a data cluster composed only of the new data, and set as the latest analysis target cluster, A stream data processing method, comprising: performing analysis on a data set belonging to the analysis target cluster group each time the analysis target cluster group is updated, and processing stream data. The stream data processing method according to claim 4, wherein:
When new data is input, refer to the value of the specified column in the data, and when an arbitrary column is specified, each data is a unit of a data set having the same column value. Among the clusters grouped as follows, the analysis target cluster by group consisting of the analysis target cluster to which the data belongs is extracted, and the time stamp of the new data and the newest data among the data belonging to the analysis target cluster by group are extracted. Calculating a time difference from the time stamp;
When the calculated time difference is within the time threshold, the new data is added to the group-by-group analysis target cluster, and when the calculated time difference exceeds the time threshold, the group Delete all data belonging to another analysis target cluster, generate a data cluster composed only of the new data, and set it as a group analysis target cluster, each time an update of the group analysis target cluster group occurs, A stream data processing method comprising: performing analysis on a data set belonging to the analysis target cluster group and processing stream data. A stream data processing system that continuously analyzes a stream that is a flow of time-series data that continuously arrives in a plurality of pieces of data to which time stamps are attached, arranged in a line in ascending order of the time stamps,
On the stream, a data analysis unit that performs an analysis process on a data set belonging to a data cluster composed of a partial sequence of data that continuously arrives on the time axis;
A time threshold storage area for storing the time threshold; and
When new data arrives, the time stamp of the new data is used as the time stamp of the oldest or newest data among the data belonging to the analysis target cluster that is the data cluster to be analyzed at a certain point in time. A data time comparison unit for determining before and after both times in comparison with a cluster time consisting of a time obtained by adding the time threshold to
When the time stamp is before the cluster time, the new data is added to the analysis target cluster at that time, and when the time stamp is after the cluster time, all the data belonging to the analysis target cluster at that time Data from the data analysis unit, a data cluster composed only of the new data as a new analysis target cluster, and an analysis target data determination unit that inserts the new data into the data analysis unit A featured stream data processing system. The stream data processing system according to claim 7, wherein
It has a simultaneous analysis target cluster number storage area to store any specified number of simultaneous target clusters,
The data time comparison unit
When new data arrives, the time stamp of the new data is the time obtained by adding the time threshold to the oldest or newest data time stamp among the data belonging to the latest analysis target cluster. Compare with the cluster time to determine before and after both times,
When the time stamp is before the cluster time, the analysis target data determination unit adds the new data to the latest analysis target cluster at that time, and when the time stamp is after the cluster time, The oldest one data cluster among the latest analysis target clusters consisting of the newest one data cluster in the analysis target cluster group, which is the number of data clusters as many as the specified number of simultaneous target clusters, counting from the newest one. Delete all data belonging to the oldest analysis target cluster from the data analysis unit, make the data cluster consisting only of the new data a new latest analysis target cluster, and insert the new data into the data analysis unit And a stream data processing system. The stream data processing system according to claim 7, wherein
When an arbitrary column is specified, each data is grouped in units of data sets in which the column values are the same, and the analysis target cluster for each group is classified by group. A storage area for each group that stores a cluster time obtained by adding the time threshold to the time stamp of the oldest or newest data in the data belonging to the analysis target cluster;
A storage area management unit for each group that stores a storage area management table for each group that holds the correspondence between the value of the column and the storage area for each group;
The data time comparison unit
When new data is input to the stream data processing system, the storage area for each group to which the data belongs is extracted from the storage area management table for each group to which the data belongs by referring to the column value of the data. Compare the time stamp and the cluster time stored in the group storage area to determine before and after both times,
The analysis target data determination unit
When the time stamp is before the cluster time, the new data is added to the group analysis target cluster at that time, and when the time stamp is later than the cluster time, analysis by group at the time is performed. Deleting all data belonging to the target cluster from the data analysis unit, setting a data cluster composed only of the new data as a new group analysis target cluster, and inserting the new data into the data analysis unit. A featured stream data processing system. The stream data processing system according to claim 7, wherein
In a stream that is a flow of time series data in which a plurality of data to which time stamps are attached continuously arrives in a line in ascending order of the time stamps, from a partial sequence of data that continuously arrives on the time axis A cluster number storage area for storing the number of data clusters, and
If the time stamp is before the cluster time, the cluster number stored in the cluster number storage area at that time is added to the data and stored in the database, and the time stamp is after the cluster time In this case, add 1 to the cluster number stored in the cluster number storage area at that time, store the cluster number in the cluster number storage area, and add the cluster number after addition to the data to store the database The data time comparison unit comprises:
When new data arrives, the time threshold of the new data is added to the time stamp of the oldest or newest data in the latest cluster that is the latest data cluster at a certain time. A stream data processing system for determining before and after both times by comparing with a cluster time which is a determined time. The data processing system of claim 10, wherein
The cluster determination unit
When new data arrives, the time stamp of the new data is compared with the cluster time. If the time stamp is before the cluster time, the new data is output, and the time stamp is the cluster time. If the time is later than the time, the stream data processing system outputs the new data after sending a signal indicating the delimiter of the data set.

Images