JP2014157510A - Stream data processing system, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To increase the speed of response report generation processing for a query in a system that processes stream data coming in real time.SOLUTION: A system for processing real-time stream data is allowed to generate, when setting a query definition including a time resolution in advance, a pre-aggregation table according to the query definition; update (adding or deleting) an aggregation result for each time resolution; and perform a query to data, which is generated in a certain time range, by using the created pre-aggregation table.

Description

  The present invention relates to a technique for processing stream data arriving in real time, and more particularly to a technique for obtaining a query result of stream data arriving in real time.

  For example, in a road traffic simulation, there is a demand for quickly obtaining a query result based on a large amount of data such as the speed and area of a vehicle that arrives every moment.

An example of a query for real-time data is described as follows in SQL-like pseudo code.
SELECT MAX (speed) from "TABLE"
TIME
FROM time.now ()
TO time.now ()-time.MIN (10)
GroupBy AREA
In this equation, “TABLE” is a table of real-time data accumulated on the hard disk from moment to moment. This formula is the data from the current time to 10 minutes before, and the maximum speed is calculated, but when the query is entered, the cost of scanning all data, disk access, etc. occurs, so for the query It takes a lot of time to generate a response report.

  If the calculation range (time window) is determined in advance, the pre-aggregation table can be pre-aggregated, thereby shortening the query result generation time. In the case of accumulated data, the pre-aggregation table cannot be prepared because the calculation range changes every moment and cannot be determined uniquely.

  Japanese Patent Laid-Open No. 2006-338432 links a plurality of stream data processing systems with a mechanism capable of replicating part or all of stream data, archiving the data in a nonvolatile storage medium, and seamlessly using real-time data and archive data. And a mechanism for improving the performance of query processing is disclosed.

  Japanese Patent Laid-Open No. 2010-108073 discloses a stream data processing server in order to provide stream data that realizes accurate calculation processing in consideration of all input data while keeping the memory usage of a query including a time window constant. However, the query time resolution changer divides the time window into smaller sub-windows, and the query processing engine performs aggregation processing for each sub-window when stream data is received, generates an aggregate tuple, and calculates the query including the time window. A technique for calculating a result by an aggregation process for the aggregation tuple is disclosed.

  Japanese Patent Laid-Open No. 2010-217968 is a computer system that receives data to which time information is attached in chronological order and processes the data according to a plurality of queries registered in advance, and an active computer that processes the received data, This includes a standby computer that processes received data instead of the active computer when it occurs, and the active computer processes the data received by each query in a predefined order and receives it by each query The processing result of the processed data is sent as intermediate data to the standby computer for each query at a predetermined timing. When a failure occurs in the active computer, the standby computer receives the received data and intermediate data Disclosing data by processing the data is disclosed.

  Japanese Patent Laid-Open No. 2011-59967 generates stream data to which time information is given in time series, and generates stream data for a computer system that performs stream data processing on the generated stream data based on a registered query A method for storing, in a computer system, a storage unit that stores query information indicating a component corresponding to the query from a query and stream definitions representing a plurality of types of components constituting the stream data, and stream data. A data generation unit to generate / transmit and a stream data processing unit to process the stream data transmitted from the data generation unit are provided, and the data generation unit transmits to the stream data processing unit based on the query information Generate less stream data from stream data To disclosure.

  However, these prior arts do not address the problem of reducing the time to generate response reports for queries.

JP 2006-338432 A JP 2010-108073 A JP 2010-217968 A JP 2011-59967 A

  An object of the present invention is to speed up a response report generation process for a query in a system for processing stream data coming in real time.

  According to the present invention, when a query definition including time resolution is set in advance in a system for processing real-time stream data based on time constraints, a pre-aggregation table is set according to the query definition. Generate, update (add / delete) the aggregation results for each time resolution, and use the created pre-aggregation table to make it possible to query the data generated in a certain time range The above problem is solved by configuring the system.

  The system according to the present invention analyzes a pre-aggregation table request and registers a pre-aggregation table update scheduler according to a query definition including a time resolution and time window information created in advance by a user, and registers a pre-aggregation table. Automatically generate an update program.

  That is, the pre-aggregation table update scheduler generates a data extraction filter in the time resolution range for each time resolution, and the pre-aggregation table update program reads the data in the time resolution range into stream data that arrives in real time from a sensor or the like. By applying the extraction filter, a value of Key and Value including only the aggregation result is generated and added to the pre-aggregation table.

  In response to receiving a query from a client system, the system according to the present invention applies a reduction process with a key that filters data in the time range in the query to the pre-aggregation table to speed up the process. Generate a report corresponding to the query.

  The system according to the present invention deletes data older than the time window by the pre-aggregation table update program from the end of the pre-aggregation table, thereby preventing the data of the pre-aggregation table from continuing to increase.

  As described above, according to the present invention, the pre-aggregation table is generated by generating the pre-aggregation table having the aggregated entries and deleting the entries so that the size is within the time window range. By generating reports for queries based on tables, you can streamline the aggregation process using the aggregated entries and scan a limited size pre-aggregation table, so you can quickly query The effect is that a report corresponding to can be generated.

It is a figure which shows the outline | summary of the hardware constitutions for implementing this invention. It is a figure which shows the outline | summary of the function structure for implementing this invention. It is a figure which shows the function structure for producing | generating a pre-aggregation table. It is a figure which shows the flowchart for producing | generating a pre-aggregation table. It is a figure which shows the example for generating the pre-aggregation table, and the entry of a pre-aggregation table. It is a figure which shows a mode that the entry which remove | deviated from the time window is deleted in a pre-aggregation table. It is a figure which shows the function structure of a query process. It is a figure which shows the flowchart of a query process. It is a figure which shows the example of a query process typically.

  Embodiments of the present invention will be described below with reference to the drawings. It should be understood that these examples are for the purpose of illustrating preferred embodiments of the invention and are not intended to limit the scope of the invention to what is shown here. Further, throughout the following drawings, the same reference numerals denote the same objects unless otherwise specified.

  FIG. 1 is a diagram showing a schematic configuration of an example of a system for carrying out the present invention. In FIG. 1, vehicles 104 and 106 transmit probe car data to a server 110a through a packet communication network 102. In practice, probe car data is received from thousands or tens of thousands of vehicles, but it should be noted that only two are displayed here as an example.

  Traffic data collection techniques using probe car data are known. For example, JP 2004-110458 A, JP 2004-156882 A, JP 2007-193705 A, JP 2004-241987 A, etc. Please refer. The system uses such well-known techniques to receive data such as vehicle speed, position, and acceleration from time to time.

  The server 110a is connected to the network 108 having any connection configuration such as LAN, WAN, FHHT, etc., and the network 108 further includes servers 110b, 110c,..., 110m, servers 112, 116, 118a,. .. 118k are connected. These servers include, but are not limited to, IBM (R) System X, System i, System p, and other servers that can be purchased from International Business Machines Corporation. Operating systems that can be used with these servers include AIX (trademark), UNIX (trademark), Linux (trademark), and Windows (trademark) 2003 Server. In this embodiment, Linux (trademark) is introduced. Shall.

  The server 110a, the servers 110b, 110c,..., 110m are installed with IBM® InfoSphere Streams, and the server 110a, the servers 110b, 110c,.

  Hadoop is installed in the server 116, and a columnar table 116a is realized in the server 116 using Hbase which is one of Hadoop projects.

  Hadoop is also introduced into the server 112, and the server 112 implements the query server 112a by using Hive, which is one of Hadoop projects. A client computer 114 is connected to the server 112, and a query definition file including information on time resolution and time window, which will be described later, is generated by operating the GUI on the client computer 114. A program has been introduced.

  The servers 118a,..., 118k configure MapReduce by assigning the roles of JobTracker, NameNode, TaskTracker, and DataNode to each of the plurality of servers. Alternatively, a single server can play all these roles.

  Next, an overview of the functional configuration of the system of the present invention will be described with reference to FIG. In FIG. 2, a vehicle 104 includes a sensor 104a, and transmits probe car data to a stream system 200 including a server 110a, servers 110b, 110c,..., 110m via an alerter 104b. Send. The stream system 200 includes a connection manager 202, an alerter 204, and a materialized view ETL 206, and the connection manager 202 sends the received probe car data to the alerter 204 and the materialized view ETL 206.

  Based on the data received from the materialized view ETL 206, the server 116 creates a columnar table 116a on the main memory by Hbase.

  The server 116 generates a pre-aggregation table (PAT) 116b on the main memory by MapReduce 208 configured by a predetermined server group among the servers 118a,. The server 116 also extracts a report from the pre-aggregation table 116b according to the query received from the query server 112a by the MapReduce 210 configured by a predetermined server group among the servers 118a,. Process to create.

  The program executed on MapReduce 208 is generated based on the contents of the query definition file created by the user on the client computer 114.

An example of the contents of the query application definition file is as follows.
Application: {
AggFunction: {avg, sum, cnt, max, min},
GroupBy: {
"Area": Hash1 (gps), // 47 types
"Age": Hash2 (age) // 6 types
},
Measures: {
"speed": float, // per hour
"LineChange": int, // per intr
"quickBrake": int, // per intr
}
resolution: 60 // 1 min
window: 3600 // 60 mins
}

  In the example query app definition file content, AggFunction lists the aggregate functions used. avg is average, sum is total, cnt is count, max is maximum, min is minimum.

GroupBy indicates which key is used for aggregation. Here, it is specified to be aggregated by Area (area) and Age (age). Here, for example, Hash1 (gps) is a function that inputs the value of gps and returns the prefecture name, and has the following code.
Hash1 (gps) {
if gps.lt> 10 and gps.lg <100:
return "tokyo"
elif gps.lt> 12 and gps.lg> 10:
return "osaka"
...
}

  Hash2 (age) is also appropriately defined as a function that stratifies age.

  In the pre-aggregation table, the record key is generated by time + aggregation key. In one example, the key is like 2012/01 / 01_10: 00: 00_tokyo_10.

 “Resolution” is the time granularity for aggregating data in the pre-aggregation table, and “window” is the width of time that records are deleted from the pre-aggregation table after the time has elapsed.

  FIG. 3 shows a functional block for generating a pre-aggregation table (PAT) based on the query application definition file thus created, registering the PAT scheduler, and further generating a program for operating MapReduce 208. FIG.

  In FIG. 3, it is assumed that the query application definition file 302 is created by the client computer 114 and is held on the hard disk or the main memory of the client computer 114, for example.

  On the hard disk of the server 112 that executes the query service, a PAT request analysis module 304 that performs request analysis of the query application definition file 302, and a PAT creation module 306 that creates a pre-aggregation table 116b based on the analysis result of the PAT request analysis module 304. And a PAT scheduler registration module 308 for registering a schedule in the PAT scheduler 312 based on the analysis result of the PAT request analysis module 304, and a MAPPER / REDUCER program for operating MapReduce 208 based on the analysis result of the PAT request analysis module 304 The PAT update program creation module 310 is stored, and is appropriately called, loaded into the main memory and executed in response to an operation from the client computer 114 or the like.

  In this embodiment, the server 112 that executes the query service is also used as a server that performs PAT request analysis, etc., but another server is set up to perform PAT request analysis, PAT creation, PAT scheduler registration, and Processing such as creation of a PAT update program may be executed.

  In the server 116, based on the columner table 116a formed on the main memory according to the data received from the stream system 200, MapReduce 208 operates according to the scheduling of the PAT scheduler 312 to create the aggregated data 314, and pre-aggregation Add to table 116b.

  Next, processing related to generation of a pre-aggregation table (PAT) will be described with reference to the flowchart of FIG. In step 402 of FIG. 4, the PAT request analysis module 304 reads the query application definition file 302 and starts PAT generation processing for the query definition D.

  In step 404, the PAT request analysis module 304 determines whether or not PAT exists, and if not, in step 406, the PAT creation module 306 is defined by defining a schema (K, V) that matches the PAT definition. Call and generate PAT to save data. Here, the PAT definition is the combination of time resolution, (arbitrary) grouping instruction string, target data string, and aggregate function as KEY. This is a Key Value Store type table that can refer to scalar values as VALUE. FIG. 5 is a diagram illustrating an example of processing of the PAT creation module 306 based on the schema. That is, the PAT creation module 306 creates a new table in step 502, generates a key in step 504 based on the query application definition file 302, defines a column family (CF) in step 506, and creates a pre-aggregation table. 116b is generated.

  The key is represented as a time + summary field. In this example, Area + Age is the aggregate field.

  At this stage, there is only an empty pre-aggregation table 116b, but an example of a specific entry is also shown for reference. In this example, time resolution (resolution) = 60 seconds, so the first entry 2012/01 / 01_10: 00: 00_tokyo_10 {“avg”: 42.3, “max”: 120.2, “min”: 0.0, “cnt”: 10 , “Sum”: 423} ... is aggregated data for 60 seconds.

  Returning to the flowchart of FIG. 4, in step 408, the PAT request analysis module 304 determines whether there is an aggregation program (= P) that adds data to the PAT corresponding to the query definition D. In 410, the PAT update program creation module 310 is called, the sensor data is input, the data is aggregated according to the definition of D, the aggregation time, the grouping column, the target data column, the aggregation function is KEY, and this aggregation result is VALUE. Generate a program that generates

The following shows
Application: {
AggFunction: {avg, sum, cnt, max, min},
GroupBy: {
"Area": Hash1 (gps), // 47 types
"Age": Hash2 (age) // 6 types
},
Measures: {
"speed": float, // per hour
"LineChange": int, // per intr
"quickBrake": int, // per intr
}
resolution: 60 // 1 min
window: 3600 // 60 mins
}
An example of Map / Reduce aggregation program code generated by the PAT creation module 306 from the query application definition file 302 is shown. This is pseudo code, but in fact it can be written in any existing programming language such as Java (R), Python, etc.
class MAPPER:
method INITIALIZE ():
Set <K, V> set = new Set <K, V> () // K = str, V = tuple
method MAP (string t, row r):
str s = String.format (“% s-% s-% s”, time, Hash1 (r.gps), Hash2 (r.age))
// for CF: Speed
float sum1 = r.speed, float cnt1 = 1
float max1 = max (r.speed), float min1 = min (r.speed)
tuple t1 = new tuple (sum: sum1, cnt: cnt1, max: max1, min: min1)
set.add (“SP”, t1)
// for CF: LineChange
float sum2 = r.lineChange, float cnt2 = 1
float max2 = max (r.lineChange), float min2 = min (r.lineChange)
tuple t2 = new tuple (sum: sum2, cnt: cnt2, max: max2, min: min2)
set.add (“LC”, t2)
EMIT (str s, Set set)
class REDUCER:
method INITIALIZE ():
float sum1, sum2, cnt1, cnt2, max1, max2, min1, min2 = 0

method REDUCE (str s, sets [r1, r2, ..]):
for all Set set ∈ sets [r1, r2,…] do
sum1 = sum1 + set.get (“SP”). sum, sum2 = sum2 + set.get (“LC”). sum
cnt1 = cnt1 + set.get (“SP”). cnt, cnt2 = cnt2 + set.get (“LC”). cnt
max1 = set.get (“SP”). max> max1? Set.get (“SP”). max: max1

done
writeToIncrementalDB (s, CF = “SP”, sum1, cnt1, max1, min1)
writeToIncrementalDB (s, CF = “LC”, sum2, cnt2, max2, min2)
The code of the aggregate program generated in this way causes the MapReduce system 208 to operate. Note that the resolution and window values do not appear here, but their setting values are set in the PAT scheduler 312.

Returning to the flowchart of FIG. 4, the PAT request analysis module 304 schedules the PAT scheduler 312 to start the program at each time resolution by the PAT scheduler registration aggregation module 308 in step 412. The PAT scheduler 312 can be handled in a driver that is substantially outside the MapReduce code described above. For example:
* / 1 * * * * hadoop jar PAT-Update-MR.jar
For example, the update program starts every minute. The driver r of the activated program first obtains the current time, and decides at what point of time it should be consolidated. For example, when the current time is acquired and it is 12:01:02, the reference aggregation time is determined to be 12:01. This is an implementation issue and can be changed freely. If this 12:01 criterion is dynamically determined at the time of program execution, the resolution time is traced back from here, and the data in the time range (= resolution) of 12:00:00-12:00:59 It is uniquely determined as input. Since data within this range is linked with respect to the time of 12:01, it is used as one of the keys at the stage of writing the final Reduce result to the PAT.

  In step 414, the aggregation program P stores the current time = T1 on MapReduce 208, and in step 416, the PAT scheduler 312 executes the last execution time (= T0) of the sensor data input and the current time. A rule for filtering the time (= T1) into a range is given to P, the aggregation program P is executed, and the result is output (= R). The rule here is a filter rule in which stored data is input from the current time to the data before the time resolution. Based on the analysis result of the PAT request analysis module 304, the PAT scheduler 312 Create in advance. Then, MapReduce 208 designates the data filtered to the time resolution width as input, and executes the aggregation program MapReduce. In this case, scheduling refers to scheduling so that the rule creation / MR execution is repeated after the time resolution has elapsed.

  More specifically, Map processing is to generate a Tuple that selects the value specified by Measure from the input data, using the key that combines the time resolution (current time) and GroupBy ID, and outputs it as Map processing output. In the Reduce process, a Tuple group grouped for each key is input, and an aggregation function is divided into values in the Tuple to perform an aggregation operation. The aggregation result for Key is used as a new Tuple for the output of reduce processing. The Reduce process further includes adding this output to the aggregation table and simultaneously deleting the data before window time from the current time. This corresponds to the processing of adding R to the beginning of the PAT and deleting data older than the Time Window time set in D from the end of the PAT in step 418 in FIG. FIG. 6 shows how the data at the end of the PAT is deleted.

  In step 420, the PAT scheduler 312 stores the time T1 in T0 and then stops until the time resolution time elapses. In step 422, the PAT scheduler 312 determines whether an execution stop request is received from the client computer 114 or the like. If so, execution is terminated; otherwise, return to step 414.

  FIG. 7 is a diagram showing functional blocks of processing for creating a report for a query according to the present invention based on the OLAP query 702 created by the client computer 114.

The OLAP query 702 is, for example, as shown in SQL-like pseudo code as follows.
SELECT AVG (speed) from PAT
TIME
FROM time.now ()
TO time.now ()-time.MIN (2)
GroupBy area
This is a query that finds the average speed over a period of 2 minutes from now.

  The query server 112a in the server 112 converts the OLAP query 702 into filter conditions, preferably using Hive, one of Hadoop's projects. This filter condition is added to the filter 706 in the server 116 as a rule. In response to this, the filter 706 extracts data in the selected range from the pre-aggregation table 116b according to the set rule.

  On the other hand, the query server 112 a provides the MapReduce 210 with the MapReduce program 708, whereby the MapReduce 210 performs aggregation processing specified by the MapReduce program 708 and generates a result report 710. The result report 710 is sent to the display module 704 of the client computer 114 by the communication module 712 of the server 116, and the display module 704 displays the result report 710 as a graph as appropriate.

  Next, query processing will be described with reference to the flowchart of FIG. In step 802 of FIG. 8, the query server 112a begins generating results for the submitted query request. Throughout the flowchart of FIG. 8, D is a query application definition file, and Q is a submitted OLAP query.

  In step 804, the query server 112a determines whether data necessary for generating the Q result is defined by D. If not, in step 806, the query server 112a returns a termination processing error to the query request source. Proceed to step 808.

  In step 808, the query server 112a determines whether or not the time resolution requested in Q is longer than the time resolution defined in D. If not, in step 806, an end processing error is returned to the query requester. If so, in step 810, the current time is saved as T0.

  At step 812, query server 112a determines whether the time range required for Q result generation includes a future time relative to T0, otherwise, by sending a rule to filter 706, step 814. In step 816, the time range described in Q is filtered and the corresponding record is extracted, and in step 816, the aggregate function applied value corresponding to the aggregate function, grouping column, and target data column required in Q Extracting from the filtered records, further applying the same aggregate function to these, outputting the calculation result (= R), returning the query result to the query request source in step 818, and ending the processing.

  Returning to step 812, if the query server 112a determines that the time range required for Q result generation includes a future time relative to T0, the query server 112a, in step 820, performs processing 1 ( That is, step 814 and subsequent step 816) are performed, a record of time past T0 is extracted from the PAT, and the result of Q is output (= R).

  The query server 112a then saves at step 822 as the earliest time required in Q = T2, and stops at step 824 until the time resolution time has elapsed.

  Then, in step 826, the query server 112a reads the head of the PAT, acquires the aggregation result corresponding to the Q of the latest entry, combines it with R and outputs the result (= R), and in step 828, ( It is determined whether or not (current time = T1) <T2, and if so, the process returns to step 824. When T1 passes T2, the query server 112a returns the query result to the query request source in step 818 and ends the process.

  FIG. 9 is a diagram schematically illustrating an example of the query processing illustrated in FIGS. 7 and 8. In FIG. 9, when a user submits an OLAP query 702, the query server 112a generates a time key filtering 902 and a column family value filtering 904, as shown. Then, the query server 112a generates a rule from these filterings and applies it to the filter 706, thereby obtaining table data 906 in the selected range.

  The query server 112a calls the Map function 908 and the Reduce function 910 on the MapReduce 210 to perform the final reduction on the table data 906 in the selected range, and obtains a result report 710. Such a series of processes is called a reduction process. The result report 710 is returned to the user.

  As described above, in the embodiment in which the present invention is implemented by MapReduce of the Hadoop (trademark) framework, the Hbase is used for generating the pre-aggregation table and the Hive is used for the query server. In addition to Hbase, Cassandra, Memcached, etc. can be used to generate the aggregation table.

 The present invention is not limited to Hadoop or MapReduce, and can be implemented on a computer platform having an arbitrary operating system and hardware.

  Further, the present invention can be used not only for probe car data but also for any application example that receives data with time and aggregates it into a table such as weather data, financial data, and other simulations.

112 ... Query server 112a ... Query server 114 ... Client computer 116a ... Pre-aggregation table 200 ... Stream system 208, 210 ... MapReduce system 302 ... Query application Definition file 304 ... PAT analysis request module 306 ... PAT creation module 308 ... PAT scheduler registration module 310 ... PAT update program creation module 312 ... PAT scheduler 702 ... OLAP query

Claims (18)

In a system for querying data generated in real time by computer processing,
Storing an intermediate result for generating a query processing result from the data from the current time until the time of the time resolution elapses, using the specified time resolution and storage period value; and the storage period An intermediate result processing step for deleting the stored intermediate result after elapse of
Scheduling the intermediate result processing step to operate every time of a specified time resolution;
Generating a query result from the intermediate result in response to receiving a query;
Method.   The method of claim 1, wherein the intermediate result processing step is processed by a system implementing MapReduce.   The program further comprising: generating a program that executes the intermediate result processing step and that operates on a system that implements the MapReduce from the specified time resolution and storage period value by the processing of the computer. The method described in 1.   The method according to claim 1, further comprising the step of generating a program for executing the scheduling step from the specified time resolution and retention period value by the processing of the computer.   The method of claim 1, wherein the intermediate result has a field keyed by a value including time.   6. The method of claim 5, wherein the query includes a time range, and generating the query result includes aggregating data having a key value in the time range and a value including the time. In a system for querying data generated in real time by computer processing,
In the computer,
Storing an intermediate result for generating a query processing result from the data from the current time until the time of the time resolution elapses, using the specified time resolution and storage period value; and the storage period An intermediate result processing step for deleting the stored intermediate result after elapse of
Scheduling the intermediate result processing step to operate every time of a specified time resolution;
Generating a query result from the intermediate result in response to receiving the query;
program.   The program according to claim 7, wherein the intermediate result processing step is processed by a system that implements MapReduce.   9. The program further comprising: generating a program that executes the intermediate result processing step and that operates on a system that implements the MapReduce from the specified time resolution and storage period value by the processing of the computer. The program described in.   The program according to claim 7, further comprising a step of generating a program for executing the scheduling step based on the designated time resolution and storage period value by processing of the computer.   The program according to claim 7, wherein the intermediate result has a field having a value including time as a key.   The program according to claim 11, wherein the query includes a range of time, and the step of generating the query result includes a step of aggregating data whose key is in the range of time with a value including the time. In a system for querying data generated in real time by computer processing,
Storing an intermediate result for generating a query processing result from the data from the current time until the time of the time resolution elapses, using the specified time resolution and storage period value; and the storage period Intermediate result processing means for deleting the stored intermediate result after elapse of
Means for scheduling said intermediate result processing step to operate every time of a specified time resolution;
Means for generating a query result from the intermediate result in response to receiving a query;
system.   The system according to claim 13, wherein the intermediate result processing means is processed by a system that implements MapReduce.   15. The program further comprising: means for executing the intermediate result processing means, and generating a program operating on a system implementing MapReduce from the designated time resolution and storage period values by the processing of the computer. The system described in.   14. The system of claim 13, further comprising means for generating a program for executing the scheduling step from the specified time resolution and retention period values.   14. The system of claim 13, wherein the intermediate result has a field keyed by a value including time.   18. The system of claim 17, wherein the query includes a time range, and the means for generating the query result performs the function of aggregating data whose key is in the time range with a value including the time.

Images