US20120124233A1  Method and apparatus for adaptive load shedding  Google Patents
Method and apparatus for adaptive load shedding Download PDFInfo
 Publication number
 US20120124233A1 US20120124233A1 US13/342,487 US201213342487A US2012124233A1 US 20120124233 A1 US20120124233 A1 US 20120124233A1 US 201213342487 A US201213342487 A US 201213342487A US 2012124233 A1 US2012124233 A1 US 2012124233A1
 Authority
 US
 United States
 Prior art keywords
 tuples
 processing
 number
 data stream
 method
 Prior art date
 Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
 Abandoned
Links
Images
Classifications

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
 H04L49/00—Packet switching elements
 H04L49/90—Queuing arrangements
Abstract
One embodiment of the present method and apparatus adaptive load shedding includes receiving at least one data stream (comprising a plurality of tuples, or data items) into a first sliding window of memory. A subset of tuples from the received data stream is then selected for processing in accordance with at least one data stream operation, such as a data stream join operation. Tuples that are not selected for processing are ignored. The number of tuples selected and the specific tuples selected depend at least in part on a variety of dynamic parameters, including the rate at which the data stream (and any other processed data streams) is received, time delays associated with the received data stream, a direction of a join operation performed on the data stream and the values of the individual tuples with respect to an expected output.
Description
 This application is a division of copending U.S. patent application Ser. No. 12/165,524, filed Jun. 30, 2008, which in turn is a continuation of U.S. patent application Ser. No. 11/068,137, filed Feb. 28, 2005 (now U.S. Pat. No. 7,610,397), both of which are herein incorporated by reference in their entireties.
 This invention was made with Government support under Contract No.: H98230043001 awarded by the U.S. Department of Defense. The Government has certain rights in this invention.
 The present invention relates generally to data stream processing and relates more particularly to the optimization of data stream operations.
 With the proliferation of Internet connections and networkconnected sensor devices comes an increasing rate of digital information available from a large number of online sources. These online sources continually generate and provide data (e.g., news items, financial data, sensor readings, Internet transaction records, and the like) to a network in the form of data streams. Data stream processing units are typically implemented in a network to receive or monitor these data streams and process them to produce results in a usable format. For example, a data stream processing unit may be implemented to perform a join operation in which related data items from two or more data streams (e.g., from two or more news sources) are culled and then aggregated or evaluated, for example to produce a list of results or to corroborate each other.
 However, the input rates of typical data streams present a challenge. Because data stream processing units have no control over the sometimes sporadic and unpredictable rates at which data streams are input, it is not uncommon for a data stream processing unit to become loaded beyond its capacity, especially during rate spikes. Typical data stream processing units deal with such loading problems by arbitrarily dropping data streams (e.g., declining to receive the data streams). While this does reduce loading, the arbitrary nature of the strategy tends to result in unpredictable and suboptimal data processing results, because data streams containing useful data may unknowingly be dropped while data streams containing irrelevant data are retained and processed.
 Thus, there is a need in the art for a method and apparatus for adaptive load shedding.
 One embodiment of the present method and apparatus adaptive load shedding includes receiving at least one data stream (comprising a plurality of tuples, or data items) in a first sliding window of memory. A subset of tuples from the received data stream is then selected for processing in accordance with at least one data stream operation, such as a data stream join operation. Tuples that are not selected for processing are ignored. The number of tuples selected and the specific tuples selected depend at least in part on a variety of dynamic parameters, including the rate at which the data stream (and any other processed data streams) is received, time delays associated with the received data stream, a direction of a join operation performed on the data stream and the values of the individual tuples with respect to an expected output.
 So that the manner in which the above recited embodiments of the invention are attained and can be understood in detail, a more particular description of the invention, briefly summarized above, may be obtained by reference to the embodiments thereof which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, for the invention may admit to other equally effective embodiments.

FIG. 1 is a schematic diagram illustrating one embodiment of a data stream processing unit adapted for use with the present invention; 
FIG. 2 is a flow diagram illustrating one embodiment of a method for adaptive load shedding for data stream processing according to the present invention; 
FIG. 3 is a flow diagram illustrating one embodiment of a method for determining the quantity of data to be processed, in accordance with the method illustrated inFIG. 2 ; 
FIG. 4 is a schematic diagram illustrating the basis for one embodiment of an adaptive tuple selection method based on time correlation; 
FIG. 5 is a flow diagram illustrating one embodiment of a method for prioritizing subwindows of a given sliding window for use in tuple selection, in accordance with the method illustrated inFIG. 2 ; 
FIG. 6 is a flow diagram illustrating one embodiment of a method for selecting tuples for processing, in accordance with the method illustrated inFIG. 2 ; 
FIG. 7 is a flow diagram illustrating another embodiment of a method for selecting tuples for processing, in accordance with the method illustrated inFIG. 2 ; and 
FIG. 8 is a flow diagram illustrating yet another embodiment of a method for selecting tuples for processing, in accordance with the method illustrated inFIG. 2 .  To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures.
 In one embodiment, the present invention is a method and apparatus for adaptive load shedding, e.g., for data stream operations. Embodiments of the present invention make it possible for load shedding to be performed in an “intelligent” (e.g., nonarbitrary) manner, thereby maximizing the quality of the data stream operation output (e.g., in terms of a total number of output items generated or in terms of the value of the output generated).
 Within the context of the present invention, the term “tuple” may be understood to be a discrete data item within a stream of data (e.g., where the stream of data may comprise multiple tuples).

FIG. 1 is a schematic diagram illustrating one embodiment of a data stream processing unit 100 adapted for use with the present invention. The data stream processing unit 100 illustrated inFIG. 1 is configured as a general purpose computing device and is further configured for performing data stream joins. Although the present invention will be described within the exemplary context of data stream joins, those skilled in the art will appreciate that the teachings of the invention described herein may be applied to optimize a variety of data stream operations, including filtering, transforming and the like.  As illustrated, the data stream processing unit 100 is configured to receive two or more input data streams 102 _{1}102 _{n }(hereinafter collectively referred to as “input data streams 102”), e.g., from two or more different data sources (not shown), and processes these input data streams 102 to produce a single output data stream 104. The data stream processing unit 100 thus comprises a processor (e.g., a central processing unit or CPU) 106, a memory 108 (such as a random access memory, or RAM) and a storage device 110 (such as a disk drive, an optical disk drive, a floppy disk drive and the like). Those skilled in the art will appreciate that some data stream processing units may be configured to receive only a single input data stream and still be adaptable for use with the present invention.
 As each input data stream 102 is received by the data stream processing unit 100, tuples (e.g., discrete data items) from the input data streams 102 are stored in a respective sliding window 112 _{1}112 _{n }(hereinafter collectively referred to as “sliding windows 112”) in the memory 108. These sliding windows can be userconfigurable or systemdefined (e.g., based on available memory space) and may be countbased (e.g. configured to store “the last x tuples” of the input data streams) or timebased (e.g., configured to store “the last x seconds” of the input data streams). Thus, as a new tuple from an input data stream 102 arrives in a respective sliding window 112, the new tuple may force an existing tuple to leave the sliding window 112 (if the sliding window 112 was full before receipt of the new tuple). The memory 108 also stores program logic for the adaptive load shedding method of the present invention, as well as logic for other miscellaneous applications. Alternatively, portions of the input data stream and program logic can be stored on the storage medium 110.
 To perform a join operation, the processor 106 executes the program logic stored in the memory 108 to process tuples from the input data streams 102 that are stored in the sliding windows 112. Specifically, the join operation is performed by comparing a tuple (e.g., tuple x) from a first sliding window 112 _{1 }with at least one tuple from at least a second sliding window 112 _{n}. If one or more tuples from the second sliding window 112 _{n }(e.g., tuples y, v, and u) match the join condition for the tuple x, then the matching tuples will be joined such that the output data stream 104 will comprise one or more matched sets of tuples, e.g., (x, y), (x, v) and (x, u).
 Thus, the adaptive load shedding method of the present invention may be represented by one or more software application (or even a combination of software and hardware, e.g., using Application Specific Integrated Circuits (ASIC)), where the software is loaded from a storage medium (e.g., storage device 110) and operated by the processor 106 in the memory 108 of the data stream processing unit 100. Thus, in one embodiment, the method for adaptive load shedding described in greater detail below can be stored on a computer readable medium or carrier (e.g., RAM, magnetic or optical driven or diskette, and the like).
 Alternatively, the method for adaptive load shedding described in greater detail below can be represented as a discrete load shedding module (e.g., a physical device or subsystem that is coupled to the processor 106 through a communication channel) within the data stream processing unit.

FIG. 2 is a flow diagram illustrating one embodiment of a method 200 for adaptive load shedding for data stream processing according to the present invention. The method 200 may be implemented at, for example, a data stream processing unit such as the data stream processing unit 100 illustrated inFIG. 1 .  The method 200 is initialized at step 202 and proceeds to step 204, where the method 200 receives at least one input data stream. The input data stream is received, for example, within a sliding window of memory as discussed with reference to
FIG. 1 . The method 200 then proceeds to step 206 and determines what resources are available to process the input data stream.  In step 208, the method 200 determines, based at least in part on the availability of processing resources, the quantity of data (e.g., how many tuples from within the sliding window) that should be processed. In one embodiment, a determination of how much data should be processed is based at least in part on the rate at which the input data stream is currently being received.
 The method 200 then proceeds to step 210 and, based on the amount of data to be processed, selects specific tuples from within the sliding window for processing. Thus, the number of tuples selected for processing will not exceed the total amount of data that was identified for processing in step 208. Tuples not selected in step 210 are then shed (e.g., not processed). In one embodiment, selection of specific tuples for processing is based at least in part on at least one of: the values of the tuples (e.g., tuples most closely related to the purpose motivating the data stream processing operation), the time correlation between two or more tuples, and the join direction of the data stream processing application (e.g., where the method 200 is being implemented to shed load for a data stream join).
 In step 212, the method 200 processes the selected tuples in accordance with at least one data stream operation (e.g., joining, filtering, transforming and the like). Received tuples that are not selected for processing are ignored, meaning that the unselected tuples are not immediately processed, but may be processed at a later point in time, e.g., if the processing resources become available and if the unselected tuples are still present in a sliding window of memory. The method 200 then terminates in step 214.
 The method 200 thereby optimizes a data stream operation by intelligently shedding load. Rather than processing every tuple in a sliding window of memory (and shedding load by arbitrarily discarding tuples before they can even enter the sliding window), the method 200 allows all tuples to enter the sliding window and then processes only selected tuples from within the sliding window based on one or more stated parameters and on resource availability. Thus all data provided to the method 200 remains available for processing, but only a tailored subset of this available data is actually processed. Thus, the method 200 maximizes the quality of the data stream operation output for a given set of available processing resources.

FIG. 3 is a flow diagram illustrating one embodiment of a method 300 for determining the quantity of data (e.g., number of tuples) to be processed, e.g., in accordance with step 208 of the method 200. The method 300 enables the quantity of data that is selected for processing to be adjusted according to the rate at which new data is being received (e.g., the rates at which input data streams are arriving), thereby facilitating efficient use of processing resources.  The method 300 is initialized at step 302 and proceeds to step 304, where the method 300 sets a fraction, r, of the data in each sliding window to be processed to a default value. In one embodiment, the default value is either one or zero, with a default value of one implying an assumption that a stream join operation can be performed fully without any knowledge of data streams yet to be received. In one embodiment, this fraction, r, is applied to all sliding windows containing available tuples for processing.
 At substantially the same time that the value for r is set, the method 300 proceeds to step 306 and sets the time, t, to T. The method 300 then proceeds to step 308 and calculates an adaptation factor, β, where β is based on the numbers of tuples fetched from the sliding windows since a last execution of the method 300 and on the arrival rates of the input data streams in the sliding windows since the last execution of the method 300. In one embodiment, β is calculated as:

$\begin{array}{cc}\beta =\frac{{\alpha}_{1}+\dots +{\alpha}_{n}}{\left({\lambda}_{1}+\dots +{\lambda}_{n}\right)\ue89e{T}_{r}}& \left(\mathrm{EQN}.\phantom{\rule{0.8em}{0.8ex}}\ue89e1\right)\end{array}$  where α_{1n }is a number of tuples fetched from a designated sliding window (e.g., where n sliding windows are being processed) since the last execution of the method 300, λ_{1n }is the average arrival rate of an input data stream in a designated sliding window since the last execution of the method 300, and T_{r }is the adaptation period (e.g., such that the method 300 is configured to execute every T_{r }seconds). In one embodiment, the size of the adaptation period T_{r }is selected to be adaptive to “bursty” or sporadic data input rates. In one embodiment, T_{r }is approximately 5 seconds.
 Once the adaptation factor β is calculated, the method 300 proceeds to step 310 and determines whether β is less than one. If the method 300 concludes that β0 is less than one, the method 300 proceeds to step 312 and resets r to β*r, which effectively results in smaller fractions of the sliding windows being selected for processing. Alternatively, if the method 300 concludes that β is greater than or equal to one, the method 300 proceeds to step 314 and resets r to the smaller value of one and δ_{r}*r, which effectively results in larger fractions of the sliding windows being selected for processing. In this case, δ_{r }is a fraction boost factor. In one embodiment, the fraction boost factor δ_{r }is predefined by a user or by the data stream processing unit. In one embodiment, the fraction boost factor δ_{r }is approximately 1.2. Those skilled in the art will appreciate that selecting higher values for the fraction boost factor δ_{r }will result in a more aggressive increase of the fraction, r.
 Once the value of r has been appropriately reset, the method 300 proceeds to step 316 and waits for the time t to equal T+T_{r}. That is, the method 300 waits for the start of the next adaptation period. Once t=T+T_{r}, the method 300 returns to step 306 and proceeds as described above so that the fractions r of the sliding windows that are selected for processing continually adapt to the arrival rates of the input data streams. In this manner, load shedding can be adapted based on available processing resources even when the arrival rates of input data streams are sporadic or unpredictable.
 Once the amount of data to be processed is determined, specific tuples may be selected for processing from within each sliding window. One method in which tuples may be selected for processing adapts the selection process according to temporal correlations between tuples by prioritizing tuples according to the times in which the tuples were generated or entered the sliding windows.

FIG. 4 is a schematic diagram illustrating the basis for one embodiment of an adaptive tuple selection method based on time correlation. Consider a data stream processing unit that is configured to receive x input data streams S_{1}S_{x }in x respective sliding windows W_{1}W_{x}. Sliding window S_{i}, where iε[1, . . . , x], is a representative sliding window. Sliding window S_{i }contains a total of w_{i }seconds worth of tuples and is divided into n subwindows B_{i,1}B_{i,n }each containing b seconds worth of tuples (i.e., such that n=1+[w_{i}/b]).  Those skilled in the art will appreciate that the size, b, of each subwindow is subject to certain considerations. For example, smaller subwindows are better for capturing the peak of a match probability distribution, but because there is a larger number of subwindows, more processing capacity is needed. On the other hand, larger subwindows are less costly from a processing standpoint, but are less adaptive to the dynamic natures of the input data streams.
 New tuples enter the sliding window S_{i }at a first subwindow B_{i,1 }and continue to enter the first subwindow B_{i,1 }until the most recent tuple to enter the first subwindow B_{i,1 }has a timestamp that is b seconds longer than the timestamp of the first tuple to enter first subwindow B_{i,1}. At this point, all tuples residing in the last subwindow B_{i,n }are discarded, and all subwindows shift over by one position (i.e., so that the last subwindow B_{i,n }becomes the first subwindow, the first subwindow B_{i,1 }becomes the second subwindow, and so on). Thus, the sliding window W_{i }is maintained as a circular buffer, and tuples do not move from one subwindow to another, but remain in a single subwindow until that subwindow is emptied and shifts to the position of the first subwindow B_{i,1}.
 As will be discussed in greater detail below, one embodiment of an adaptive tuple selection method based on time correlation establishes a time correlation period, T_{c}, where the method executes once every T_{c }seconds to adapt the manner in which specific tuples are selected based on time correlation between incoming tuples. In the case where the tuple selection method is implemented in conjunction with a data stream join operation, one of two tuple processing methods may be performed between two consecutive executions of the tuple selection method. These two tuple processing methods are referred to as full processing and selective processing. Full processing involves comparing a newly input tuple from a first sliding window against all tuples in at least a second sliding window. Selective processing involves comparing a newly input tuple from a first sliding window against tuples contained only in highpriority subwindows of at least a second sliding window. As will be described in greater detail below, in one embodiment subwindows are prioritized based on a number of output tuples expected to be produced by comparing the newly input tuple from the first sliding window against tuples in each subwindow of the second sliding window.
 Whether a newly input tuple is subjected to full or selective processing is dictated by the tuple's respective sampling probability, γ. The probability of a newly input tuple being subjected to full processing is r*γ; conversely, the probability of the same newly input tuple being subjected to selective processing is 1−r*γ. Thus, the fraction, r, that is determined by the method 300 is used to scale the sampling probability γ so that full processing will not consume all processing resources during periods of heavy loading. In one embodiment, the sampling probability γ is predefined by a user or by the data stream processing unit. The value of the sampling probability γ should be small enough to avoid undermining selective processing, but large enough to be able to capture a match probability distribution. In one embodiment, the sampling probability γ is set to approximately 0.1.
 Full processing facilitates the collection of statistics that indicate the “usefulness” of the first sliding window's subwindows for the data stream join operation. In one embodiment, full processing calculates, for each subwindow B_{i,j}, a number of expected output tuples produced by comparing a newly input tuple with a tuple from the subwindow B_{i,j}. This number of expected output tuples may be referred to as o_{i,j}. Specifically, during full processing, for each match found with a tuple in the subwindow B_{i,j}, a counter ô_{i,j }is incremented. The ô_{i,j }values are later normalized (e.g., by γ*r*b*λ_{1}* . . . *λ_{n}*T_{c}) to calculate the number of expected output tuples o_{i,j}.

FIG. 5 is a flow diagram illustrating one embodiment of a method 500 for prioritizing subwindows of a given sliding window for use in tuple selection, e.g., in accordance with step 210 of the method 200. Specifically, the method 500 enables subwindows of a sliding window to be sorted based on time delays (e.g., application dependent or communication related) between matching tuples in the sliding window and tuples to be compared there against, thereby facilitating the selection of the most relevant tuples for processing.  The method 500 is initialized at step 502 and proceeds to step 504, where the method 500 sets the current time, t, to T and sets i to one, where i identifies a sliding window to be examined (e.g., sliding window W_{i }of
FIG. 4 ). The method 500 then proceeds to step 506 and sorts the subwindows of the selected sliding window into an array, O. Specifically, the subwindows are sorted in descending order based on their respective numbers of expected output tuples (unnormalized), ô_{i,j}, such that {ô_{i,j}jε[1, . . . , n]}.  The method 500 then proceeds to step 508 and, for each subwindow, B_{i,j}, (where jε[1, . . . , n]), calculates new values for the respective numbers of expected output tuples, o_{i,j}, and s_{i} ^{j}. In this case, s_{i} ^{j}=k means that the j^{th }item in the sorted list {o_{i,l}/lε[1, . . . , n]} is o_{i,k}, where the list {o_{i,l}/lε[1, . . . , n]} is sorted in descending order. In one embodiment, the new value for o_{i,j }is calculated as:

$\begin{array}{cc}{o}_{i,j}=\frac{{\hat{o}}_{i,j}}{\gamma *r*b*{\lambda}_{i}*\sum _{j\ne i}\ue89e{\lambda}_{j}*{T}_{c}}& \left(\mathrm{EQN}.\phantom{\rule{0.8em}{0.8ex}}\ue89e2\right)\end{array}$  and s_{i} ^{j}=k, where O[j]=ô_{i,j}.
 In step 510, the method 500 then resets all ô_{i,j }values to zero. The method 500 then proceeds to step 512 and sets i to i+1, e.g., so that the method 500 focuses on the next sliding window to be examined. Thus, the method 500 inquires, at step 514, if i is now less than 3. If the method 500 determines that i is less than three, the method 500 returns to step 506 and proceeds as described above, e.g., in order to analyze the subwindows of the next sliding window to be examined.
 Alternatively, if the method 500 determines in step 514 that i is not less than three, the method 500 proceeds to step 516 and waits until the time, t, is T+T_{c}. That is, the method 500 waits for the start of the next time correlation adaptation period. Once the next time correlation adaptation period starts, the method 500 returns to step 504 and proceeds as described above so that the o_{i,j }and s_{i} ^{j }values of each subwindow under examination continually adapt to the temporal correlations between the incoming data streams.

FIG. 6 is a flow diagram illustrating one embodiment of a method 600 for selecting tuples for processing, e.g., in accordance with step 212 of the method 200. Specifically, the method 600 processes a given tuple, y, from a first sliding window W_{1 }against one or more selected tuples in a second sliding window W_{2}. As will be described in further detail below, the method 600 exploits knowledge gained from the method 500 regarding the prioritizing of subwindows within the second sliding window W_{2}.  The method 600 is initialized at step 602 and proceeds to step 604, where the method 600 identifies the tuple, y, for processing from the first sliding window W_{1}. In one embodiment, the identified tuple, y, is a newly received tuple. The method 600 also identifies the second window W_{2 }against which to process the identified tuple y, e.g., in accordance with a data stream join operation.
 In step 606, the method 600 obtains or generates a random number R. The method 600 then proceeds to step 608 and inquires if R is less than r*γ. If the method 600 determines that R is less than r*γ, the method 600 proceeds to step 612 and commences full processing on the tuple y from the first window W_{1}.
 Specifically, in step 612, the method 600 processes the tuple y from the first sliding window W_{1 }against all tuples in the second sliding window W_{2 }in accordance with at least one data stream operation (e.g., a join). The method 600 then proceeds to step 614 and, for each matched output in each subwindow of the second sliding window W_{2}, increments the subwindow's unnormalized output count ô_{i,j }(e.g., by one). The method 600 then terminates in step 628.
 Alternatively, if the method 600 determines in step 608 that R is not less than r*γ, the method 600 proceeds to step 610 and commences selective processing on the tuple y from the first sliding window W_{1}. Specifically, in step 610, the method 600 determines the number of tuples to be processed from the second sliding window W_{2}. In one embodiment, the number of tuples to be processed, a, is calculated as:

a=r*W _{1} (EQN. 3)  where W_{1} is the size of the first sliding window W_{1 }(e.g., as measured in terms of a number of tuples or a duration of time contained within the first sliding window W_{1}).
 The method 600 then proceeds to step 616 and starts to processes the tuple y from the first window W_{1 }against tuples in the second sliding window W_{2}, starting with the highest priority subwindow in the second sliding window W_{2 }(e.g., B_{i}, s_{i} ^{j}) and working through the remaining subwindows in descending order of priority until the tuple y from the first sliding window W_{1 }has been processed against a tuples from the second sliding window W_{2}. Specifically, in step 616, the method 600 inquires whether any subwindows remain for processing in the second sliding window W_{2 }(e.g., whether the current subwindow is the last subwindow). If the method 600 concludes that no subwindows remain for processing in the second sliding window W_{2}, the method 600 terminates in step 628.
 Alternatively, if the method 600 concludes in step 616 that there are subwindows that are available for processing in the second sliding window W_{2}, the method 600 proceeds to step 618 and adjusts the number of tuples available for processing in the second sliding window W_{2 }to account for the tuples contained in the first available subwindow (e.g., the highestpriority available subwindow, B_{i}, s_{i} ^{j}). That is, the method 600 subtracts the number of tuples in the first available subwindow from the total number of tuples, a, to be selected for processing from the second sliding window W_{2 }(e.g., such that the new value for a=a−B_{i}, s_{i} ^{j}). Thus, a−B_{i}, s_{i} ^{j} more tuples from the second sliding window W_{2 }may still be processed against the tuple y from the first sliding window W_{1}.
 The method 600 then proceeds to step 620 and inquires whether any more tuples from the second sliding window W_{2 }are available for processing (e.g., whether the adjusted a>0). If the method 600 concludes that a number of tuples in the second sliding window W_{2 }can still be processed, the method 600 proceeds to step 624 and processes the tuple y from the first sliding window W_{1 }against all tuples in the first available subwindow B_{i,s} _{ i } _{ j }of the second sliding window W_{2}. The method 600 then proceeds to step 626 and proceeds to the next available (e.g., nexthighest priority) subwindow in the second sliding window W_{2}. The method 600 then returns to step 616 and proceeds as described above in order to determine how many tuples from the next available subwindow can be used for processing.
 Alternatively, if the method 600 concludes in step 620 that no more tuples can be processed from the second sliding window W_{2 }(e.g., that the adjusted a is <0), the method 600 proceeds to step 622 and processes the tuple y from the first sliding window W_{1 }against a fraction of the tuples contained within the first available subwindow B_{i,s} _{ i } _{ j }. In one embodiment, this fraction, r_{c}, is calculated as:

$\begin{array}{cc}{r}_{e}=1+\frac{a}{\uf603{B}_{i,{s}_{i}^{j}}\uf604}& \left(\mathrm{EQN}.\phantom{\rule{0.8em}{0.8ex}}\ue89e4\right)\end{array}$  where r_{c }is a fraction with a value in the range of zero to one. Once the tuple y from the first sliding window W_{1 }has been processed against the fraction r_{c }of the first available subwindow B_{i, s} _{ i } _{ j }, the method 600 terminates in step 628.
 In yet another embodiment, once the amount of data to be processed is determined, specific tuples may be selected for processing from within each sliding window based on the join direction of a data stream join operation. The “direction” of a data stream join operation is defined by the numbers of tuples that are processed from each input data stream (e.g., if more tuples are being processed from a first data stream S_{1 }than a second data stream, S_{2}, the join is in the direction of the second data stream S_{2}). Because of the time delay difference between data streams, one direction of a data stream join operation may be more valuable than the opposite direction. For example, comparing a single tuple from the first data stream S_{1 }against many tuples from a second data stream S_{2 }may produce more usable output than the converse operation. Thus, in this case, load shedding should be performed in the converse direction (e.g., more tuples should be shed from the first sliding window W_{1 }than the second sliding window W_{2}).

FIG. 7 is a flow diagram illustrating one embodiment of a method 700 for selecting tuples for processing, e.g., in accordance with step 212 of the method 200. Specifically, the method 700 determines the individual fractions, r_{1 }and r_{2}, that should be applied, respectively, to process a fraction of the tuples in first and second sliding windows W_{1 }and W_{2}. This optimizes the direction of the join operation in order to maximize output.  The method 700 is initialized at step 702 and proceeds to step 704, where the method 700 sets the fraction r_{1 }of the first sliding window W_{1 }to be processed to one. The method 700 also sets the fraction r_{2 }of the second sliding window W_{2 }to be processed to one. The method 700 then proceeds to step 706 and computes a generic r value, e.g., in accordance with the method 300.
 In step 708, the method 700 computes the expected numbers of output tuples o_{1 }and o_{2 }to be produced, respectively, by the first and second sliding windows W_{1 }and W_{2}. In one embodiment, the values for o_{1 }and o_{2 }calculated as:

$\begin{array}{cc}{o}_{i}=\frac{1}{{n}_{i}}*\sum _{j=1}^{{n}_{i}}\ue89e{o}_{i,j}& \left(\mathrm{EQN}.\phantom{\rule{0.8em}{0.8ex}}\ue89e5\right)\end{array}$  where i indicates the specific sliding window W_{1 }or W_{2 }for which the expected number of output tuples is being calculated (e.g., i being 1 or 2 in this example), n_{i }is the total number of subwindows in the sliding window (e.g., W_{1 }or W_{2}) under consideration, and j indicating any subwindow 1n within the sliding window W_{1 }or W_{2 }under consideration.
 Once the expected numbers of output tuples o_{1 }and o_{2 }are calculated for each sliding window W_{1 }and W_{2}, the method 700 proceeds to step 710 and inquires if o_{1}≧o_{2}. If the method 700 determines that o_{1 }is greater than or equal to o_{2}, the method 700 proceeds to step 712 and resets r_{1 }to the smaller of one and

$r*\frac{{w}_{1}+{w}_{2}}{{w}_{1}}.$  Alternatively, if the method 700 determines in step 710 that o_{1 }is not greater than or equal to o_{2}, the method 700 proceeds to step 714 and resets r_{1 }to the larger of zero and

$r*\phantom{\rule{0.3em}{0.3ex}}\ue89e\frac{{w}_{1}+{w}_{2}}{{w}_{1}}1*\frac{{w}_{2}}{{w}_{1}}.$  In step 716, once the new value for r_{1 }has been computed, the method 700 calculates a new value for r_{2}. In one embodiment, r_{2 }is calculated as:

$\begin{array}{cc}{r}_{2}=r*\frac{{w}_{1}+{w}_{2}}{{w}_{1}}{r}_{1}*\frac{{w}_{1}}{{w}_{2}}& \left(\mathrm{EQN}.\phantom{\rule{0.8em}{0.8ex}}\ue89e6\right)\\ \mathrm{such}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{that}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89er*\left({w}_{1}+{w}_{2}\right)={r}_{1}*{w}_{1}+{r}_{2}*{w}_{2}& \left(\mathrm{EQN}.\phantom{\rule{0.8em}{0.8ex}}\ue89e7\right)\end{array}$  the method 700 then terminates in step 718.
 As described herein, the methods 500, 600 and 700 are aimed at maximizing the number of output tuples, o_{i,j}, generated by a data stream processing operation given limited processing resources. However, for some data stream processing operations, it may be desirable to maximize not just the quantity, but the value of the output data. Thus, in one embodiment, each tuple received via an input data stream is associated with an importance value, which is defined by the type of tuple and specified by a utility value attached to that type of tuple.
 In one embodiment, the type of a tuple, y, is defined as Z(y)=zεZ. The utility value of the same tuple, y, is thus defined as V(Z(y))=V(z). In one embodiment, type and utility value parameters are set based on application needs. For example, in news matching applications (e.g., where tuples representing news items from two or more different sources are matched), tuples representing news items can be assigned utility values from the domain [1, . . . , 10], where a value of 10 is assigned to the tuples representing the most important news items. Moreover, the frequency of appearance of a tuple of type z in an input data stream S_{i }is denoted as f_{i,z}.
 Thus, in one embodiment, load shedding may be performed in a manner that sheds a proportionally smaller number of tuples of types that provide higher output utilities. The extra processing resources that are allocated to process these highoutput utility tuple types are balanced by shedding a proportionally larger number of tuple types having low output utilities. This can be accomplished by applying different processing fractions, r_{i,z}, to different types of tuples, based on the output utilities of those types of tuples. In one embodiment, the expected output utility obtained from comparing a tuple y of type z from a first sliding window W_{1 }with a tuple in a second sliding window W_{2 }is denoted as u_{i,z }and is used to determine r_{i,z }values.
 The computation of r_{i,z }can be formulated as a fractional knapsack problem having a greedy optimal solution. For example, consider I_{i,j,z }as an item that represents the processing of a tuple y of type z (from the first sliding window W_{1}) against subwindow B_{i,j }of the second sliding window W_{2}. Item I_{i,j,z }has a volume of λ_{1}*λ_{2}*w_{i,z*}b units and a value of λ_{1}*λ_{2}*w_{i,z}*u_{i,z}*b*p_{i, s} _{ i } _{ j }units, where p_{i,j }denotes the probability of a match for subwindow B_{i,j}. Thus, the goal is to select a maximum number of items, where fractional items are acceptable, so that the total value is maximized and the total volume of the selected items is at most λ_{1}*λ_{2}*r*(w_{1}+w_{2}). Here, r_{i,j,z}ε[0, . . . , 1] is used to denote how much of item I_{i,j,z }is selected. Those skilled in the art will appreciate that the number of unknown variables (e.g., r_{i,j,z}′s) can be calculated as (B_{1,n}+B_{2,n})*Z, and the solution of the original problem (e.g., determining a value for r_{i,z}) can be calculated from these variables as:

$\begin{array}{cc}{r}_{i,z}=\sum _{j\in \left[1,\phantom{\rule{0.8em}{0.8ex}}\ue89e\dots \ue89e\phantom{\rule{0.8em}{0.8ex}},n\right]}\ue89e\mathrm{ri},j,z& \left(\mathrm{EQN}.\phantom{\rule{0.8em}{0.8ex}}\ue89e7\right)\end{array}$  In one embodiment, the values of the fraction variables (e.g., r_{i,j,z}′s) are determined during a join direction adaptation (e.g., as described in the method 700). In one embodiment, a simple way to do this is to sort the items I_{i,j,z }based on their respective value over volume ratios, u_{i,z}, *p_{i,s} _{ i } _{ j }, and to select as much as possible of the item I_{i,j,z }that is most valuable per unit volume. However, since the total number of items I_{i,j,z }may be large, this sorting can be costly in terms of processing resources, especially for a large number of subwindows and larger sized tuple types.
 Thus, in another embodiment, use is made of the s_{i} ^{j }values that define an order between value over volume ratios of items I_{i,j,z }for a fixed type z and sliding window W_{i}. Items I_{i,j,z }representing different data streams S and different types z with the highest value over volume ratios are maintained in a heap H. An item I_{i,j,z }is iteratively selected from the heap H and replaced with an item I_{i,j,z }having the next highest value over volume ratio with the same data stream and same type subscript index. This iterative process repeats until a capacity constraint is reached.

FIG. 8 is a flow diagram illustrating one embodiment of a method 800 for selecting tuples for processing, e.g., in accordance with step 212 of the method 200. Specifically, the method 800 selects tuples for processing based not only on an optimal join direction, but also on the respective values of the tuples as discussed above.  The method 800 is initialized at step 802 and proceeds to step 804, where the method 800 calculates a fraction, r, of the tuples to be processed (e.g., in accordance with the method 300) and also establishes a heap, H.
 The method 800 then proceeds to step 806 and sets an initial value of r_{i,z }to zero and an initial value of ν_{i}, S_{i} ^{1} _{,z }to u_{i,z}*p_{i}, s_{i} ^{1}, where ν_{i}, s_{i} ^{1} _{z }is the value over volume ratio of the item I_{i}, s_{i} ^{1} _{,z}. In step 808, the method 800 initializes the heap, H, with ν_{i}, s_{i} ^{1} _{,z}iε[1, . . . , 2], zεZ] and sets the total number of tuples to be processed, a, to a=*λ_{1}*λ_{2}*r*(w_{1}+w_{2}).
 Once the heap, H, has been initialized and the number of tuples to be processed, a, set, the method 800 proceeds to step 810 and inquires if the heap, H, is empty. If the method 800 concludes that the heap, H, is empty, the method 800 terminates in step 824.
 Alternatively, if the method 800 determines in step 810 that the heap, H, is not empty, the method 800 proceeds to step 812 and selects the first (e.g., topmost) item from the heap, H. Moreover, based on the selection of the first item, the method 800 adjusts the total number of tuples, a, that can still be processed. In one embodiment, the total number of tuples, a, is now a−(w_{i,z}*λ_{1}*λ_{2}*b).
 The method 800 then proceeds to step 814 and inquires if the adjusted value of a is still greater than zero. If the method 800 concludes that a is not greater than zero (e.g., no more tuples can be processed after subtracting the first item from the heap, H), the method 800 proceeds to step 816 and adjusts the fraction r_{c }of the first available subwindow to be processed such that:

$\begin{array}{cc}{r}_{e}=1+\frac{a}{{\lambda}_{1}*{\lambda}_{2}*{w}_{i,z}*b}& \left(\mathrm{EQN}.\phantom{\rule{0.8em}{0.8ex}}\ue89e8\right)\end{array}$  Moreover, the method 800 resets r_{i,z }to

${r}_{i,z}+\frac{{r}_{e}}{n}.$  The method 800 then terminates in step 824.
 Alternatively, if the method 800 determines in step 814 that a is greater than zero (e.g., tuples remain available for processing after subtracting the first item from the heap, H), the method 800 proceeds to step 818 and resets r_{i,z }to

${r}_{i,z}+\frac{1}{n}.$  The method 800 then proceeds to step 820 and determines whether the current subwindow, j, from which the last processed item was taken is the last subwindow, n (e.g., whether j<n) in the sliding window under examination. If the current subwindow j is not the last subwindow, n (e.g., if j<n), then the method 800 proceeds to step 822 and sets ν_{i}, s_{i} ^{j+1} _{,z}=u_{i,z}*p_{i}, s_{i} ^{1 }and inserts ν_{i}, s_{i} ^{j+1 }into the heap, H. The method 800 then returns to step 810 and proceeds as described above. Alternatively, if the method 800 determines in step 820 than the current subwindow, j, is the last subwindow, n (e.g., j=n), the method 800 bypasses step 822 and returns directly to step 810.
 Thus, the present invention represents a significant advancement in the field of data stream processing. The present invention allows all incoming data streams to be received in memory, but selects only a subset of the tuples contained within the received data streams for processing, based on available processing resources and on one or more characteristics of the subset of tuples. The invention thus makes it possible for load shedding to be performed in an “intelligent” (e.g., nonarbitrary) manner, thereby maximizing the quality of the data stream operation output.
 While foregoing is directed to the preferred embodiment of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.
Claims (18)
1. A method for processing data streams, the method comprising:
receiving at least a first data stream into at least a first sliding window of memory;
selecting tuples from said at least a first data stream for processing in accordance with at least one data stream operation, where said tuples that are selected represent a subset of all tuples contained within said at least a first sliding window; and
ignoring tuples from said at least a first data stream that are not selected for processing.
2. The method of claim 1 , wherein said selecting comprises:
determining a total number of tuples to be selected for processing; and
selecting specific tuples for processing in accordance with said total number of tuples.
3. The method of claim 2 , wherein said determining is based at least in part on available processing resources.
4. The method of claim 2 , wherein said determining adapts to a rate at which said at least a first data stream is received.
5. The method of claim 4 , wherein said determining comprises:
calculating a fraction of said at least a first sliding window, said fraction indicating how much of said at least a first sliding window can be selected for processing.
6. The method of claim 5 , wherein said calculating comprises:
counting a first number of tuples, said first number of tuples representing a number of tuples selected from said at least a first sliding window for processing in a period of time;
counting a second number of tuples, said second number of tuples representing a number of tuples received by said at least a first sliding window in said period of time; and
basing said fraction, at least in part, on a ratio of said first number of tuples to said second number of tuples.
7. A computer readable storage device containing an executable program for processing data streams, where the program performs steps of:
receiving at least a first data stream into at least a first sliding window of memory;
selecting tuples from said at least a first data stream for processing in accordance with at least one data stream operation, where said tuples that are selected represent a subset of all tuples contained within said at least a first sliding window; and
ignoring tuples from said at least a first data stream that are not selected for processing.
8. The computer readable storage device of claim 7 , wherein said selecting comprises:
determining a total number of tuples to be selected for processing; and
selecting specific tuples for processing in accordance with said total number of tuples.
9. The computer readable storage device of claim 8 , wherein said determining is based at least in part on available processing resources.
10. The computer readable storage device of claim 8 , wherein said determining adapts to a rate at which said at least a first data stream is received.
11. The computer readable storage device of claim 10 , wherein said determining comprises:
calculating a fraction of said at least a first sliding window, said fraction indicating how much of said at least a first sliding window can be selected for processing.
12. The computer readable storage device of claim 11 , wherein said calculating comprises:
counting a first number of tuples, said first number of tuples representing a number of tuples selected from said at least a first sliding window for processing in a period of time;
counting a second number of tuples, said second number of tuples representing a number of tuples received by said at least a first sliding window in said period of time; and
basing said fraction, at least in part, on a ratio of said first number of tuples to said second number of tuples.
13. An apparatus comprising:
means for receiving at least a first data stream into at least a first sliding window of memory;
means for selecting tuples from said at least a first data stream for processing in accordance with at least one data stream operation, where said tuples that are selected represent a subset of all tuples contained within said at least a first sliding window; and
means for ignoring tuples from said at least a first data stream that are not selected for processing.
14. The apparatus of claim 13 , wherein said means for selecting comprises:
means for determining a total number of tuples to be selected for processing; and
means for selecting specific tuples for processing in accordance with said total number of tuples.
15. The apparatus of claim 14 , wherein said means for determining makes a determination based at least in part on available processing resources.
16. The apparatus of claim 14 , wherein said means for determining adapts to a rate at which said at least a first data stream is received.
17. The apparatus of claim 16 , wherein said means for determining comprises:
means for calculating a fraction of said at least a first sliding window, said fraction indicating how much of said at least a first sliding window can be selected for processing.
18. The apparatus of claim 17 , wherein said means for calculating comprises:
means for counting a first number of tuples, said first number of tuples representing a number of tuples selected from said at least a first sliding window for processing in a period of time;
means for counting a second number of tuples, said second number of tuples representing a number of tuples received by said at least a first sliding window in said period of time,
wherein said fraction is based, at least in part, on a ratio of said first number of tuples to said second number of tuples.
Priority Applications (3)
Application Number  Priority Date  Filing Date  Title 

US11/068,137 US7610397B2 (en)  20050228  20050228  Method and apparatus for adaptive load shedding 
US12/165,524 US8117331B2 (en)  20050228  20080630  Method and apparatus for adaptive load shedding 
US13/342,487 US20120124233A1 (en)  20050228  20120103  Method and apparatus for adaptive load shedding 
Applications Claiming Priority (1)
Application Number  Priority Date  Filing Date  Title 

US13/342,487 US20120124233A1 (en)  20050228  20120103  Method and apparatus for adaptive load shedding 
Related Parent Applications (1)
Application Number  Title  Priority Date  Filing Date  

US12/165,524 Division US8117331B2 (en)  20050228  20080630  Method and apparatus for adaptive load shedding 
Publications (1)
Publication Number  Publication Date 

US20120124233A1 true US20120124233A1 (en)  20120517 
Family
ID=36933087
Family Applications (3)
Application Number  Title  Priority Date  Filing Date 

US11/068,137 Expired  Fee Related US7610397B2 (en)  20050228  20050228  Method and apparatus for adaptive load shedding 
US12/165,524 Expired  Fee Related US8117331B2 (en)  20050228  20080630  Method and apparatus for adaptive load shedding 
US13/342,487 Abandoned US20120124233A1 (en)  20050228  20120103  Method and apparatus for adaptive load shedding 
Family Applications Before (2)
Application Number  Title  Priority Date  Filing Date 

US11/068,137 Expired  Fee Related US7610397B2 (en)  20050228  20050228  Method and apparatus for adaptive load shedding 
US12/165,524 Expired  Fee Related US8117331B2 (en)  20050228  20080630  Method and apparatus for adaptive load shedding 
Country Status (1)
Country  Link 

US (3)  US7610397B2 (en) 
Cited By (1)
Publication number  Priority date  Publication date  Assignee  Title 

US9305031B2 (en)  20130417  20160405  International Business Machines Corporation  Exiting windowing early for stream computing 
Families Citing this family (14)
Publication number  Priority date  Publication date  Assignee  Title 

US8572274B2 (en) *  20110217  20131029  International Business Machines Corporation  Estimating load shed data in streaming database applications 
KR20120122136A (en) *  20110428  20121107  삼성전자주식회사  A method of controlling a load shedding for data stream management system and an apparatus therefor 
JP5862245B2 (en)  20111130  20160216  富士通株式会社  Deployment device placement program and arrangement method 
CN103780741B (en) *  20121018  20180313  腾讯科技（深圳）有限公司  Tip speed method and mobile device 
US9756184B2 (en)  20121108  20170905  Genesys Telecommunications Laboratories, Inc.  System and method of distributed maintenance of contact center state 
US9900432B2 (en)  20121108  20180220  Genesys Telecommunications Laboratories, Inc.  Scalable approach to agentgroup state maintenance in a contact center 
US20140143373A1 (en) *  20121120  20140522  Barinov Y. Vitaly  Distributed Aggregation for Contact Center AgentGroups On Growing Interval 
US9477464B2 (en)  20121120  20161025  Genesys Telecommunications Laboratories, Inc.  Distributed aggregation for contact center agentgroups on sliding interval 
US9195559B2 (en)  20121212  20151124  International Business Machines Corporation  Management of stream operators with dynamic connections 
US9087082B2 (en) *  20130307  20150721  International Business Machines Corporation  Processing control in a streaming application 
US9578171B2 (en)  20130326  20170221  Genesys Telecommunications Laboratories, Inc.  Low latency distributed aggregation for contact center agentgroups on sliding interval 
US9515965B2 (en)  20130918  20161206  International Business Machines Corporation  Managing data paths in an operator graph 
US9298801B2 (en)  20130925  20160329  International Business Machines Corporation  Managing multiple windows on an operator graph 
US9313110B2 (en)  20140122  20160412  International Business Machines Corporation  Managing processing branches in an operator graph 
Citations (13)
Publication number  Priority date  Publication date  Assignee  Title 

US6339772B1 (en) *  19990706  20020115  Compaq Computer Corporation  System and method for performing database operations on a continuous stream of tuples 
US6502089B1 (en) *  19991117  20021231  International Business Machines Corporation  Generating restriction queries using tensor representations 
US6507834B1 (en) *  19991222  20030114  Ncr Corporation  Method and apparatus for parallel execution of SQL from stored procedures 
US6564204B1 (en) *  20000414  20030513  International Business Machines Corporation  Generating join queries using tensor representations 
US6728694B1 (en) *  20000417  20040427  Ncr Corporation  Set containment join operation in an object/relational database management system 
US20040148420A1 (en) *  20020918  20040729  Netezza Corporation  Programmable streaming data processor for database appliance having multiple processing unit groups 
US7010538B1 (en) *  20030315  20060307  Damian Black  Method for distributed RDSMS 
US7031928B1 (en) *  20001002  20060418  HewlettPackard Development Company, L.P.  Method and system for throttling I/O request servicing on behalf of an I/O request generator to prevent monopolization of a storage device by the I/O request generator 
US7257515B2 (en) *  20040303  20070814  HewlettPackard Development Company, L.P.  Sliding window for alert generation 
US7328220B2 (en) *  20041229  20080205  Lucent Technologies Inc.  Sketchbased multiquery processing over data streams 
US7668856B2 (en) *  20040930  20100223  AlcatelLucent Usa Inc.  Method for distinct count estimation over joins of continuous update stream 
US7716215B2 (en) *  20031031  20100511  International Business Machines Corporation  System, method, and computer program product for progressive query processing 
US7882100B2 (en) *  20050124  20110201  Sybase, Inc.  Database system with methodology for generating bushy nested loop join trees 
Family Cites Families (13)
Publication number  Priority date  Publication date  Assignee  Title 

US6859496B1 (en) *  19980529  20050222  International Business Machines Corporation  Adaptively encoding multiple streams of video data in parallel for multiplexing onto a constant bit rate channel 
GB2351572B (en) *  19990626  20020206  Univ York  Data procesors 
AU7176101A (en) *  20000629  20020114  Univ Columbia  Method and system for analyzing multidimensional data 
US7162698B2 (en) *  20010717  20070109  Mcafee, Inc.  Sliding window packet management systems 
AT314783T (en) *  20011123  20060115  Nokia Corp  Method and system for the treatment of network congestion 
US20030236904A1 (en) *  20020619  20031225  Jonathan Walpole  Priority progress multicast streaming for qualityadaptive transmission of data 
US7313092B2 (en) *  20020930  20071225  Lucent Technologies Inc.  Apparatus and method for an overload control procedure against denial of service attack 
US7454416B2 (en) *  20030430  20081118  International Business Machines Corporation  Method for aggregation subquery join elimination 
GB0310689D0 (en) *  20030509  20030611  Ibm  Monitoring operational data in data processing systems 
US7698267B2 (en) *  20040827  20100413  The Regents Of The University Of California  Searching digital information and databases 
US7765221B2 (en) *  20040930  20100727  Sap Ag  Normalization of a multidimensional set object 
US7383253B1 (en) *  20041217  20080603  Coral 8, Inc.  Publish and subscribe capable continuous query processor for realtime data streams 
US7493346B2 (en) *  20050216  20090217  International Business Machines Corporation  System and method for load shedding in data mining and knowledge discovery from stream data 

2005
 20050228 US US11/068,137 patent/US7610397B2/en not_active Expired  Fee Related

2008
 20080630 US US12/165,524 patent/US8117331B2/en not_active Expired  Fee Related

2012
 20120103 US US13/342,487 patent/US20120124233A1/en not_active Abandoned
Patent Citations (16)
Publication number  Priority date  Publication date  Assignee  Title 

US6339772B1 (en) *  19990706  20020115  Compaq Computer Corporation  System and method for performing database operations on a continuous stream of tuples 
US6502089B1 (en) *  19991117  20021231  International Business Machines Corporation  Generating restriction queries using tensor representations 
US6507834B1 (en) *  19991222  20030114  Ncr Corporation  Method and apparatus for parallel execution of SQL from stored procedures 
US6564204B1 (en) *  20000414  20030513  International Business Machines Corporation  Generating join queries using tensor representations 
US6728694B1 (en) *  20000417  20040427  Ncr Corporation  Set containment join operation in an object/relational database management system 
US7031928B1 (en) *  20001002  20060418  HewlettPackard Development Company, L.P.  Method and system for throttling I/O request servicing on behalf of an I/O request generator to prevent monopolization of a storage device by the I/O request generator 
US7698338B2 (en) *  20020918  20100413  Netezza Corporation  Field oriented pipeline architecture for a programmable data streaming processor 
US20040148420A1 (en) *  20020918  20040729  Netezza Corporation  Programmable streaming data processor for database appliance having multiple processing unit groups 
US7730077B2 (en) *  20020918  20100601  Netezza Corporation  Intelligent storage device controller 
US7010538B1 (en) *  20030315  20060307  Damian Black  Method for distributed RDSMS 
US8078609B2 (en) *  20030315  20111213  SQLStream, Inc.  Method for distributed RDSMS 
US7716215B2 (en) *  20031031  20100511  International Business Machines Corporation  System, method, and computer program product for progressive query processing 
US7257515B2 (en) *  20040303  20070814  HewlettPackard Development Company, L.P.  Sliding window for alert generation 
US7668856B2 (en) *  20040930  20100223  AlcatelLucent Usa Inc.  Method for distinct count estimation over joins of continuous update stream 
US7328220B2 (en) *  20041229  20080205  Lucent Technologies Inc.  Sketchbased multiquery processing over data streams 
US7882100B2 (en) *  20050124  20110201  Sybase, Inc.  Database system with methodology for generating bushy nested loop join trees 
Cited By (3)
Publication number  Priority date  Publication date  Assignee  Title 

US9305031B2 (en)  20130417  20160405  International Business Machines Corporation  Exiting windowing early for stream computing 
US9330118B2 (en)  20130417  20160503  International Business Machines Corporation  Exiting windowing early for stream computing 
US9641586B2 (en)  20130417  20170502  International Business Machines Corporation  Exiting windowing early for stream computing 
Also Published As
Publication number  Publication date 

US8117331B2 (en)  20120214 
US20060195599A1 (en)  20060831 
US20090049187A1 (en)  20090219 
US7610397B2 (en)  20091027 
Similar Documents
Publication  Publication Date  Title 

US8185909B2 (en)  Predictive database resource utilization and load balancing using neural network model  
US8510374B2 (en)  Polling protocol for automatic load limiting  
Rahman et al.  A dynamic critical path algorithm for scheduling scientific workflow applications on global grids  
US6098052A (en)  Credit card collection strategy model  
US8352607B2 (en)  Colocation and offloading of web site traffic based on traffic pattern recognition  
US8543711B2 (en)  System and method for evaluating a pattern of resource demands of a workload  
US6144639A (en)  Apparatus and method for congestion control in high speed networks  
US20040230675A1 (en)  System and method for adaptive admission control and resource management for service time guarantees  
US20050108380A1 (en)  Capacity planning for server resources  
JP2720910B2 (en)  Apparatus and method for managing the workload of the data processing system  
JP5212381B2 (en)  Feedforward control method, service providing quality control apparatus, system, program, and the record medium  
US7818417B2 (en)  Method for predicting performance of distributed stream processing systems  
US9183058B2 (en)  Heuristicsbased scheduling for data analytics  
US20040093351A1 (en)  System and method for controlling task assignment and work schedules  
US9720941B2 (en)  Fully automated SQL tuning  
US6842783B1 (en)  System and method for enforcing communications bandwidth based service level agreements to plurality of customers hosted on a clustered web server  
Duffield et al.  Learn more, sample less: control of volume and variance in network measurement  
US7721288B2 (en)  Organizing transmission of repository data  
US20100067379A1 (en)  Picking an optimal channel for an access point in a wireless network  
EP1993231A1 (en)  Allocation method, system and device for network resource in communication network  
Manjhi et al.  Finding (recently) frequent items in distributed data streams  
US7437169B2 (en)  System and method for optimizing network communication in response to network conditions  
US8001277B2 (en)  Determining, transmitting, and receiving performance information with respect to an operation performed locally and at remote nodes  
Mandjes  Pricing strategies under heterogeneous service requirements  
EP1008938A2 (en)  Method of analysing delay factor in job system 
Legal Events
Date  Code  Title  Description 

AS  Assignment 
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GEDIK, BUGRA;WU, KUNLUNG;YU, PHILIP S.;SIGNING DATES FROM 20050223 TO 20050225;REEL/FRAME:031858/0083 

STCB  Information on status: application discontinuation 
Free format text: ABANDONED  FAILURE TO PAY ISSUE FEE 