US9405858B2 - On-the-fly encoding method for efficient grouping and aggregation - Google Patents

On-the-fly encoding method for efficient grouping and aggregation Download PDF

Info

Publication number
US9405858B2
US9405858B2 US14/070,990 US201314070990A US9405858B2 US 9405858 B2 US9405858 B2 US 9405858B2 US 201314070990 A US201314070990 A US 201314070990A US 9405858 B2 US9405858 B2 US 9405858B2
Authority
US
United States
Prior art keywords
values
query
set
system
un
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US14/070,990
Other versions
US20140372470A1 (en
Inventor
Gopi K. Attaluri
Ronald J. Barber
Vincent KULANDAISAMY
Sam S. Lightstone
Guy M. Lohman
Ippokratis Pandis
Vijayshankar Raman
Richard S. Sidle
Liping Zhang
Naresh Chainani
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US13/918,302 priority Critical patent/US9471710B2/en
Application filed by International Business Machines Corp filed Critical International Business Machines Corp
Priority to US14/070,990 priority patent/US9405858B2/en
Assigned to INTERNATIONAL BUSINESS MACHINES CORPORATION reassignment INTERNATIONAL BUSINESS MACHINES CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: LIGHTSTONE, SAM S., KULANDAISAMY, VINCENT, PANDIS, IPPOKRATIS, SIDLE, RICHARD S., BARBER, RONALD J., RAMAN, VIJAYSHANKAR, CHAINANI, NARESH, ATTALURI, GOPI K., ZHANG, LIPING, LOHMAN, GUY M.
Publication of US20140372470A1 publication Critical patent/US20140372470A1/en
Application granted granted Critical
Publication of US9405858B2 publication Critical patent/US9405858B2/en
Application status is Active legal-status Critical
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G06F17/30979
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/903Querying
    • G06F16/90335Query processing
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/242Query formulation
    • G06F16/2428Query predicate definition using graphical user interfaces, including menus and forms
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • G06F16/24553Query execution of query operations
    • G06F16/24558Binary matching operations
    • G06F16/2456Join operations
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • G06F16/258Data format conversion from or to a database
    • G06F17/30398

Abstract

Embodiments include a system for encoding data while it is being processed. The system includes a processor, an encoder and a decoder. The processor is configured to process a query request by determining a set of values. The encoder is configured for encoding the set of values, such that a subsequent processing operation can be performed on the encoded values. The processor performs the subsequent processing operations. The decoder is configured for decoding each value back to its value prior to being encoded upon completion of the processor completing the requested query.

Description

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No. 13/918,302, filed Jun. 14, 2013, the content of which is incorporated by reference herein in its entirety.

BACKGROUND

The present invention relates generally to relational databases, and more specifically to relational databases that perform aggregation during run-time operations.

A database is a collection of information organized in such a way that a computer program can quickly select desired pieces of data. It may be organized by fields, records, and files or through use of linking such as hypertext. In some embodiments, to access information from a database, a database management system (DBMS) is also used, which may include a collection of programs for organizing and storing data.

Many database tables are organized in a matrix type structure, that is, made of rows and columns. The intersection of a row with a column is defined as a cell that holds data. In many relational databases, a hash table is used. A hash table is a data structure used to implement an associative array which often can map keys to values. A hash table uses a hash function to compute an index into an array of buckets from which the correct value can be found. Ideally a hash function assigns each possible key to each bucket.

Management of today's relational databases, especially those that use hash tables, is complex. One reason is because data retrieval is performed by initiating a function that most often results in a query operation being performed. Therefore the ability to handle a large quantity of query operations at one time in a time efficient manner is necessary. Unfortunately, designing data management systems can be challenging because, while storage and operational footprint remains limited or is being actively reduced, the size of the databases keep growing. Under such conditions, performing runtime operations on these large databases with limited resources either requires additional cost or is time prohibitive.

SUMMARY

A system for encoding data while it is being processed includes a processor configured to process a query request by determining a set of values; an encoder for encoding the set of values such that a subsequent processing operation can be performed on the encoded values, the processor performing the subsequent processing operations; and a decoder for decoding each value back to its value prior to being encoded upon completion of the processor completing the requested query.

Additional features and advantages are realized through the techniques of the present disclosure. Other embodiments and aspects of the disclosure are described in detail herein. For a better understanding of the disclosure with the advantages and the features, refer to the description and to the drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other features, and advantages of the disclosure are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:

FIG. 1 depicts an example of a data tables that can be used in accordance with one embodiment of the present invention;

FIG. 2 depicts another example of a table that can be used in accordance with an embodiment;

FIG. 3 depicts a process flow for performing an aggregation operation in accordance with an embodiment; and

FIG. 4 is a block diagram illustration of a computing environment as can be utilized in conjunction with different embodiments.

DETAILED DESCRIPTION

In computing environments, databases are a collection of stored data that is logically related. Many databases include one or more tables that can be related to one another. Each table is made up of a matrix of rows and columns. Each row represents an entry that can vary in the scope and nature of stored information it provides.

A relational database is a popular form of database used for data management. Many of these relational databases use hash tables to optimize time and operational resources. A standard database-query language is the structured query language, referenced as SQL. SQL is used often with relational databases that also utilize hash tables. In SQL, to retrieve data or update data included in the relational table, queries are used.

A popular database operation is an aggregation operation. Aggregation is an abstraction which turns a relationship between objects into an aggregate object, based on performing a grouping request. The group-by operation involved in aggregation often involves an associated join operation as well to establish the grouped objects as a consolidated result.

Besides data aggregation, join and group-by operation are also important operations. Group-by operation result in the actual groupings of one or more group of data as preselected by an aggregation request or query. Join operation can be performed to combine all groups into a single final desired list or document. The problem, however, with large databases is that group-by and join operations typically consume a lot of machine resources, and thus provide a challenge with limited footprints and other associated time constraints.

Columns used in group-by are typically categorical attributes that include long strings of data or data types (hereinafter wide), and group-by clause can have many columns. Also, these columns are usually accessed from dimension tables. In data warehousing, a dimension table contains descriptive attributes that are typically textual in nature and are used for query constraining and/or filtering. In both join and group-by operations, this increases the width of the hash table, sometimes forcing data storage to be spilled to other storage devices such as external disks. Consequently, where wide group-by tables are used, the operational structure causes slowdown in both join and group-by related operations.

In addition, in performing a join of multiple tables, the intermediate results are typically stored in a spool table. In some cases, the join of multiple tables (such as a cross-product join) can generate a large amount of data. As a result, a spool space problem may be encountered if the spool table becomes too big. Consequently, database system performance may suffer.

It is conceivable to reduce the width of the group-by columns by first running the group-by function on the foreign key column first and then do a join and finally end the operation by performing a second grouping on the dimension columns. This presents several problems, however. For example, performing the first grouping can be prohibitively expensive because there may be too many groups involved. This method may also not be applicable to some aggregate operations.

An alternative is using a group-by operation that uses encoded data in order to reduce the width. However, this method also presents its own set of problems. For example, one drawback of this method is that it does not work well on unencoded data or expressions. Therefore, other ways are needed to encode them, which is especially expensive in some multiprocessing environments such as those using a non-uniform memory access (NUMA) architecture.

FIG. 1 shows examples of illustrating a plurality of tables to be used in accordance with one embodiment. In FIG. 1 a first table 110 and a second table 120 are depicted each having multiple rows and columns. The entries of middle rows and columns (depicted by periods) are omitted for ease of illustration to show that just a few or a lengthy number of rows can be stored in a memory or accumulated during run-time.

The two illustrated tables of FIG. 1 are related to one another in that they both display information about employees of the MXZ company. Table 1 provides information about the employee name (column 112), serial number (column 114) and location (column 116), while Table 2 still identifies the employees (column 122) but also provide information about their positions (column 115) and salaries (column 117).

In one example, it may be desired to use a predicate on the result of a join operation about each employee that earns more than $50,000 annual salary in City A by position. Therefore, the available data will be searched by the preselected grouping and searched based on that grouping predicate. In this case, an employee's name, salary amount and location are considered as grouping predicates. In this case, accessing both tables is required, since salaries and work locations are in two different tables. Once the search is completed, the results are accumulated as a single document.

FIG. 2 provides another example showing a plurality of tables that can be used in accordance with one embodiment. In tables 210 and 220 of FIG. 2, information about customer names, account numbers, and their associated cities, states and zip codes are provided alongside with the amount of payment due for each customer.

In one example, a query is then requested that does a grouping on customer city, state and zip code. In this example, the overall group-by width may be 50 to 100 bytes long. However, the number of distinct groups will be less than 50000, suggesting that a limited binary code with a limited or compact length can suffice to encode the data to be accessed based upon the query's criteria. This encoding can both be used subsequently for joining and grouping as well. In this particular example, 2 bytes (16-bits) of transmission/storage space can suffice for the grouping code used in this example.

In one embodiment, an on-the-fly (OTF) coding is provided that can be used dynamically during runtime that can provide the suggested scheme as discussed in conjunction with FIG. 2. In such an embodiment, OTF coding can be provided to assign a compact encoding scheme having easy-to-manage codes. In one embodiment, to make the scheme even more efficient, the encoding scheme can be associated to the dimension payloads. The encoding and joining can occur during runtime. The scheme can have the codes cover only the actual values encountered after applying query predicates, and only for the valid combinations of the grouping columns encountered (so as to take advantage of correlation established). Consequently, both join and group-by performances are improved.

One example can now be used to make some of the presented concepts more clear. Suppose the query provides:

select . . . from Fact, Dim where . . . group by

Dim.A, Dim.B, Dim.C having . . . .

In one embodiment, a technique can be used that pushes the group-by and join operations so as to compute select uniqueId( ) from Dim group by A, B, C where (same predicates as in original query) to construct an encoding table for all the distinct groups. This is a group-by operation done not for computing/completing the aggregation process, but rather performed only to assign a unique code to each group combination. The unique code can be generated using the generate-unique or serial number functions. When the original query is performed during run-time operations, an extra join of the dimension is added with the result of the above group-by operation to convert A,B,C into encoded values. Subsequently, the join operation is completed (with the fact) and the grouping on the computed code is then performed such that even the having clause can be applied if desired. An alternative method that does not use a generate-unique or serial number function is to use some physical identifier associated with the records of Dim as the unique code. For instance, in many DBMSs each record of a table has a record identifier (RID). The RID of a characteristic record in each group can be used as the code for that group—e.g., instead of invoking generate_unique, compute the smallest or largest RID for the records in each group. Another variation that does not involve a separate group by query over Dim is to insert group values from Dim into a hash table as part of the regular query, and use a physical identifier from the hash table (e.g., the bucket offset in the hash table where a group value was inserted) as the unique code for that group.

In this example, the join operation can be performed to join with the lookup table that is computed above, to recapture the A, B, C values. This technique does not necessarily involve changes to the DBMS runtime, and it can be performed entirely via query rewrite operations in SQL. There is, however, a more efficient technique that modifies the DBMS runtime so that a faster lookup table can be implemented. For example, the query:

select from Fact, Dim Fact.fkey = Dim.key group by Dim.A, Dim.B, Dim.C having can be rewritten as with temp as select A,B,C, generate_ unique( ) as OTF_Code from Dim group by A,B,C ( select .. from Fact, (select Dim.key, temp.OTF_Code --- (1) from Dim, temp where Dim.a = temp.A and Dim.b = temp.B and Dim.C = temp.C  and other local predicates on Dim from the original  query )  group by temp.OTF_Code  having ...)


plus a final join with temp to get back the actual A,B,C values, if needed in the select clause. If the RID method is used, generate_unique will be replaced by min(RID) or max(RID) in the preceding SQL statement.

If other non-group by columns are also accessed from Dim, in this embodiment, they will also be part of the select clause in line (1). This can be used with other embodiments and arrangements. For example, in a snowflake query with multiple joins, this OTF code is computed separately for each subtree of the snowflake, to form an encoding for the grouping columns from that snowflake. As an optimization, it is possible to avoid this double scan of the dimension by doing an aggregation distinct during the dimension scan, with the distinct being on the key columns of the dimension (and non-group by columns from the dimension, if there are any). This allows the subsequent join with the fact to directly probe into the results of the aggregation distinct to access the join columns and not have to go back to scan the dimension table.

This scheme applies even when the grouping is done on expressions. For example, suppose one entry of the group-by clause is month(date). With OTF coding, a code that occupies only log(#distinct months) bits, which can be much smaller than the space taken up to store the full date or the unencoded string-valued month can be assigned. A scheme that just did group-by on encoded data would not have this benefit.

This scheme is particularly well suited to systems that lay out data in columnar format because the repeated dimension scans need only access the needed attributes. Furthermore, when the data is compressed with a global dictionary encoding, it can use the concatenation of the dictionary codes itself as the encoding for the group. However, if unencoded values are encountered, they need to be assigned a code by inserting those values into a hash table and essentially performing a group-by operation to assign the unique key, but only on unencoded grouping values.

FIG. 3 is an illustration of a flow diagram in accordance with one embodiment which depicts a technique for encoding data while it is being processed as part of a query. In blocks 310 and 320, upon receiving a query request, a set of values associated with data to be encoded for completing the query request is determined. This set of values may have been reduced by the earlier application of predicates, computation of expressions, or due to correlation among the values of different columns (such as city, state, and zip code). Proceeding to block, 330, these values are encoded such that any subsequent processing operations can be performed on the encoded values to complete the requested query. In block 340, after performing the subsequent processing operations to complete the requested query, each value is decoded back to its original value.

FIG. 4 is a block diagram illustration of a computing environment as per one embodiment of the present invention having a facilitated network system 400. As illustrated in FIG. 4, the system comprises a plurality of nodes 401 that are in processing communication with one another. This communication can be enabled through wired or wireless processing or other alternate means as can be appreciated by those skilled in the art. Each node 401 can also include one or more processors 430. Each node 401 can include its own internal memory as shown at 420, or be in processing communication with that node's internal memories, or in communication with one or more external memories such as the one shown in FIG. 4 at 480. The system memories 420 and 480 can also be in communication directly with one another or through the node and either be comprised of a single unit that is shared among the many nodes 201 or be comprised of a plurality of components that are in processing communication with one another. In this manner, each node 201 can either exclusively use a single memory or alternatively use multiple memories at the same time or at different times to achieve processing optimization. In one embodiment, one or nodes 401 or processors 430 can be used while in processing communication with one another one of the memories such as 420 or 480 to provide instructions for carrying out the techniques discussed above such as the processor identifying keys associated with values in a database.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosure. The embodiments were chosen and described in order to best explain the principles of the disclosure and the practical application, and to enable others of ordinary skill in the art to understand the disclosure for various embodiments with various modifications as are suited to the particular use contemplated.

Further, as will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method, or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

Aspects of the present disclosure are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Claims (13)

What is claimed is:
1. A system for encoding data, the system comprising:
a processor configured to
determine a first set of values for completing a query, the first set of values being associated with the data;
reduce the first set of values, wherein reducing the first set of values comprises:
a computing an expression;
correlating a first un-encoded value in the first set of values to a second un-encoded value in the first set of values based at least in part on the expression;
excluding the first un-encoded value from the first set of values to obtain the second set of values; and
process a query during run-time by determining that the query includes a query predicate and evaluating the query predicate to determine that the query predicate excludes a third un-encoded value in the second set of values and does not exclude the second un-encoded value in the second set of values;
an encoder configured to encode, during run-time and subsequent to evaluating the query predicate, the second un-encoded value to obtain an encoded value;
the processor being further configured to complete the query at least in part by performing at least one processing operation on the encoded value during run-time; and
a decoder configured to decode the encoded value to obtain the second un-encoded value upon completion of the query.
2. The system of claim 1, wherein at least part of the data is retrieved from a plurality of tables stored in a relational database.
3. The system of claim 2, wherein the at least one processing operation is selected from the group consisting of a join
operation and a grouping operation.
4. The system of claim 2, wherein the at least one processing operation includes a plurality of relational query processing operations.
5. The system of claim 1, wherein the second set of values is concatenated based on a plurality of attributes referenced in the query.
6. The system of claim 1, wherein the data includes data that is being transmitted dynamically during the execution of the query.
7. The system of claim 1, wherein the the encoder is configured to encode the second un-encoded value by determining that the second un-encoded value is a distinct value in the second set of values and assigning a distinct code to the second un-encoded value.
8. The system of claim 7, wherein the distinct code is provided by a function selected from the group consisting of a serial number function and a unique number generating function.
9. The system of claim 7, wherein the distinct code is generated using physical properties of a plurality of records containing the second un-encoded value.
10. The system of claim 9, wherein the physical properties include record identifiers of the plurality of records, and wherein the distinct code is a particular record identifier of a particular record of the plurality of records.
11. The system of claim 1, wherein at least one attribute of the data is stored and encoded using dictionary encoding.
12. The system of claim 11, wherein the at least one attribute includes a first attribute and a second attribute, and wherein the encoder is configured to encode a fourth un-encoded value in the second set of values by concatenating a first dictionary code of the first attribute and a second dictionary code of the second attribute.
13. The system of claim 1, wherein the encoder is configured to encode one or more un-encoded values in the second set of values by selecting a scheme optimal to the requested query.
US14/070,990 2013-06-14 2013-11-04 On-the-fly encoding method for efficient grouping and aggregation Active 2033-08-15 US9405858B2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US13/918,302 US9471710B2 (en) 2013-06-14 2013-06-14 On-the-fly encoding method for efficient grouping and aggregation
US14/070,990 US9405858B2 (en) 2013-06-14 2013-11-04 On-the-fly encoding method for efficient grouping and aggregation

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US14/070,990 US9405858B2 (en) 2013-06-14 2013-11-04 On-the-fly encoding method for efficient grouping and aggregation

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US13/918,302 Continuation US9471710B2 (en) 2013-06-14 2013-06-14 On-the-fly encoding method for efficient grouping and aggregation

Publications (2)

Publication Number Publication Date
US20140372470A1 US20140372470A1 (en) 2014-12-18
US9405858B2 true US9405858B2 (en) 2016-08-02

Family

ID=52020137

Family Applications (3)

Application Number Title Priority Date Filing Date
US13/918,302 Active 2034-04-23 US9471710B2 (en) 2013-06-14 2013-06-14 On-the-fly encoding method for efficient grouping and aggregation
US14/070,990 Active 2033-08-15 US9405858B2 (en) 2013-06-14 2013-11-04 On-the-fly encoding method for efficient grouping and aggregation
US15/150,493 Pending US20160253390A1 (en) 2013-06-14 2016-05-10 On-the-fly encoding method for efficient grouping and aggregation

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US13/918,302 Active 2034-04-23 US9471710B2 (en) 2013-06-14 2013-06-14 On-the-fly encoding method for efficient grouping and aggregation

Family Applications After (1)

Application Number Title Priority Date Filing Date
US15/150,493 Pending US20160253390A1 (en) 2013-06-14 2016-05-10 On-the-fly encoding method for efficient grouping and aggregation

Country Status (1)

Country Link
US (3) US9471710B2 (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160246810A1 (en) * 2015-02-25 2016-08-25 International Business Machines Corporation Query predicate evaluation and computation for hierarchically compressed data
US9665624B2 (en) 2013-01-30 2017-05-30 International Business Machines Corporation Join operation partitioning
US9672248B2 (en) 2014-10-08 2017-06-06 International Business Machines Corporation Embracing and exploiting data skew during a join or groupby
US9935650B2 (en) 2014-04-07 2018-04-03 International Business Machines Corporation Compression of floating-point data by identifying a previous loss of precision
US9959299B2 (en) 2014-12-02 2018-05-01 International Business Machines Corporation Compression-aware partial sort of streaming columnar data

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9317548B2 (en) 2013-01-30 2016-04-19 International Business Machines Corporation Reducing collisions within a hash table
US9367556B2 (en) 2013-06-14 2016-06-14 International Business Machines Corporation Hashing scheme using compact array tables
US9760599B2 (en) 2014-04-09 2017-09-12 International Business Machines Corporation Group-by processing for data containing singleton groups
US10176226B2 (en) * 2014-11-26 2019-01-08 Looker Data Sciences, Inc. Relation aware aggregation (RAA) on normalized datasets
US10007700B2 (en) * 2015-06-29 2018-06-26 Oracle International Corporation Query optimization for group-by extensions and distinct aggregate functions
US10067954B2 (en) * 2015-07-22 2018-09-04 Oracle International Corporation Use of dynamic dictionary encoding with an associated hash table to support many-to-many joins and aggregations
US10108667B2 (en) 2016-03-07 2018-10-23 International Business Machines Corporation Query plan optimization for large payload columns

Citations (66)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5598559A (en) 1994-07-01 1997-01-28 Hewlett-Packard Company Method and apparatus for optimizing queries having group-by operators
US5893086A (en) 1997-07-11 1999-04-06 International Business Machines Corporation Parallel file system and method with extensible hashing
US5930785A (en) * 1995-03-31 1999-07-27 International Business Machines Corporation Method for detecting and optimizing queries with encoding/decoding tables
US6026394A (en) 1993-01-20 2000-02-15 Hitachi, Ltd. System and method for implementing parallel operations in a database management system
US20020016820A1 (en) * 2000-05-30 2002-02-07 Jordan Du Val Distributing datacast signals embedded in broadcast transmissions over a computer network
US6505189B1 (en) 2000-06-15 2003-01-07 Ncr Corporation Aggregate join index for relational databases
US6609131B1 (en) 1999-09-27 2003-08-19 Oracle International Corporation Parallel partition-wise joins
US6757677B2 (en) 2001-09-28 2004-06-29 Ncr Corporation Providing a join plan using group-by operator
US6775681B1 (en) 2002-02-26 2004-08-10 Oracle International Corporation Evaluation of grouping sets by reduction to group-by clause, with or without a rollup operator, using temporary tables
US20040260684A1 (en) 2003-06-23 2004-12-23 Microsoft Corporation Integrating horizontal partitioning into physical database design
US20050018683A1 (en) 2003-07-21 2005-01-27 Zhao Yigiang Q. IP address storage technique for longest prefix match
US20050033741A1 (en) 2003-05-07 2005-02-10 Oracle International Corporation Efficient processing of relational joins of multidimensional data
US6941432B2 (en) 1999-12-20 2005-09-06 My Sql Ab Caching of objects in disk-based databases
US6954776B1 (en) 2001-05-07 2005-10-11 Oracle International Corporation Enabling intra-partition parallelism for partition-based operations
US7062481B2 (en) 2001-09-28 2006-06-13 Ncr Corp. Eliminating group-by operations in a join plan
US20070136317A1 (en) 2005-12-08 2007-06-14 International Business Machines Corporation Estimating the size of a join by generating and combining partial join estimates
US20070244850A1 (en) 2006-04-17 2007-10-18 Microsoft Corporation Perfect multidimensional spatial hashing
US20070245119A1 (en) 2006-04-17 2007-10-18 Microsoft Corporation Perfect hashing of variably-sized data
US7343363B1 (en) 2004-09-29 2008-03-11 Unisys Corporation Methods and apparatus for grouping elements of element pairs into element sets
US20080126706A1 (en) 2006-11-29 2008-05-29 William Thomas Newport Recoverable Cache Preload in Clustered Computer System
US20080133583A1 (en) 2006-10-26 2008-06-05 Nabi Sertac Artan Generating a hierarchical data structure associated with a plurality of known arbitrary-length bit strings used for detecting whether an arbitrary-length bit string input matches one of a plurality of known arbitrary-length bit string
US20080162402A1 (en) * 2006-12-28 2008-07-03 David Holmes Techniques for establishing and enforcing row level database security
US20090006399A1 (en) 2007-06-29 2009-01-01 International Business Machines Corporation Compression method for relational tables based on combined column and row coding
US20090024568A1 (en) 2007-07-20 2009-01-22 Al-Omari Awny K Data skew insensitive parallel join scheme
US7499960B2 (en) 2001-10-01 2009-03-03 Oracle International Corporation Adaptive memory allocation
US20090210445A1 (en) 2008-02-19 2009-08-20 International Business Machines Corporation Method and system for optimizing data access in a database using multi-class objects
US20090222659A1 (en) 2008-03-03 2009-09-03 Sony Corporation Communication device and communication method
US7653670B2 (en) 2005-11-28 2010-01-26 Nec Laboratories America, Inc. Storage-efficient and collision-free hash-based packet processing architecture and method
US20100088309A1 (en) * 2008-10-05 2010-04-08 Microsoft Corporation Efficient large-scale joining for querying of column based data encoded structures
US20100114868A1 (en) * 2008-10-21 2010-05-06 International Business Machines Corporation Query execution plan efficiency in a database management system
US20100131540A1 (en) 2008-11-25 2010-05-27 Yu Xu System, method, and computer-readable medium for optimizing processing of distinct and aggregation queries on skewed data in a database system
US20100223253A1 (en) * 2009-03-02 2010-09-02 International Business Machines Corporation Automatic query execution plan management and performance stabilization for workloads
US7827218B1 (en) 2006-11-18 2010-11-02 X-Engines, Inc. Deterministic lookup using hashed key in a multi-stride compressed trie structure
US7827182B1 (en) 2004-06-02 2010-11-02 Cisco Technology, Inc Searching for a path to identify where to move entries among hash tables with storage for multiple entries per bucket during insert operations
US20110060876A1 (en) 2009-09-08 2011-03-10 Brocade Communications Systems, Inc. Exact Match Lookup Scheme
US20110066593A1 (en) 2009-09-16 2011-03-17 Madhu Ahluwalia Method and system for capturing change of data
US20110078134A1 (en) * 2009-09-30 2011-03-31 International Business Machines Corporation System and method for aviding three-valued logic in predicates on dictionary-encoded data
US20110283082A1 (en) 2010-05-13 2011-11-17 International Business Machines Corporation Scalable, Concurrent Resizing Of Hash Tables
US20120011144A1 (en) 2010-07-12 2012-01-12 Frederik Transier Aggregation in parallel computation environments with shared memory
US20120036134A1 (en) 2009-04-08 2012-02-09 Intel Corporation Performing concurrent rehashing of a hash table for multithreaded applications
US8145642B2 (en) 2004-11-30 2012-03-27 Oracle International Corporation Method and apparatus to support bitmap filtering in a parallel system
US20120136889A1 (en) 2010-11-30 2012-05-31 Rajesh Jagannathan Hash Collision Resolution with Key Compression in a MAC Forwarding Data Structure
US20120136846A1 (en) 2010-11-30 2012-05-31 Haoyu Song Methods of hashing for networks and systems thereof
US8195644B2 (en) 2008-10-06 2012-06-05 Teradata Us, Inc. System, method, and computer-readable medium for optimization of multiple parallel join operations on skewed data
US20120143877A1 (en) 2010-12-03 2012-06-07 Futurewei Technologies, Inc. Method and Apparatus for High Performance, Updatable, and Deterministic Hash Table for Network Equipment
US20120158729A1 (en) 2010-05-18 2012-06-21 Lsi Corporation Concurrent linked-list traversal for real-time hash processing in multi-core, multi-thread network processors
US20120166400A1 (en) 2010-12-28 2012-06-28 Teradata Us, Inc. Techniques for processing operations on column partitions in a database
US20120173517A1 (en) * 2011-01-04 2012-07-05 International Business Machines Corporation Query-aware compression of join results
US8271564B2 (en) 2008-07-14 2012-09-18 Symbol Technologies, Inc. Lookup table arrangement and related management method for accommodating concurrent processors
US20120260349A1 (en) 2011-04-08 2012-10-11 Kabushiki Kaisha Toshiba Storage device, storage system, and authentication method
US8321385B2 (en) 2010-03-12 2012-11-27 Lsi Corporation Hash processing in a network communications processor architecture
US20120303633A1 (en) 2011-05-26 2012-11-29 International Business Machines Corporation Systems and methods for querying column oriented databases
US20120310917A1 (en) 2011-05-31 2012-12-06 International Business Machines Corporation Accelerated Join Process in Relational Database Management System
US8438574B1 (en) 2009-08-14 2013-05-07 Translattice, Inc. Generating monotone hash preferences
US20130218934A1 (en) 2012-02-17 2013-08-22 Hitachi, Ltd. Method for directory entries split and merge in distributed file system
US20140006379A1 (en) 2012-06-29 2014-01-02 International Business Machines Corporation Efficient partitioned joins in a database with column-major layout
US20140025648A1 (en) 2010-02-22 2014-01-23 Data Accelerator Ltd. Method of Optimizing Data Flow Between a Software Application and a Database Server
US20140074819A1 (en) * 2012-09-12 2014-03-13 Oracle International Corporation Optimal Data Representation and Auxiliary Structures For In-Memory Database Query Processing
US20140108489A1 (en) 2012-10-15 2014-04-17 Et International, Inc. Flowlet-based processing
US20140129568A1 (en) 2012-11-08 2014-05-08 Texas Instruments Incorporated Reduced complexity hashing
US8768927B2 (en) 2011-12-22 2014-07-01 Sap Ag Hybrid database table stored as both row and column store
US20140215019A1 (en) 2013-01-28 2014-07-31 Spencer Ahrens Static resource caching
US20140214794A1 (en) 2013-01-30 2014-07-31 International Business Machines Corporation Join operation partitioning
US20140214795A1 (en) * 2013-01-31 2014-07-31 International Business Machines Corporation Dynamically determining join order
US20140214855A1 (en) 2013-01-30 2014-07-31 International Business Machines Corporation Reducing collisions within a hash table
US20140372388A1 (en) 2013-06-14 2014-12-18 International Business Machines Corporation Hashing scheme using compact array tables

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8692695B2 (en) * 2000-10-03 2014-04-08 Realtime Data, Llc Methods for encoding and decoding data

Patent Citations (73)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6026394A (en) 1993-01-20 2000-02-15 Hitachi, Ltd. System and method for implementing parallel operations in a database management system
US5598559A (en) 1994-07-01 1997-01-28 Hewlett-Packard Company Method and apparatus for optimizing queries having group-by operators
US5930785A (en) * 1995-03-31 1999-07-27 International Business Machines Corporation Method for detecting and optimizing queries with encoding/decoding tables
US5893086A (en) 1997-07-11 1999-04-06 International Business Machines Corporation Parallel file system and method with extensible hashing
US6609131B1 (en) 1999-09-27 2003-08-19 Oracle International Corporation Parallel partition-wise joins
US6941432B2 (en) 1999-12-20 2005-09-06 My Sql Ab Caching of objects in disk-based databases
US20020016820A1 (en) * 2000-05-30 2002-02-07 Jordan Du Val Distributing datacast signals embedded in broadcast transmissions over a computer network
US6505189B1 (en) 2000-06-15 2003-01-07 Ncr Corporation Aggregate join index for relational databases
US6954776B1 (en) 2001-05-07 2005-10-11 Oracle International Corporation Enabling intra-partition parallelism for partition-based operations
US7062481B2 (en) 2001-09-28 2006-06-13 Ncr Corp. Eliminating group-by operations in a join plan
US6757677B2 (en) 2001-09-28 2004-06-29 Ncr Corporation Providing a join plan using group-by operator
US7499960B2 (en) 2001-10-01 2009-03-03 Oracle International Corporation Adaptive memory allocation
US6775681B1 (en) 2002-02-26 2004-08-10 Oracle International Corporation Evaluation of grouping sets by reduction to group-by clause, with or without a rollup operator, using temporary tables
US20050033741A1 (en) 2003-05-07 2005-02-10 Oracle International Corporation Efficient processing of relational joins of multidimensional data
US20040260684A1 (en) 2003-06-23 2004-12-23 Microsoft Corporation Integrating horizontal partitioning into physical database design
US20050018683A1 (en) 2003-07-21 2005-01-27 Zhao Yigiang Q. IP address storage technique for longest prefix match
US7827182B1 (en) 2004-06-02 2010-11-02 Cisco Technology, Inc Searching for a path to identify where to move entries among hash tables with storage for multiple entries per bucket during insert operations
US7343363B1 (en) 2004-09-29 2008-03-11 Unisys Corporation Methods and apparatus for grouping elements of element pairs into element sets
US8145642B2 (en) 2004-11-30 2012-03-27 Oracle International Corporation Method and apparatus to support bitmap filtering in a parallel system
US7653670B2 (en) 2005-11-28 2010-01-26 Nec Laboratories America, Inc. Storage-efficient and collision-free hash-based packet processing architecture and method
US20070136317A1 (en) 2005-12-08 2007-06-14 International Business Machines Corporation Estimating the size of a join by generating and combining partial join estimates
US20070245119A1 (en) 2006-04-17 2007-10-18 Microsoft Corporation Perfect hashing of variably-sized data
US20070244850A1 (en) 2006-04-17 2007-10-18 Microsoft Corporation Perfect multidimensional spatial hashing
US20080133583A1 (en) 2006-10-26 2008-06-05 Nabi Sertac Artan Generating a hierarchical data structure associated with a plurality of known arbitrary-length bit strings used for detecting whether an arbitrary-length bit string input matches one of a plurality of known arbitrary-length bit string
US7827218B1 (en) 2006-11-18 2010-11-02 X-Engines, Inc. Deterministic lookup using hashed key in a multi-stride compressed trie structure
US20080126706A1 (en) 2006-11-29 2008-05-29 William Thomas Newport Recoverable Cache Preload in Clustered Computer System
US20080162402A1 (en) * 2006-12-28 2008-07-03 David Holmes Techniques for establishing and enforcing row level database security
US20090006399A1 (en) 2007-06-29 2009-01-01 International Business Machines Corporation Compression method for relational tables based on combined column and row coding
US20090024568A1 (en) 2007-07-20 2009-01-22 Al-Omari Awny K Data skew insensitive parallel join scheme
US20120117055A1 (en) 2007-07-20 2012-05-10 Al-Omari Awny K Data Skew Insensitive Parallel Join Scheme
US20090210445A1 (en) 2008-02-19 2009-08-20 International Business Machines Corporation Method and system for optimizing data access in a database using multi-class objects
US20090222659A1 (en) 2008-03-03 2009-09-03 Sony Corporation Communication device and communication method
US8271564B2 (en) 2008-07-14 2012-09-18 Symbol Technologies, Inc. Lookup table arrangement and related management method for accommodating concurrent processors
US20100088309A1 (en) * 2008-10-05 2010-04-08 Microsoft Corporation Efficient large-scale joining for querying of column based data encoded structures
US8195644B2 (en) 2008-10-06 2012-06-05 Teradata Us, Inc. System, method, and computer-readable medium for optimization of multiple parallel join operations on skewed data
US20100114868A1 (en) * 2008-10-21 2010-05-06 International Business Machines Corporation Query execution plan efficiency in a database management system
US20100131540A1 (en) 2008-11-25 2010-05-27 Yu Xu System, method, and computer-readable medium for optimizing processing of distinct and aggregation queries on skewed data in a database system
US20100223253A1 (en) * 2009-03-02 2010-09-02 International Business Machines Corporation Automatic query execution plan management and performance stabilization for workloads
US20120036134A1 (en) 2009-04-08 2012-02-09 Intel Corporation Performing concurrent rehashing of a hash table for multithreaded applications
US8438574B1 (en) 2009-08-14 2013-05-07 Translattice, Inc. Generating monotone hash preferences
US20110060876A1 (en) 2009-09-08 2011-03-10 Brocade Communications Systems, Inc. Exact Match Lookup Scheme
US20110066593A1 (en) 2009-09-16 2011-03-17 Madhu Ahluwalia Method and system for capturing change of data
US20110078134A1 (en) * 2009-09-30 2011-03-31 International Business Machines Corporation System and method for aviding three-valued logic in predicates on dictionary-encoded data
US20140025648A1 (en) 2010-02-22 2014-01-23 Data Accelerator Ltd. Method of Optimizing Data Flow Between a Software Application and a Database Server
US8321385B2 (en) 2010-03-12 2012-11-27 Lsi Corporation Hash processing in a network communications processor architecture
US20110283082A1 (en) 2010-05-13 2011-11-17 International Business Machines Corporation Scalable, Concurrent Resizing Of Hash Tables
US20120158729A1 (en) 2010-05-18 2012-06-21 Lsi Corporation Concurrent linked-list traversal for real-time hash processing in multi-core, multi-thread network processors
US20120011144A1 (en) 2010-07-12 2012-01-12 Frederik Transier Aggregation in parallel computation environments with shared memory
US20130138628A1 (en) 2010-07-12 2013-05-30 Christian Bensberg Hash-join in parallel computation environments
US20120011108A1 (en) 2010-07-12 2012-01-12 Christian Bensberg Hash-join in parallel computation environments
US8370316B2 (en) 2010-07-12 2013-02-05 Sap Ag Hash-join in parallel computation environments
US20120011133A1 (en) 2010-07-12 2012-01-12 Franz Faerber Interdistinct Operator
US20120136846A1 (en) 2010-11-30 2012-05-31 Haoyu Song Methods of hashing for networks and systems thereof
US20120136889A1 (en) 2010-11-30 2012-05-31 Rajesh Jagannathan Hash Collision Resolution with Key Compression in a MAC Forwarding Data Structure
US20120143877A1 (en) 2010-12-03 2012-06-07 Futurewei Technologies, Inc. Method and Apparatus for High Performance, Updatable, and Deterministic Hash Table for Network Equipment
US20120166400A1 (en) 2010-12-28 2012-06-28 Teradata Us, Inc. Techniques for processing operations on column partitions in a database
US20120173517A1 (en) * 2011-01-04 2012-07-05 International Business Machines Corporation Query-aware compression of join results
US20120260349A1 (en) 2011-04-08 2012-10-11 Kabushiki Kaisha Toshiba Storage device, storage system, and authentication method
US20120303633A1 (en) 2011-05-26 2012-11-29 International Business Machines Corporation Systems and methods for querying column oriented databases
US20120310917A1 (en) 2011-05-31 2012-12-06 International Business Machines Corporation Accelerated Join Process in Relational Database Management System
US8768927B2 (en) 2011-12-22 2014-07-01 Sap Ag Hybrid database table stored as both row and column store
US20130218934A1 (en) 2012-02-17 2013-08-22 Hitachi, Ltd. Method for directory entries split and merge in distributed file system
US20140006379A1 (en) 2012-06-29 2014-01-02 International Business Machines Corporation Efficient partitioned joins in a database with column-major layout
US20140074819A1 (en) * 2012-09-12 2014-03-13 Oracle International Corporation Optimal Data Representation and Auxiliary Structures For In-Memory Database Query Processing
US20140108489A1 (en) 2012-10-15 2014-04-17 Et International, Inc. Flowlet-based processing
US20140129568A1 (en) 2012-11-08 2014-05-08 Texas Instruments Incorporated Reduced complexity hashing
US20140215019A1 (en) 2013-01-28 2014-07-31 Spencer Ahrens Static resource caching
US20140372407A1 (en) 2013-01-30 2014-12-18 International Business Machines Corporation Join operation partitioning
US20140214794A1 (en) 2013-01-30 2014-07-31 International Business Machines Corporation Join operation partitioning
US20140214855A1 (en) 2013-01-30 2014-07-31 International Business Machines Corporation Reducing collisions within a hash table
US20140372392A1 (en) 2013-01-30 2014-12-18 International Business Machines Corporation Reducing collisions within a hash table
US20140214795A1 (en) * 2013-01-31 2014-07-31 International Business Machines Corporation Dynamically determining join order
US20140372388A1 (en) 2013-06-14 2014-12-18 International Business Machines Corporation Hashing scheme using compact array tables

Non-Patent Citations (24)

* Cited by examiner, † Cited by third party
Title
Anonymous, "Efficient Grouping Over Joins of Compressed Tables", IP.com, Apr. 6, 2010, pp. 1-6.
Anonymous, "High Performance Technique Using Join Collocation in a Massively Parallel Processing Relational Database Implementation", IP.com, Jun. 14, 2012, pp. 1-5.
Anonymous, "System and Method for Usage Aware Row Storage in Database Systems", IP.Com, Jul. 23, 2010, pp. 1-4.
Anonymous, CashMap: Processor Cache-Aware Implementation of Hash Tables, IP.com, Jul. 5, 2013, pp. 1-7.
Attaiuri et al., U.S. Appl. No. 13/753,740, "Join Operation Partitioning", Filed Jan. 30, 2013; 51 pages.
Attaluri et al., U.S. Appl. No. 13/753,740 "Reducing Collisions Within a Hash Table" filed Jan. 30, 2013, 50 pages.
Attaluri et al., U.S. Appl. No. 13/755,784, "Dynamically Determining Join Order", Filed Jan. 31, 2013; 42 pages.
Chang, Shih-Fu, "Recent Advances of Compact Hasing for Large-Scale Visual Search", Columbia University, Oct. 2012, pp. 1-44.
Cleary, John G., "Compact Hash Tables Using Bidirectional Linear Probing", IEEE Transactions on Computers, vol. C-33, No. 9, Sep. 1994, pp. 824-834.
Cutt et al., "Improviing Join Performance for Skewed Databases", IEEE, 2008, 5 pages.
Hu et al., "Rapid multi-demention hierarchical algorithm in data warehouse system", Computer Integrated Manufacturing Systems, Jan. 2007, pp. 196-201, vol. 13, No. 1 China [English-language translation: Abstract Only].
Hua, Nan, et al., "Rank-Indexed Hasing: A Compact Construction of Bloom Filters and Variants", IEEE, 2008, pp. 73-82.
Korn et al., "The VCDIFF Generic Differencing and Cojpression Data Format (RFC3284)", Network Working Group, IP.com, Jul. 1, 2002, pp. 1-31.
Li et al., "Adaptively Reordering Joins During Query Execution", IEEEE, 2007, pp. 26-35.
Marek et al., "TID Hash Joins," CIKM, 1994, pp. 42-29.
Mell et al., "The NIST Definition of Could Computing", National Institute of Standands and Technolgy Special Publication 800-145, Department of Commerce, Sep. 2011, pp. 1-7.
Raman et al., "DB2 with BLU Acceleration: So Much More than Just a Column Store", Proceedings of the VLDB Endowment, ACM, Aug. 2013, pp. 1-12, vol. 6, No. 11.
Spyros et al., "Design and Evaluation of Main Memory Hash Join Algorithms for Multi-core CPU's", SIGMOD Int'l Conference on Management of Data ACM, Jun. 12, 2011, pp. 1-12.
U.S. Appl. No. 13/537,745, "Efficient Partitioned Joins in a Database With Column-Major Layout", Filed Jun. 29, 2012, 41 pages.
U.S. Appl. No. 13/918,302; Non-Final Office Action; Date Filed: Jun. 14, 2013; Date Mailed: Jun. 17, 2015; 33 pages.
U.S. Appl. No. 14/509,336, "Embracing and Exploiting Data Skew During a Join or Groupby", Filed Oct. 8, 2014, 38 pages.
Wang et al., "Investigating Memory Optimization of Hash-Index . . . Sequencing on Multi-Core Architecture", IPDPSW, IEEE 26th Inter. Conference on May 21-25, 2012, pp. 665-674.
Xu, Yang, "A Multi-Dimensional Progressive Perfect Hashing for High Speed String Matching", 2011 Seventh ACM/IEEE Symposium on Architectures for Networking and Communications Systems, pp. 167-177.
Yan, Weipeng P. et al., "Performing Group-By before Join [query processing]," Proceedings 10th International Conference on Data Engineering, 1994, pp. 1-30, IEEE, 1994.

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9665624B2 (en) 2013-01-30 2017-05-30 International Business Machines Corporation Join operation partitioning
US9935650B2 (en) 2014-04-07 2018-04-03 International Business Machines Corporation Compression of floating-point data by identifying a previous loss of precision
US9672248B2 (en) 2014-10-08 2017-06-06 International Business Machines Corporation Embracing and exploiting data skew during a join or groupby
US10489403B2 (en) 2014-10-08 2019-11-26 International Business Machines Corporation Embracing and exploiting data skew during a join or groupby
US9959299B2 (en) 2014-12-02 2018-05-01 International Business Machines Corporation Compression-aware partial sort of streaming columnar data
US20160246810A1 (en) * 2015-02-25 2016-08-25 International Business Machines Corporation Query predicate evaluation and computation for hierarchically compressed data

Also Published As

Publication number Publication date
US20160253390A1 (en) 2016-09-01
US20140372411A1 (en) 2014-12-18
US9471710B2 (en) 2016-10-18
US20140372470A1 (en) 2014-12-18

Similar Documents

Publication Publication Date Title
Valduriez Join indices
Leis et al. The adaptive radix tree: ARTful indexing for main-memory databases
Stonebraker et al. C-store: a column-oriented DBMS
Raman et al. DB2 with BLU acceleration: So much more than just a column store
CA2232938C (en) Method and apparatus for performing a join query in a database system
US6775681B1 (en) Evaluation of grouping sets by reduction to group-by clause, with or without a rollup operator, using temporary tables
US6484181B2 (en) Method and system for handling foreign key update in an object-oriented database environment
US10318493B2 (en) Custom policy driven data placement and information lifecycle management
US10346383B2 (en) Hybrid database table stored as both row and column store
US20120016901A1 (en) Data Storage and Processing Service
US7895151B2 (en) Fast bulk loading and incremental loading of data into a database
US20120054225A1 (en) Query and Exadata Support for Hybrid Columnar Compressed Data
AU773177B2 (en) System for maintaining precomputed views
US8156134B2 (en) Using different groups of query graph transform modules to generate execution plans for queries for different database types
US8782100B2 (en) Hybrid database table stored as both row and column store
JP6617117B2 (en) Scalable analysis platform for semi-structured data
US9798772B2 (en) Using persistent data samples and query-time statistics for query optimization
US9507822B2 (en) Methods and systems for optimizing queries in a database system
US7158996B2 (en) Method, system, and program for managing database operations with respect to a database table
US8935232B2 (en) Query execution systems and methods
US7171427B2 (en) Methods of navigating a cube that is implemented as a relational object
US20110213775A1 (en) Database Table Look-up
US20100005077A1 (en) Methods and systems for generating query plans that are compatible for execution in hardware
US8396861B2 (en) Determining a density of a key value referenced in a database query over a range of rows
Chattopadhyay et al. Tenzing a sql implementation on the mapreduce framework

Legal Events

Date Code Title Description
AS Assignment

Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ATTALURI, GOPI K.;BARBER, RONALD J.;KULANDAISAMY, VINCENT;AND OTHERS;SIGNING DATES FROM 20130830 TO 20140320;REEL/FRAME:032527/0627

STCF Information on status: patent grant

Free format text: PATENTED CASE