WO2004023328A1 - Indexed data storage system, method and data structure - Google Patents

Indexed data storage system, method and data structure Download PDF

Info

Publication number
WO2004023328A1
WO2004023328A1 PCT/IN2003/000297 IN0300297W WO2004023328A1 WO 2004023328 A1 WO2004023328 A1 WO 2004023328A1 IN 0300297 W IN0300297 W IN 0300297W WO 2004023328 A1 WO2004023328 A1 WO 2004023328A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
record
value
unique
attribute
Prior art date
Application number
PCT/IN2003/000297
Other languages
French (fr)
Inventor
Tridib Roy Chowdhury
Original Assignee
Stex Technologies Private Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to IN524/CAL/02 priority Critical
Priority to IN524KO2002 priority
Application filed by Stex Technologies Private Limited filed Critical Stex Technologies Private Limited
Priority claimed from AU2003278589A external-priority patent/AU2003278589A1/en
Publication of WO2004023328A1 publication Critical patent/WO2004023328A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2228Indexing structures
    • G06F16/2246Trees, e.g. B+trees
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28Databases characterised by their database models, e.g. relational or object models
    • G06F16/284Relational databases

Abstract

An indexed data structure and associated method and system are disclosed. Data records to be stored are reduced to a single dimension and then stored and indexed using a plurality of inter-related data stores (100). The data stores (100) store both data and metadata within generic records. In a preferred embodiment, the data stores (100) comprise 4 B+ Trees. A method for storing data, inserting data into an existing data structure, deleting data records and searching the data structure is disclosed, as is a computer system for use with the data structure.

Description

INDEXED DATA STORAGE SYSTEM, METHOD AND DATA STRUCTURE

FIELD OF INVENTION The present invention relates to a system, method and data structure for compact storage and efficient retrieval of data from domains where the dimension of the data-set is large and variable.

BACKGROUND TO THE INVENTION

RELATIONAL DATA STORAGE

Data in computers is stored mainly in databases. Of all the different database models that are used, the relational model (RM) described in A relational model of data for large shared data banks, Communications of ACM, Vol 13, No 6, pp 377-387 by E.Codd, is the most popular and prevalent. The RM is designed to store information as a table, each row storing a record, and each column of a row storing a value for an attribute of that record.

For example, a possible set of attributes which would which would help define a database storing the records of all the employees in an organization could be Employee Code (8 bytes), Name (40), Sex (1), Date of Birth (10), Department (20), Designation (20), Qualifications (10), Skill-Set (50), Languages Spoken (15). The attributes in Italics may contain more than one value for an employee. Attributes having multiple values for a record are of type sets.

One of the primary drawbacks of the RM is its inherent inability to store attributes of type set. For the purpose of non-redundancy and data integrity, the relational model expects data to be stored in some normal form as described in A relational model of data for large shared data banks, Communications of ACM, Vol 13, No 6, pp 377-387 by E.Codd. The most basic of these normal forms defines that each and every value for every attribute in a table must at least be atomic, that is they cannot be multi-valued. Though a normalization process allows us to overcome some of these problems, it is always at the expense of efficiency.

Index files, typically B+Tree or hashed index, are created on certain attribute(s), and are used to improve access performance from this unsorted RM table. The attribute(s) are chosen based on a pre-deter ined set of queries that are expected on this table. Arbitrary queries not made on these attributes cannot take advantage of these indices, and indeed sometimes suffer in performance due to the presence of these indices. B+ TREE INDEX

One of the most popular index structures is the B+ Tree described in The Ubiquitous B-Tree, Computing Surveys, Vol. 11 , No. 2, pp 121-137, June 1979 by Douglas Come. The B+ Tree is a balanced tree.

In a B+ Tree any record stored within its leaf nodes can be accessed in [logmN] disk accesses. The number m is the number of attributes that can be stored in any node, and N is the number of records stored in the B+ Tree file, which is also equal to the number of records in the table. The number m is also called the fan-out of a B+ Tree.

The size of each node block is typically between 8 KB and 32 KB (KB = 1024 bytes), though this size can vary. The size of the block is typically equal to the hard disk block size. But whatever the block size is, it follows that the value of m decreases with an increase in the size of the key attribute. Therefore, the number of disk accesses increases with an increase in the key attribute size.

The index files might store a composite key comprising of a set of attributes, instead of a singleton key of one single attribute. The choice of the key composition is dependent on the analysis of the application environment in which the data will be accessed from. For example, in a case where the frequency of queries on "Sex" and "Department" is more significant than any other combination, then an index file with a composite key "Designation; Department" would need to be created. The size of the composite key would be 40 bytes. In case of indexing on "Skill Sets" the key would be 50 bytes. Depending on the application environment, the possibility of having large key sizes is a very real possibility, reducing fan-out of the B+ tree, and hence its performance.

A popular database design to reduce the potential key size of an index is by "coding". Coding maps the real-world index values to numerical values, and these numerical values are used in the index. The code map is stored as a separate database table. It is easy to see that if the possible unique values for this coded attributes are large, then access to this coded table will add to the overhead. Besides, range searches on "coded" values are not possible unless the sort-order of the values in the original domain is maintained in the coded transform, which is seldom the case.

When creating a composite key it is also important to note that the order of the attributes in the key structure are also very relevant. For example, in a case where an index file has a composite key of attribute "Sex", "D.O.B" and "Department", and the composite key is {Sex; D.O.B.; Department}, then a query on "D.O.B." and "Department" cannot use this index file, whereas a query on "Sex" and "Department" can use it. There is an explicit left-right ordering of the attributes, which is only effective for some pre-determined queries, but not for any arbitrary query.

When a query is processed, the query optimizer has to choose which of the index files to use for the answering of queries. As the number of index files grows, so does the need to maintain metric information, as well as the need for extra computation to decide which one(s) to use. This slows down the query optimization process. Using composite indices on multiple attributes can potentially reduce the number of index files. However, if the query does not maintain the left-right attribute hierarchy, then that specific index file becomes useless. Also the increased size of the composite key reduces efficiency of the index. If only single attribute indices are used, then the number of index files grows with the increasing dimensions, increasing the query optimization times as well as the storage required. Efficiency of cost effective index selection tools for commercial databases systems is very important in the overall performance of the database engine as described in A Efficient Cost-Driven Index Selection Tool for Microsoft SQL Server, Proceedings of 23rd VLDB Conference, Athens Greece 1997, pp 146-155 by Surajit Chaudhari et al.

These index schemes cannot be used in the processing of queries with the NOT predicate. Any queries having a NOT condition would require a TABLE- SCAN with on-the-fly selection operation.

MULTI-DIMENSIONAL INDEX Multidimensional index structures have been developed to address some of these problems. These can be categorized into three basic sets:

1. Spatial databases which deal specifically with spatial data structures use R-Trees [Antonin Guttman, 1984, R-Trees: A Dynamic Index Structure For Spatial Searching, Proceedings of ACM SIGMOD], R+ Trees [Timos Sellis, Nick Roussopoulos, Christos Faloutos, 1987, The R+Tree: A Dynamic Index for Multi-Dimensional Objects, Procedings of 13th International Conference on Very Large Databases, pp 507-518], k-d-bTrees[J.T.Robinson, 1981, The k-d- b-Tree A Search Structure for Large Multidimensional Dynamic Indexes, Proceedings ACM SIGMOD International Conference on Management of Data, pp 10-18, Ann Arbor, April-May 1981], hB Trees[David B. Lomet, Betty Salzburg, 1990, The hB-Tree: A Multiattribute Indexing Method with Good Guranteed Performance, ACM Transactions on Database Systems, Vol. 15, No.4, December 1990, pp 625-658], TV-Trees[K-I.Lin, H.V. Jagdish, C. Faloutsos, 1994, The TV Tree-An Index Structure for High-Dimensional Data,

VLDB Journal, Vol. 3 pp517-542, 1994] and X-Trees[S. Berchtold, D.A. Keim, H. Kriegel, 1996, The X-tree: An Index Structure for High-dimensional Data, Proceedings of the 22nd VLDB Conference, Mumbai, India, pp 28-39]. These index structures attempt to extend the B+Tree structure to multiple dimension, and differ essentially how the Minimum Bounding Rectangle (MBR) is constructed and how the internal nodes spilt when required on insertions.

2. Index structures that are based on space-filling curve like z- ordering[J.A. Orenstein, 1986, Spatial Query Processing in an Object-Oriented Database System, Proc. ACM SIGMOD International Conf. On Management of Data, 1986, pp 326-336], Hubert curve[D. Hubert, 1891 , Uber die steitige

Abbildung emer Linie auf eun Flachenstuck, Math Ann, 38, 1891] techniques partition the data space independent of the data set available. These techniques are attempts to linearize a multi dimensional space to a single dimension, with attention paid to ensuring to the maximum the proximity of the points in the n dimensional space, translates to proximity of the points in the 1 dimension. A recent development in this space has been the Pyramid Trees[S .Berchtold, Christian Boeh , Hans-Peter Kriegel, 1998, High Dimensional Index Structure, U.S. Patent Number 6,154,746, Date of patent 28 Nov, 2000.]. The points in the data-set are then mapped onto a single number, which is typically formed by interleaving bits of the co-ordinates in the various dimensions and then stored in a B+Tree like structure.

3. Grid FilesfJ. Nievergelt, H. Hintenberger, K. Sevcik, 1984, The Grid File: An adaptable symmetric multikey file structure, ACM Transactions on Database Systems, Vol 9, No 1., pp 38-71 , 1984]. Unlike the previous structures, the grid file attempts to partition space dependent on the data-set. The grid files rely upon a grid directory, to identify which page has the target point.

Lossiness in representation results in inefficiency in search. While going down the respective trees from the root to a leaf, the search algorithm generates more than one possible route. Each of these routes needs to be checked to the logical end to compile the complete answer, even though some or most of the routes are dead-ends. A measure of lossiness in representation is defined by the overlap metric. Most multidimensional indexing methods based on MBR, suffer from two problems. One is that increasing dimensions reduces the fan-out of their trees, reducing performance. The second problem is that of increased overlap with increasing dimensions. Most of these techniques work well in dimensions less than or equal to 3, but suffer in higher dimensions, where accesses reduces to sequential scan to the tree.

The TV-Tree described in The TV Tree-An Index Structure for High- Dimensional Data, VLDB Journal, Vol. 3 pp517-542, 1994 by K-I.Lin et al. is an attempt to reduce this fan-out loss, but is forced to impose an artificial ordering on the attributes. X-trees described in The X-tree: An Index Structure for High-dimensional

Data, Proceedings of the 22nd VLDB Conference, Mumbai, India, pp 28-39 by S. Berchtold, D.A. et al reduce the overlap by creating supernodes, i.e. by expanding the node block size in integral of basic node block size. But in higher dimensions, these supernodes may grow and result in sequential scans. For methods using space filling curves, the hypersurface bounding the query region, which determines the query performance, grows exponentially with dimensionality.

The grid file's directory also grows exponentially with dimensionality.

The Pyramid Tree described in High Dimensional Index Structure, U.S. Patent Number 6,154,746, uses a different method of partitioning the space, which ensure that there is no overlap or lossiness, and its performance is more stable under higher dimensions. But the structure fails to handle attributes of mixed types, and does not handle variable dimensions. It is tempting to map multi-attribute indexing to a multidimensional index structure. After all in abstract, a tuple in a relation can be thought of being a point in a multidimensional space. But there are some important differences. In a multidimensional data set of spatial data, the proximity between the two point A(23, 4, 5) and B(24, 4, 5), and that between A(23, 4,5) and B(30, 10, 10) have a semantic basis. Point A and B are closer than points A and C. That is because the ordering on each of the dimension has a semantic basis. However, ordering on the "Qualification" attribute (or dimension) of the values "BA", "B-ARCH", "B-TECH", "D-Phil", "MA", "MBA", "MS", "M-TECH", "PhD" often has no basis except their alphabetical ordering. It is possible to map these values to numerical values, wherein the order values will have some semantic in the real world. A possible mapping would be 0, 0, 0, 2, 1 , 1 , 1 , 1 , 2, where we use 0 for undergraduate, 1 for graduate, and 2 for post-graduate degrees. But, this leads to a query hit on employees with an "MA" when we were looking for employees with "MS" degrees. Any unique mapping of values to numbers may not have a semantic basis. This will affect range searches, disallowing them on almost all non-numerical attributes .For attributes having alphanumeric values, using of wild cards searches will not be possible by any kind of numerical mapping, except possibly something like XYZ* wild cards. Most of the methods mentioned above work essentially on spatial data, or data-sets where the attributes are purely numeric. In case, of attributes having different data type, like character strings, most of the methods suffer from performance degradation because of low fan-out, while index structures based on space partitioning schemes might not work at all. While in the case of spatial domain, queries are made across all attributes, in other domains like traditional RM, Object Oriented Database Management Systems (OODBMS), datawarehousing etc. a typical query are made only on certain attributes. In such case, framing a query on these spatial index structures results in a query region, which extends the complete span of the hyperspace on the undefined dimensions. This large query region forces a larger overlap.

All these structures take an atomic data as a record. RD-Trees described in The RD-Tree: An Index Structure for Sets, Technical Report @1274, University of Wisconsin at Madison, October 1994 by Joseph M. Hellerstrin et al., addresses the possibility of creating index on attributes of data-type set. But, the query set is restricted to superset, overlap and join queries. With a variation of the RD-Tree called Inverted RD-Tree described in the same publication, we can also address the issue of subset efficiently. But the RD Trees suffer from representational problem. If the nodes store sets a collection of atomic objects, the objects grow rapidly in size as we move from the leaf to the node. This reduces fan-out and increases tree height and reduces efficiency. There is also the issue of an internal node splitting even if a child node has not split. Suggestions to represent sets as either a set of pointers to the actual set, or with Bloom filters or signatures or as a set of finite range sets suffer from lossiness in representation. Besides these drawbacks, the RD tree is designed purely for attributes of data type sets, but will not perform at all when dealing with attributes storing atomic data.

Most of these structures at the core is a B+ Tree, with complex data structures being stored as keys and index.

The Multidimensional B Trees (MB Trees) described in Efficient Search of Multidimensional B-Trees, Proceedings of the 21st VLDB Conference, Zurich, Switzerland, 1995, pp 710- 719, by Harry Leslie et al., look purely at the problem of multi-attribute indexing. The Multidimensional Access Methods (MDAM) are used to query on this MB tree. In a MB Tree the full record is stored in the nodes and leaf of a B - Tree, which severely restricts the value of the fan-outs of the tree. While the MDAM does have a scheme to tackle null values, NOT predicates, index-ordering, it is not very efficient. Though attribute index ordering is overcome in MDAM, but the order of the attribute index does pay a big role in determining the efficiency of execution for any arbitrary query. In fact, certain queries, specially based on queries with undefined intermediate column values, it may result in scanning the whole table.

BITMAP INDEX

A bitmap index is a bit string of the size of a table. If a table being has 1 million tuples, then any bitmap of this table has 1 million bits. The value of b in bitmap (P), has a value of 1 if the tuple satisfies the condition P, and 0 otherwise. Suppose we want to create a bitmap on the attribute "Sex", then we would require 2 bit maps, one for P = "Sex=Male" and one for P = "Sex=Female". The advantage of the bitmap index is its efficiency to process queries having complex logical selection operations involving multiple attributes. For an attribute A, having a cardinality of |A|, you will require \A\ bitmaps. Therefore, the potential number of indices can be very high.

Bit-sliced indices suggested in Improved query performance with variant indices, ACM SIGMOD '97, pp 38-49 by P. O'Neil et al. allows one to reduce the number of bitmaps to log2l 4| for attributes which can be represented numerically.

Encoded bitmap index Encoded Bitmap Indexing for Data Warehouses, ICDE 1998, pp 220-230 by Ming-Chuan Wu et al. store attributes of any type(numeric, decimal, character...) using the same number of log2l 4| bitmaps. Encoded bitmaps however, has an added overhead to store the map table which converts the attribute value to numerical values taking log2)>4| bits.

Compressed bitmaps reduce storage overheads, but increase computational overheads in compression and decompression. Performance of compression algorithms is dependent on the data distribution, attribute selectivity, and also the query to be processed. While simple bitmaps do have the ability to store attributes of type sets, encoded bitmaps (including bit-sliced ones) do not work.

None of the existing indexing structures, bitmap indexes or multidimensional indexes can address all of the following requirements on their own: 1. Provide performance stability under high dimensions.

2. Ability to store non-atomic set data.

3. Ability to index attributes of varying data types.

4. Provide good and stable performance for any arbitrary query.

5. Update operations are proportional with the dimensions being inserted or modified.

While some of these issues are not critical in the traditional domain of Online Transaction Processing (OLTP), these issues are of significant importance when we are dealing with the domain of document management, content management, digital asset management (DAM), DNA databases, XML databases, On-Line Application Processing (OLAP) application, datawarehousing, and object oriented databases.

DNA databases may require 1024 dimensions, or DAM databases, which require about 256 attributes for every image! Query on these domains cannot be predetermined, and queries on any of these attributes are equally likely. SUMMARY OF INVENTION

According to a first aspect of the present invention, there is provided a data structure for storing data records, each data record including a plurality of data fields capable of storing one or more data items, the data structure comprising: a plurality of data stores arranged, in combination, to store and index said data records when reduced to a single dimension.

Motivated by the need to address the drawbacks of the current data storage and multidimensional indexing schemes, the method, system and data structure of the present invention is proposed.

The present invention seeks to provide support for arbitrary data sets (at least arbitrary in dimension and type) and also for handling arbitrary (as in not predetermined) query sets. The present invention also seeks to provide operational efficiency within acceptable variations over the complete set of arbitrary queries. Variations in the access time for example, do not depend on the query being executed, or the dimension of the database, but only on the size of the database.

In a preferred embodiment, each data store comprises a B+ tree structure. The data stores preferably include a first data store including a generic record of a first type for each unique data item of each of said data fields. The first data store may include a generic record of said first type for each of said data fields. A further data store including a generic record of said first type may be used to store the data fields, said further data store being associated with said first data store. Preferably, for a data field each generic record of said first type includes: an attribute identifier, the attribute identifier for each data field being unique; a value ID defining a data type of the data field's data items; and, a value defining the data field. Preferably, for a data item each generic record of said first type includes: an attribute identifier comprising the attribute identifier of its respective data field; a value ID comprising a unique identifier with respect to its respective data field; and, a value comprising the data item.

Each generic record of said first type may include a frequency count, the frequency count for a data field comprising the number of data items stored for said data field. The frequency count for a data item may comprise the number of instances of said data item occurring in said stored data items for the respective data field.

If the frequency count for a data item is 1 , the frequency count stored in the generic record of said first type may be replaced with an encoded unique record ID corresponding to said data record. The first data store is preferably indexed on a composite key of the attribute identifier and the value.

Preferably, the second data store includes a generic record of a second type for each data item, each generic record of said second type including said attribute identifier, said value ID and said value, the second data store being indexed on a composite key of the attribute identifier and value ID.

Preferably, the third data store includes a generic record of a third type for each data item, each generic record of said third type including a unique Record

ID identifying the data item's respective data record, said attribute identifier and said value ID, the third data store being indexed on a composite key of the Record ID and the attribute identifier.

Preferably, the fourth data store includes a generic record of said third type for each data item, the fourth data store being indexed on a composite key of attribute identifier and value ID.

Records in the leaves of one or more of the B+ Trees may be compressed. Data items of multi-dimensional data fields are preferably mapped in a numerical and order-preserving manner, said mapped values being stored in the data structure.

According to a second aspect of the present invention, there is provided a method of storing data records including a plurality of data fields storing one or more data items comprising: reducing the data records to a plurality of records of a single dimension; storing and indexing said plurality of records across a plurality of data stores.

Preferably, the plurality of data stores comprises 4 B+ trees. The method may further comprise the steps of:

Creating a unique record identifier for every data record; Generating a unique attribute identifier for each data field; Assigning said unique attribute identifier to each unique data item of said data field;

Assigning a value ID to each data field, the value ID defining a data type of the respective data item(s);

Generating a value ID for each unique data item, the value ID comprising a unique identifier for said data item with respect to said respective data field;

Maintaining a frequency count for each data field or data item, the frequency count for a data field comprising the number of data items stored for said data field, the frequency count for a data item comprising the number of instances of said data item occurring in said stored data items for the respective data field; and,

Storing at least selected parts of the data fields and/or data items and associated unique record identifier, attribute identifier, value ID and frequency count in each of the four B+ Trees.

For a new data record to be inserted, the method may comprise: Creating a unique record identifier for the data record;

For each data item of each data field of the data record: Identifying the unique attribute identifier for said data field; Determining if the data item is stored with respect to said unique attribute identifier; If said data item is stored, incrementing the frequency count for the data item;

If said data item is not stored, generating a value ID for the data item, the value ID comprising a unique identifier for said data item with respect to said respective data field, assigning a frequency count to said data item and incrementing the frequency count of the data field; and,

Storing at least selected parts of the data item and associated unique identifier, attribute identifier, value ID and frequency count in each of the four B+ Trees.

For a data record is to be deleted, the method may comprise: For each data item of each data field of the data record: Identifying the unique attribute identifier for said data field; If the frequency count of the data item stored with respect to said unique attribute identifier is greater than one, decrementing the frequency count;

If the frequency count is one, deleting at least selected parts of the data item and associated unique identifier, attribute identifier, value ID and frequency count from the four B+ Trees.

According to a third aspect of the present invention, there is provided a data storage system for storing data record, each data record including a plurality of data fields storing one or more data item, the data storage system comprising: an interface and a database; the database comprising four data stores encoded in a computer readable memory, the data stores being arranged, in combination, to store and index said data records when reduced to a plurality of records of a single dimension; the interface being programmed to accept at least selected ones of: input of a new data record to be stored, input of changes to a stored data record, input of a deletion request for a stored data record, input of a query on a data record; wherein the interface is arranged to communicate with the database and operate in dependence on said input.

The present invention seeks to support data type sets, such as those for use in most non-OLTP environments. Since there is no difference between data (attributes without any index) and index attributes, the physical separation of index and tables are eliminated. There is only one structure, which stores all data. All the attributes stored in the data structure of the present invention are treated with the same importance. However, unlike in the RM, the number of files that needs to be created remains 4, irrespective of the number of attributes.

The structure is designed such that while no file mirrors the other, any file of this group can be generated from the other 3 or 2. This leads to a high degree of fault-tolerance build into the database structure.

In the RM, a single table is created to map between a multi-value attribute and a key attribute of the database. Take for example, TABLE 2 of Figure 2, in which "Employee Code" is mapped with "Language". This is an attempt to break the horizontal row straightjacket of a relational table, and introduce "vertical rows". This increases space requirements, as well as the number of expensive joins during access. In contrast, in the present invention, the whole structure is based on "vertical rows", allowing easy support to multi-value fields. The relational model may use "coding" to reduce the size of the composite keys. In the present invention, a similar technique may be used. However, unlike the relational model, not only can the values be coded, but also the attributes themselves. This technique helps in reducing the key size of 3 of the 4 index tables to simply 8 bytes, irrespective of the size of the individual attributes. Using 8 bytes we can address a domain having (231-1) attributes each having (231-1) unique values. In case a lesser number of attributes are required, we could reduce the key size to 6 (i.e. 32,767 attributes) or even 5(i.e. 127 attributes). This improves the efficiency of the index operations as the fan-out increases manifold. And there is no join to be performed later with the "mapping table". We will assume a key size of 6 (maximum 32,767 attributes) for the rest of this document, although other key sizes are possible and fall within the scope of the present invention, as defined in the claims.

The present invention facilitates a high degree of parallelism in the processing of queries involving more than one attribute, and extremely fast in- memory Boolean operations ensure negligible performance degradation when compared with single attribute searches.

An index search involving an inequality (i.e. SELECT * FROM TABLE 1 where DEPARTMENT NOT IN Accounts') can be tackled with ease, unlike in most relational index schemes. In a relational model, this always requires a full table scan. In the present invention, they are tackled like any other query.

Some attributes like "EMPL.CODE" are considered highly selective attribute, as search on this attribute results in a small result set, while a search on either of the two values of a low selective attribute like "SEX" will potentially generate a result set that is half the number of records in the database. Searches on low selectivity (LS) attributes will be slower than searches on high selectivity (HS) attributes, very similar to indexing in RM. However, the structure of the index files is such that in case of low selectivity attributes, the key compression in the nodes of the FRC file can be very high. This leads to higher fan-outs on these nodes, balancing out some of the inherent inefficiencies of the low selectivity attributes.

The present invention allows fast retrieval of a unique list of values for any attribute (or a range thereof). In the relational model, that would have required a traditional SELECT statement, with the UNIQUE post processing. If the attribute does not have an index, then a full table-scan is executed. It is important to allow applications to provide end-users with a list of already entered values. This in turn allows end-users to query the system more knowledgeably, resulting in a more focused search. For example, instead of searching for "Tridib*" the user can now make a high-selectivity search on "Tridib Roy Chowdhury". The present invention allows end-users to avoid entering different values for an attribute, for the same entity. For example, entering "SHOP-FLOOR-II" instead of "SHOP FLOOR 2" leads to update anomalies. However, if a read-only list was provided the user can simply choose 'SHOP FLOOR 2". Performance on access is far superior in the present invention than in the traditional RDBMS model, especially when listing on data-attribute values.

The present invention also performs well in space-time complexities in static domain of datawarehousing (DW), when compared with the performance of the various bitmap indexes. A preferred embodiment of the present invention is based upon B+ Trees, which have been designed to handle the peculiar needs of data-index environment. The information being stored in four B+ Tree files and the uniqueness of information representation provides many of the advantages of the present invention, such advantages not being present in normal B+ Tree structures. However, it will be appreciated that other single or multi-dimensioπal indexing structures could be used in place of some or all of the B+ trees and such implementations are intended to fall within the scope of the present invention as defined by the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Examples of the present invention will now be described in detail with reference to the accompanying drawings, in which:

Figure 1 illustrates a sample data set for an employee database; Figure 2 illustrates a normalised database structure for an employee database, as is used in the prior art;

Figure 3 illustrates the normalised database structure of Figure 2 populated with data from the sample data set of Figure 1 ; Figure 5 illustrates the fields and data of an unpopulated UNI file;

Figure 6 illustrates the UNI file populated with the data from the first record of the sample data set of Figure 1 ;

Figure 7 illustrates the INU file, populated with the data from the first record of the sample data set of Figure 1 ; Figure 8 illustrates the CRF and FRC files, populated with the data from the first record of the sample data set of Figure 1 ;

Figure 9 illustrates the UNI file of Figure 6 populated with the data from the second record of the sample data set of Figure 1 ;

Figure 10 illustrates the INU file of Figure 7 populated with the data from the second record of the sample data set of Figure 1 ;

Figure 11 illustrates the CRF and FRC files of figure 8 populated with the data from the second record of the sample data set of Figure 1 ;

Figure 12 is a flow diagram of a present invention search algorithm searching a database according to an embodiment of the present invention; Figure 13 is a table showing an analysis of an update algorithm for use with a database according to an embodiment of the present invention;

Figure 14 is a table showing Fan-out and height of an embodiment of the present invention B+ Trees (N = 1 ,000,000 records). The height in brackets is the height of a B+Tree index in the RM; Figure 15 is a table showing Cache requirements of an embodiment of the present invention in KB (N=1, 000,000 records);

Figure 16 is a table showing the Cache requirement comparison of an embodiment of the present invention Vs a Fully indexed Relational Table (FIRT) in %(N = 1 ,000,000 records); Figure 17 is a table showing a Disk storage requirement comparison of an embodiment of the present invention Vs FIRT in %(N = 1,000,000 records); and,

Figure 18 illustrates aspects of the UNI file with 10000 records.

Figure 19 is a table showing an analysis of a search algorithm for use with a database according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

Databases according to the present invention include four data structures, each corresponding to a B+ tree. The four B+ Trees are referred to throughout the remaining description as the UNI file, INU file, CRF file and FRC files, although no significance is given by the naming of these files. In addition, although each B+ tree is discussed with reference to a file, the data structures do not need to each be stored within a single file and could be divided across numerous files or stored in a memory. The implementation discussed is merely an example and the skilled addressee will appreciate that many other implementations that fall within the scope of the appended claims are possible.

Data to be stored is in the form of records. Each record includes a number of data fields that store one or more data items. An example of a number of records is shown in Figure 1. With the exception of the header row, each row is considered to be a record. The header row defines the data fields (Emp. Code, Name, Sex, D.O.B., Department, Designation, Language, Qualification, Skill-set). Entries in the records directly beneath a header define the data for that data field. For example, the entry for data field Emp. Code for the first record is "1 ".

Note that the data type of data for data fields may differ. For example, the data type for Emp. Code may be numeric, the data type for Name may be alphanumeric whilst the data type for D.O.B may be alphanumeric or a predefined composite date data type. In addition, as can be seen from the Language, Qualification and Skill-Set data fields, a record may have more than one entry for a data field resulting in a set data type. Each record stored is allocated a unique record ID.

UNI FILE

A sample snapshot of the unpopulated UNI file for the employee record database is shown in Figure 5. The entries shown include to the attribute numbers (attribute #), attribute values, value IDs and a frequency count. The UNI file stores data on attributes, that is, attribute names (data fields) and data itself (data items). In essence, this file stores both the data and metadata identifying the data and its type. Data in the UNI file is stored in records of a generic format. The same generic record format is used irrespective of whether the data stored is on data, or data fields.

A database to be stored is broken down into the attribute names defining the database and the date. Data is reduced to a single dimension so that data of a set data type is split into a number of separate data items, each of which is stored separately.

Each attribute has a unique attribute number.

The Attribute number corresponding to a certain attribute say "Name", is 2 (0x8002 <bit-wise AND> 0x7FFF). Attribute number 0 is reserved for storing the number of records in the database (a count of the Record IDs stored). The attribute value field stores the data itself (i.e. the data field name or data itself). The Value ID field stores information relating to the data type and the maximum length of the attribute value. In Figure 5, the size of the "D.O.B" field is 10 (Oxa) and the data type is date (1), while that of "Name" is 30 and data type is alphanumeric (0) and that of "Language" is 20 (0x14) and data type is that of a set (2).

Figure 6 illustrates the file of Figure 5 after the first record from the sample dataset of Figure 1 is entered. We can see that there are 2 entries each for Attribute # 7(Language), 8(Qualifications) and 9(Skill Sets).

The entries as illustrated in Figure 5, are sorted according to the composite primary key of this file (Attribute #, Attribute Value).

The Frequency Count field for the data values records the number of records of the database where this value occurs for the corresponding attribute. Since only one record has been inserted, all the frequency counts are showing one.

The Frequency Count field for the attribute values (Attribute # > 0x8000) stores the number of unique values that are present for this attribute. Hence, while at this stage only one value is recorded against "Name", there are two unique values recorded against "Language" (English, Hebrew), "Qualifications" (B.Sc,

PhD.) and "Skill Sets"(Databases, OS).

The new or modified entries of this file compared to the state in Figure 5 are shown in bold. The snapshot of the file entries in the UNI after the second record is inserted is shown in Figure 9.

New entries have been made for Attribute # 1 (Empl. Code). As the Attribute Value 2 was not present for attribute # 1 , the Frequency Count for the corresponding attribute entry (0x8001) is incremented from 1 to 2. This new value of 2 is used as the Value ID for the new attribute value entry of (1 , 2, 2, 1 ).

Similarly new value entries (2, Gopalaswamy, T.M., 2,1), (4, 2/7/1954, 2, 1) are made with the corresponding increase in the Frequency Count for attributes "Name" (Attribute # 0x8002), and "D.O.B" (Attribute # 0x8004). However, as the value for "Sex" (M) is already in the table, only the

Frequency Count for that attribute value entry is changed from 1 to 2 (3, M, 1 , 2).

The Frequency Count of the "Record ID" attribute is also changed to 2 to reflect the number of records entered in the database.

The table structure information can be stored in a separate table. In fact, if the size of attribute names is substantially bigger than that of its value, it is recommended that we store this information separately. However, irrespective of whether the table structure information is within the UNI file or in a separate file, operation is substantially the same. In most application environments, the programming convenience of storing them in the same file can be availed of. Intelligent caching ensuring that the leaf storing this information (i.e. the last leaf in the leaf chain), is always in the cache ensures that we do not use disk accesses to access this information. For a disk block-size of around 8 KB, and if we limit the attribute name to 10 bytes, each of these records occupy 19 bytes. Therefore, one leaf can store around details of about 400 attributes. In a preferred embodiment, attribute identifier and Record ID fields have their highest bit set to 1 where they are used for a generic record corresponding to a data field and 0 where they are used for a generic record corresponding to a data item. This allows the data stored in the generic record to be identified more quickly.

INU FILE The INU file may be considered a cross-referenced image of the UNI file. The snapshots of the INU file after the insertion of the first and second records are shown in Figure 7 and Figure 10 respectively.

There are no entries corresponding to the entries corresponding to the attribute names (i.e. Attribute # 0x8001, 0x8002 etc.) and the "Record ID" (Attribute # 0x8000). The frequency counts of the UNI file are not recorded in the INU file.

The key for this file is a composite key, comprising of {Attribute #; Value ID}. The size of this composite key is 6 bytes. The record size is variable, as the "Attribute Value" size is variable.

The maximum number of attributes in a 'table' is 32,767, while the maximum number of values for each attribute is (231-1) or 2,147,483,647. However, the limit on the number of fields is an implementation detail and can be easily increased to (231-1 ) or 2,147,483,647!

CRF FILE

This file maps the "Record ID" to the "Attribute #" and the corresponding "Value ID". The snapshots of the CRF file after the insertion of the first and second record is shown Figure 8 and Figure 11 respectively. The CRF table essentially represents a table vertically, rather than the horizontal representation of the RM table. This allows the structure to store multiple value attributes like "Language", "Qualification", "Skill Sets". These are represented by the "Attribute #" 7, 8, 9. As can be seen in Figure 11 , "Document#"=1 has couple of entries for "Attribute#"=7, and a couple for "Attribute #"=8.

The structure is similar to the structure of TABLE2, TABLE3, and TABLE4 of Figure 2. However, in the present invention, we can reduce all these tables to one index file. This eliminates the need to perform an expensive join operation.

The other advantage of the CRF file over the RM tables is that the size of the composite index is always 6 bytes only, irrespective of the exact sizes of the respective attributes. Therefore, there is no limit in the present invention on the size of the attributes, which is contained in the composite key. Note that the CRF file the entries are not unique on their composite keys, as the same document can have multiple values for certain set-type fields, e.g. Languages, Qualifications etc.

FRC FILE

The FRC file maps the "Attribute ψ and the "Value ID" to a "Record ID". The snapshots of the FRC file after the insertion of the first and second record is shown Figure 8 and Figure 11 respectively .

The composite key for this file consists of {Attribute #; Value ID}, and the size of this key is always 6 bytes.

There is one entry in this file for every record in the CRF file. However, the composite key is different.

This is the primary table used for "table" searching. The small size of the key, which is 6 and independent of the attribute size, allows very efficient searches across alt attributes.

The record size of only 10 bytes, allows for a very efficient TABLE-SCAN (actually a leaf-scan of the FRC B+Tree), when addressing queries relating to attributes of low selectivity.

Note that the FRC file the entries are not unique on their composite keys, as the same attribute value can be found in multiple records, e.g. Sex = M.

Figure 12 is a flow diagram of a present invention search algorithm searching a database according to an embodiment of the present invention;

The result of a search is a Record ID, which uniquely identifies a collection of content object. As discussed above, attributes can be any different type, like alphanumeric, numeric, decimal, date, time etc., or a set of these atomic types. Types like numeric, decimal, date, time can have their values stored as a unique number, which will also maintain their natural sort order. Date type fields can store data internally as "yyyymmdd". These attributes are called Numerically Representable (NR) attributes. In such cases, it is important to note that a corresponding entry is not required in the INU table. The "Value ID" of the various values can be mapped to the actual attribute values. The entry in the UNI table is required to keep the "Frequency Count", and also to provide the "Unique Value" list. There is a strong possibility that in certain domains, the entry in the UNI for these attributes is also not warranted.

For entries where the Frequency Count is 1, we can store the unique Record ID against that entry. This obviates the requirement to search in the FRC file during query processing. The highest bit indicates whether the frequency count or the Record ID is being stored. These entries are known as Totally Selective (TS) entries. Assuming both the unique record ID and frequency count are numeric, we would need some way of differentiating between them when stored. We use the highest bit to indicate whether the stored value is a frequency count or a unique record ID. If the highest bit is 1, then we logically AND the value with 0x7FFF to retrieve the unique record ID. If the highest bit is 0, then the value is the frequency count. A skilled reader will appreciate that the interpretation of the highest bit could be reversed without departing from the scope of the invention, as defined by the claims.

DATABASE OPERATIONS

Figure 12 is a flow chart of the basic database search/query algorithm. The examples discussed below relate to the structured query language (SQL), although it will be apparent to the skilled addressee that queries in other formats could easily be handled and fall within the scope of the presently claimed invention.

The search algorithm will now be discussed with reference to an example query, an Exact Match Query of the form: SELECT EMP.CODE, NAME

FROM EMPLOYEES

WHERE DESIGNATION ='PROJECT LEADER' AND QUALIFICATION ='BSc.' OR QUALIFICATION ='PhD.'

The search attributes are "DESIGNATION" and "QUALIFICATION". The select attributes (or the reply attributes) are "EMPL.CODE" and "NAME". There are two search criteria - records must include a designation of "Project Leader" and a Qualification of "PhD" or "BSc". Each search criterion is initially processed separately. In step 10, the Value ID for "DESlGNATION"='LEADER' is obtained. This is done by searching the UNI index file with the composite attribute {"Attribute # 6"; "Attribute Value = PROJECT LEADER"} in step 10. This gives the Value ID, which is stored in an a-list in step 20.

In step 30, it is determined if more search terms need to be resolved. In the present case there are more search terms to be resolved and steps 10 to 20 are repeated for the other two search terms.

These processes can be run serially, as discussed above, or in parallel.

At this point we have a list of tuples {"Attribute #", "Value ID"}, one for each search term stored in the a-list.

In step 40, the FRC file is searched for each entry in the a-list with the tuple as the composite key. For example, for the designation search term, the tuple {"Attribute # = 6"; "Value ID = 1"} is the composite key. The B+ Tree search will retrieve the first record in the FRC file, which matches this composite key. From the search results, the Record ID for the record is obtained. A scan is then made through the leaf link-list of the FRC file to obtain all other records that match this composite record.

In step 50, for each Record ID obtained, a corresponding bit in a search bit map is set corresponding to this sub-query.

Steps 40 and 50 are repeated for all the entries in the a-//sf.

Again, although steps 40 and 50 are discussed as being applied serially to each entry, they could be applied in parallel.

Note for queries on HS entries, the FRC search is not required as the UNI file will automatically generate the Record ID for the sub-query. We now have 3 bit maps one for each of the sub-queries. These bitmaps are then combined in step 70 using the Boolean operators set in the query (above, in this case bitmapl AND (bitmap2 OR bitmap3)) to generate a results bitmap containing the Record ID(s) that match the query.

In step 80, for each of the Record IDs found in step 70, the CRF file is searched with the composite key {"Record ID"; "Attribute #=1"} to obtain the Value ID. Note that the select attribute with the smaller Attribute # is EMP.CODE, whose Attribute # is 1.

A leaf scan (almost always the same leaf as before) is the used to get the

Value ID for the remaining select attributes for this Record ID in step 80.

Now we have a table in memory, where the columns are attributes, and the rows are the Record IDs, each cell containing a Value ID.

In step 90, for each column (attribute), we do the following to translate the Value IDs into actual values to be output as the result of the query:

1. Generate a set of unique Value ID for this column, and sort by Value

ID.

2. For each Value ID, search INU for {Attribute # ;Value ID}.

3. Get the actual value.

4. Replace all entries in the column with the same Value ID with this value.

Note that the search in 2 may not always require a full index search or even a leaf scan. In case the next "Value ID" is in the last accessed leaf, we do not have to do any further operation. This is very relevant in case of low selectivity attributes. For NR attributes, access to the INU is not required.

Range Queries:

In the case of the SQL Query: SELECT EMP.CODE, NAME FROM EMPLOYEES

WHERE DESIGNATION ='PROJECT LEADER' AND QUALIFICATION ='BSc.' OR D.O.B. >= '01/03/1970' Such a query could be performed on a database according to an embodiment of the present invention as follows:

In step 10 the Value ID for "DESIGNATION"='PROJECT LEADER' is determined.

In step 10 the UNI index file is searched with the composite attribute {"Attribute # = 6"; "Attribute Value = PROJECT LEADER"}. This gives a "Value ID" which is stored, in step 20 in an a-list.

The same steps are repeated to determine "Value ID" for

BQUALIFICATION"='B.E.' and for "D.O.B." >=O1/03/1970'.

As before, these processes can be run in parallel.

Steps 40 to 90 are then the same as above.

Searched with wild cards and NOT condition:

In the case of the SQL Query: SELECT EMP.CODE, NAME FROM EMPLOYEES

WHERE SKILL SETS ='Data*' AND

QUALIFICATION ="BSC OR D.O.B. >= '01/03/1970' AND

DEPARTMENT <> 'R&D'

Such a query could be performed on a database according to an embodiment of the present invention as follows:

In step 10, the Value IDs for "Skill Sets"= 'Data*' is determined. In step 10, the UNI index file is searched with the composite attribute {"Attribute # = 6"; "Attribute Value = Data*"}. This will give a list of "Value ID"s. A leaf scan on UNI would be required to generate all the "Value ID"s matching the criterion these are then stored in the a-list in step 20.

The same steps are repeated to determine "Value ID" for "QUALIF(CATION"='BSc.', for "D.O.B">='01/03/1970' and for "Department"='R&D\

As before, these processes can be run in parallel.

Steps 40 to 60 are as previously described.

We now have 4 bit maps, one for each of the sub-queries. In step 70, these bitmaps are then combined using the Boolean operators according to the query above, viz. {Bitmapl AND (bitmap2 OR bitmap3) AND !bitmap4} to generate the final bitmap containing the Record ID which matches the query.

Steps 80 to 90 are then the same as described above.

Searches With Set Operators:

In the case of the Superset predicate query:

SELECT EMP.CODE, NAME FROM EMPLOYEES

WHERE QUALIFICATIONS 2 {'MBA' CPA'} AND QUALIFICATION ='CS' OR D.O.B. >= '01/03/1960' AND

DEPARTMENT <> 'FINANCE'

This can be reduced to the following query: SELECT EMP.CODE, NAME FROM EMPLOYEES

WHERE QUALIFICATIONS ='CPA' AND QUALIFICATION ='MBA' AND QUALIFICATION ='CS' OR D.O.B. >= '01/03/1970' AND DEPARTMENT <> 'FINANCE' Subset Predicate:

SELECT EMP.CODE, NAME FROM EMPLOYEES WHERE QUALIFICATIONS c {'CPA', 'MBA'} AND

QUALIFICATION ='CS' OR D.O.B. >= O1/03/1970ΑND DEPARTMENT <> 'FINANCE'

This can be reduced to the following query: SELECT EMP.CODE, NAME

FROM EMPLOYEES

WHERE QUALIFICATIONS ='CPA' OR QUALIFICATION ='MBA' AND QUALIFICATION ='CS' OR D.O.B. >= '01/03/1970' AND DEPARTMENT <> 'FINANCE'

Overlap Predicate:

SELECT EMP.CODE, NAME FROM EMPLOYEES

WHERE QUALIFICATIONS {'CPA'.'MBA, 'BSc\ 'PhD'} > 2 AND QUALIFICATION ='CS' OR D.O.B. >= '01/03/1970' AND

DEPARTMENT <> 'FINANCE'

This query will need to be processed in two ways. The standard way would be process it as we did for the subset predicate query, except we make a change to the OR operation of the bitmaps generated from the sub-queries QUALIFICATION='CPA', QUALIFICATION='MBA'... The OR operation does not do a Boolean OR. It checks for each bit-position the number of 1s. If the number is > 2, the resultant bitmap bit is set to 1. Else it is set to zero.

The second way would be to ignore the overlap predicate in step 10, but apply on-the-fly selection in step 80, on the overlap attribute QUALIFICATIONS.

QUERY OPTIMIZATION In studying the above algorithms, it is quite apparent that the algorithm for search is essentially the same irrespective of the nature of the query. Therefore, there is no need for query optimization, query path selection.

However, that is not to say that we cannot use query optimization techniques. In step 10, we can get the "Frequency Count" for various "Attribute #; Attribute Value" pair at no extra cost. We could then use this to see whether all the sub-queries of step 40, are required. For example, if the query is as follows:

SELECT NAME FROM EMPLOYEES

WHERE D.O.B = '1/3/1970' AND DEPARTMENT='FINANCE' In this case, if the D.O.B- 1/3/1970' has a frequency count of 4, while the other sub-query has 100. Then it might be better to do step 40-50 for D.O.B. and go straight to step 80, and apply 'select' on the fly on the 4 {Record ID, Attribute*, Value ID} tuples found with {Attribute #=5;Value ID=2}.

INSERTING

Input:

A set of tuples in the following format:

{"Emp. Code", "12345"},{"Name", "James Butcher"}, {"Sex", "Male"}, {"Department", "Accounts"}, {"Designation", "Manager"}, {"Languages Spoken", "English"}, {"Languages Spoken", "Hindi"}, {"Skill Sets", "As400"}, {"Skill Sets", "WebSphere"}, {"Qualification", "MBA"}, {"Date Of Birth" , "23/01/1963"}.

Output:

A Record ID - positive in case of successful entry, -1 for unsuccessful attempt. Pre-processing:

The pre-processing stage converts the set of tuples to the following tuple, using the structure stored in the UNI file. This stage checks for data validity (type and size) and does not require any disk accesses as the concerned leaf storing the structure is always in the cache. The output of the pre-process is:

KVN - the total number of tuples in the input. KVARRAY- An array of tuples of the type {1 , "12345"}, {2, "James Butcher"}, {3, "Male"}, {5, "Accounts"} ,{6, "Manager"}, {9, "English"},{9, "Hindi"}, {8, "As400"}, {8, "Websphere"}, {7, "MBA"}, {4 , "23/01/1963"}.

Sorting on "Attribute #" is not mandatory. The insert algorithm for inserting a record (having a record ID of rid) is explained below in a pseudo-C code. (Special consideration for HS entries has not been discussed). typedef struct

{ unsigned short attributeid; char valuef ]; } KVARRAY; long insert(short kvn, KVARRAY *kva)

{ int i; long vid; long rid;

if(!(rid = getnextrid()) return -1 ; for (i = 0; i < kvn; i++)

{ if (!(vid = uniinu_insert(kva[ij. attributeid, kva[i]->value, rid))) goto err; if (!(crffrc_insert(rid, kva[i]. attributeid, vid))) goto err;

} return rid; err: deletelastrid(rid); return -1;

} long getnextrid()

{ //search in the UNI file for the entry {0x8000}. / The "Frequency Count" field will contain the last Record ID (rid). // frequenct_cnt++ // return frequency__cnt } long uniinu_insert(uπsigned short ano, char *value)

{ long frq; long vid;

// if {ano, value} already exists in the UNI

{ // let frq = "Frequency Count" stored in UNI against {ano, value}.

// frq = f rq + 1 // update the "Frequency Count" in UNI against the {ano, value}.

// return vid = "Value ID" stored against {ano, alue}.

}

// else

{ // frq = "Frequency Count" stored in UNI against {ano | 0x8000}.

// frq = f rq + 1

// update the "Frequency Count" in UNI against {ano j 0x8000}. // insert a new record in UNI {ano, value, frq, 1}. // insert a new record in INU {ano, frq, value}. // return frq

} }

long crffrc_insert(long rid, unsigned short ano, long vid) {

// insert record in CRF {rid.ano, vid} // insert record in FRC {ano, vid, rid} }

DELETION The input to this process would be a Record ID (rid).

The insert algorithm is explained below in a pseudo-C code.

long delete(long rid) { char value]]; long frq;

// search CRF to generate a list of (Attribute #, Value ID), from the records which match {rid}. Call this list the FRC-list

// delete all the records of CRF which matches {rid}. // for each entry in the FRC-list (ano, vid)

// delete from FRC the entry (ano, vid, rid) // end for // for each entry in the FRC-list (ano, vid)

// find value = "Attribute Value" from INU for record {ano, vid} // find UNI record entry {ano, value} // frq = "Frequency Count" of UNI record {ano, value}. // frq = frq - 1 // iffrq = 0 then

//delete from UNI {ano.value}. //delete from INU {ano, vid}. //else // update the UNI record {ano, value} with a new value for "Frequency Count" = frq.

// end for

}

Figure 13 is a table showing an analysis of an update algorithm for use with a database according to an embodiment of the present invention. Figure 20 is a schematic diagram of a computer system according to an embodiment of the present invention.

A data storage system 100 includes a storage medium 110 and a management system 120. The storage medium 110 may be memory, a hard disk, a number of hard disks or some other computer readable medium. The B+ tree structures as discussed above are encoded and stored on the storage medium 110 in a predetermined manner.

A user wishing to access data in the data storage system 100 accesses a user terminal 130 having a user interface 140 that is arranged to communicate with the management system 120 via a network 150 such as a LAN, WAN, internet, intranet or a simple computer data bus. Data can be accessed, updated and manipulated by the user interface 140 and management system 120 in accordance with the algorithms and procedures set out above.

It will be apparent that many implementations of the present invention are possible, both in a networked computer environment and where the user interface and database reside on the same computer system. The above described computer system is just one example of a system that is intended to fall within the scope of the claims, although the skilled reader will appreciate that many modifications and variations are possible without departing from the scope of the invention, as defined by the claims.

PERFORMANCE ANALYSIS OF THE B+ TREES

We have already mentioned that the height of a B+ Tree determines how effective the tree is from the point of view of access efficiency. The height also determines the memory requirement to store the internal nodes of a B+ Tree. For a tree using page block size as B, key size = k, and fan-out of m, and height as h, and total number of records N. Some basic formulae: 1. Fan-out : m = |_B/kJ. 2. Height: h = logmN.

3. Total Node Size: tns = B*Zm', 0<=i<=(h-2).

4. Number of attributes in the table = NOA.

The number of disk accesses determines speed of access.

■ The number of disk accessed is determined by the height of the tree. The height of a B+ Tree is a function of two parameters, fan-out and number of records. Higher fan-out, or smaller record set reduces the height of the tree. The fan-out is a function of the key size. Smaller the key size, larger the fan-out and vice versa.

Assuming that we are talking about 10,000 employees, we can expect to have around 10,000 "Emp. ID" values, 10,000 "Name" values, 2 "Sex" values, 7000 "Date of Birth" values, 20 "Department" values, 30 "Designations", 20 "Qualification", 100 "Skill Sets" and 20 "Languages Spoken". The values for UNI file would be:

UNIisj = 10000+10000+2+7000+20+30+20+100 + 20 = 29,192 values. {Total number of records in the UNI file} U Ifc = (Average size of a key value + 2). {Size of the key in the UNI file}

FRC file can use key compression for both nodes and leaf storage. The compression prefix for the leaf would be {Attribute #;Value ID}, and for the node it would be {Attribute*}. The compression in the leaves would be very high, especially for leaves storing attributes of low selectivity. Therefore, the cost of a leaf scan would be proportionately cheaper (often leading to no additional overheads) when querying on the attributes of low selectivity. While querying on attributes of higher selectivity, no leaf scan is required.

Instead of storing {A1 , V1D1, DN01},{A1NID1 , DΝ02}, {A1, VID1, DNO44}, {A1 , VID2, DN045}, {A1 , VID2.DN056 },{A1 , VID2, DN077} {A2, VID1, DN01},{A2, VID1, DN034}.... we can store{ Al, VID1, DN01},{DN02}, {DN044}, {A1, VID2, DN045}, {DN056 },{DN077},....{A2, VID1, DN01},{DN034}.... The highest bit of the bit of any tuple will indicate if the previous {Attribute #, Value ID}continues or a new sequence is starting. That is why we do not use all the 32 bits to represent the Attribute #. As nodes are also compressed (though the compression is less than that at the leaf), especially in case of low selectivity attributes, fan out increases. So the performance of the FRC B+ Tree is nicely balanced to offset the adverse effects of low selectivity attributes.

F CN = ∑Rj, 1 <=i<=N. Rj is the number of attributes in itn record of the database. It is not simply (N * NOA), as some attributes may have multiple values for certain set-type attributes, and some records may have null values for certain attributes, though (N * NOA) is a good working approximation. In this case it is 90,000.

FRCk = 6. CRF file can use key compression for both node and leaf storage. The key compression prefix would be the "Record ID". However, in the CRF file, the possibility for compression at the node would be very small, unless we have relational with a very large number of attributes. However, the leaf level compression would be approximated by the following:

Compression of CRF leaf =

(4 +(6 * NOA))/(NOA * 10)

Instead of storing {RID1, A1 , VID1},{RID1 ,A1NID10}, {RID1, A2, VID200}, {RID2.A1, VID2}, {RID2.A2, VID10},{RID2, A2, VID2}...., we can store{RlD1 , A1 , VID1},{A1 NID10}, {A2, VID200}, {RID2.A1, VID2}, {A2, VID10},{ A2, VID2}... The highest bit of the bit of any tuple will indicate if the previous Record ID continues or a new Record ID sequence is starting. That is why we do not use all the 32 bits to represent the document number.

CRFfM = FRCΝ CRFk= 6.

The I U has a key structure similar to that of the FRC, and hence all that is applicable for node compression is applicable to this file also. However, in case of attributes of larger value sizes, the impact of the leaf compression is minimal, and is not used. The balance in the case of the IΝU file is attained by noting that the number of records to be stored in the IΝU file is the same as that of the UΝI file.

IΝUjvj = UΝIΝ

IΝUk = 6.

We will introduce a term called Unique Value Index (UVI) and it is defined as:

UVI = U IN FRCN S UN1N/(N * NOA).

In the above example where N = 10000, UVI = 0.29. It can be shown that UVI is a monotonicaUy decreasing function of N for any environment. Typical values of UVI would range between 0.10 and 0.30. The UVI value is significantly less when we consider the impact of NR attributes.

All the four B+ Trees are finely tuned to take advantages of the B+ Tree structure. Where the number of entry is large, the key size is small thereby increasing fan-out, and where key size is large, the number of records is small. The height and the fan-out (F.O.) of the various B+ Trees under different environments are shown in Figure 14.

The numbers in the brackets, indicate the height of a B+ Tree index for a similar environment for a relational table. We have assumed a very conservative INU node compression factor of 0.80, FRC node compression of 0.70 and FRC leaf compression of 0.30. UVI has been taken at a high 0.30.

The heights of all the trees are between 1.00 and 1.40. The heights of all the trees except UNI are totally independent of the attribute size. It is only determined primarily by N, and lesser on the number of attributes. Performance analysis based on tree heights, however, ignores the impact of caching while implementing in a practical environment. In Figure 15 we present some of the cache requirements (in KB), which will allow us to store all the internal nodes (or directory nodes) of all of the B+ Trees in memory. This means that any access will require only access of the leaf nodes. The results can be interpreted in two ways. The total cache requirement even for 1 ,000,000 records is less than 1 MB (MB = 1024 KB).

The other way to look at it, in case more memory is available, which is not uncommon in today's machines, is that a substantial percentage of the leaf nodes can also be included in the cache, further improving access speeds. Leaf caching also ensures that frequently accessed FRC leaves of low selectivity attributes are more likely to be in the cache than leaves of higher selectivity attributes. Another instance of balancing out of the negative effects of the low-selectivity attributes in a query. The Cache requirements of present invention vis-a-vis a fully indexed relational table is shown in Figure 16. A fully-index relational table (FIRT) is a RM table with a B+ Tree index on each and every attribute. As expected, the cache storage efficiency of the present invention improves with increasing attribute size as well as increase in the number of attribute. For higher values of N, the present invention performs better compared to smaller values of N as shown in Figure 16. (Note that the negative figures indicate where FIRT is better than the present invention)

The results of the total disk storage follow the same pattern as witnessed for the cache requirements and shown in Figure 17. The difference is the performance parameters for storage of the present invention are always better than that of the cache requirements. The performance is even more dominating for lower values of UVI.

It can be conclusively stated that the present invention always outperforms FIRT under almost all conditions for the criterion of cache requirements or disk storage requirements.

PERFORMANCE COMPARISONS

An analysis of the performance of an embodiment of the present invention vis-a-vis FIRT and bitmapped index is provided below. Only FIRT and bitmap index has been chosen for comparison, as none of the existing multidimensional index structures can handle all the capabilities of the present invention.

ACCESS ANALYSIS

Let us assume the query: SELECT EMP.CODE, NAME FROM EMPLOYEES

WHERE DESIGNATION ='MANAGER' AND QUALIFICATION='B.E.'

The number of search attributes is 2, i.e. "Designation" and "Qualification". (s = 2).

The number of attributes in projection is 2, "Emp. Code", "Name", (p = 2).

The number of documents be 100,000.(Λ/ = 1 ,000,000).

Page size is 8 KB. {B = 8096 - overhead s 8048).

UVI = 0.3 (u = 0.3) The number of records to satisfy the first sub-query (DESIGNATION...) is ft, and that of the second sub-query (QUALIFICATION...) is f2. The number found in the found list is f, which would be in the range of [0, mm(f1,f2)].

For both the FIRT and the present invention we will assume that the intermediate nodes are stored in cache memory.

EMBODIMENT OF THE PRESENT INVENTION Assume that the FRC leaf compression is 0.33. Figure 19 illustrates example disk access overheads.

Total number of disk accesses: s + f(1 + pu) + 1+ (fl+f2 - 2)/(LB/3j). in this example, we can further define this equation to s + f(1 + 0.3p) + (f1 + f2 - 2) /(LB/3J). In the example chosen, the result would be 2 + 1.6f + .0037 (f1 +f2- 2) disk accesses. This would be less in case there are any NR attributes are in the projection operation or if there are HS entries are in the selection operation, as p and f1 (or f2...) would be reduced. The complexity is of the order of 0(f) and O(s) and is independent of the attribute size or dimension of the relation, and only dependent on the number and selectivity of attributes in the query.

FIRT

The number of records to satisfy the sub-query DESIGNATION-'Manager" condition in TABLE1 is f1. Let us assume that we have an index file on "Department", where the index key size is k 1 and 6 bytes for ROW-ID.

No. of disk accesses for the index file (assuming caching of internal nodes) on "Designation" on Table 1: 1 + (f1Λ )/( l_B/(/ ?+6)J) = (A)

The number of disk operations to use create a temporary relation TABLE T1 such that T1 = σDES|GNATl0N= ,pROJECT LEADER,(TABLE1 ) is (A + fi). Assume this value is (C).

Let us assume that we have an index file on "Qualification", where the index key size is 10. Let the number of hits on TABLE3 for each value found in (A) is f2.

No. of disk accesses for the index file on "Qualification" on Table 3: 1+ (/2- 1)/{ B/(k2+6) = (D)

The number of disk operations to use create a temporary relation TABLETS such that T3 = σ QUAUF|CAT|0N= ,BE,(TABLE3) is (D + f2). Assume this value is (E).

An join needs to be performed between Table T1 and Table T3 on the common attribute "EMP.CODE". The cost of the index nested join between TABLE T1 and TABLE T3 would be an operation of 0( ^ 2) complexity and is dependent on the attribute size, as the important sub-components of the operation are dependent on k1, k2, If either of the index files were not available, the cost of the join would have been very high as we would have required to do table-scan for the outer relation.

DISJUNCTIVE QUERIES The query analyzed above would is a conjunctive one. In case of a disjunctive element in the query, like: SELECT EMP.CODE, NAME FROM EMPLOYEES

WHERE DESIGNATION ='PROJECT LEADER' OR QUALIFICATION='BSC

The embodiment of the present invention would result in almost a similar access formula. However the FIRT model would suffer gravely because of the need to do a TABLE-SCAN on the outer relation of the join. Most current systems do not handle disjunctions efficiently, and concentrate instead on optimizing selection without disjunctions.

IMPACT OF CACHING

Caching would positively impact the model of the present invention in two ways:

The internal nodes of the embodiment according to the present invention take substantially less space than the FIRT model, for a given cache size, a larger percentage of the leaf nodes can be cached at a given time, than the FIRT model. In the present invention, we see that instead of storing leaf nodes for all the B+ Trees equally, it is beneficial to given a greater weight to leaves of the INU and FRC than those of any others.

DATABASE DESIGN

One advantage of present invention is in the ability to provide good performance on arbitrary queries irrespective of the data set and the attribute set it represents. However, that does not mean that in case we have knowledge of the domain before hand, we cannot use that to improve performance even further.

For example, we have selected a certain order of the attributes. However, if we know that a lot of queries will be made on a certain set of queries, and then we could group them together in the attribute list. This will ensure that we increase the possibility of having to access the same FRC leaf /node from the cache for the various sub-queries of a query. This is also relevant for the INU tree.

INSERT ANALYSIS Assume that there are (p) attributes to be inserted for a certain data object.

EMBODIMENT OF CURRENT INVENTION

For each attribute, the steps need to perform is shown is Figure 13.

The insert process takes between (4 * p) and (5 * p) disk accesses. However, instead of making p CRF disk accesses, we could reduce this to only one or two disk access by delaying the write-out of the CRF leaf, where all these inserts will be made.

So, the actual # of disk accesses reduces to between (3 * p) + 1 and (4 * p )+ 1. The impact is less in case of NR attributes.

FIRT

No. of disk accesses to insert attribute value to each of the index tables: 1

Total number of disk accesses to insert in index table: p

No. of disk accesses to update table: 1 Total number of disk accesses is: 1 + p.

For both FIRT and present invention we have ignored the effects of the leaf/node splits. However, it is important to note that the effect of this would be much less for the present invention trees than the index of the FIRT, because of higher fan-out and lower key size. Though the FIRT is better than the present invention, the complexity is of the same order of magnitude O(p). COMPARISONS AGAINST BITMAP INDEX

We will compare the simple bitmap index and the encoded bitmap schemes with the present invention. We will use a simple datawarehouse database of the following fact table and descriptor tables: DESC_TABLE_1(Aι ι , A12, A13, A<| , .... )

DESC_TABLE_2(A21 , A22, A23, A24 )

DESC_TABLE_3(A31 , A32, A33> A34 )

FACT TABLE(A11 , A21 , A31 , A41 , F)

Obviously, A are the primary keys to the corresponding descriptor tables.

A typical select statement would be of the form: SELECT SUM(FACT.F) where Condition The condition would be made up of sub-queries containing various attributes of the descriptor table. Assume that the condition contains a total of d attributes.

We will use he multi-table joins as suggested in Multi-table Joins Through Bitmapped Join Indices, ACM SIGMOD Records, September 1995 by P. O'Neil and Goetz Graefe . We assume that through some process we have generated the list of fact table attributes from the descriptor table.

We will define T^\ as the number of tuples in the DESC_ TABLE_k, while \T\ is the number of tuples in the fact table.

We will define Akj- as being the im attribute in the condition, and that this attribute is same as Akj of DESC_TABLE_k.

We define ή that the number of tuples matching the itn attribute of the condition in the corresponding descriptor table.

SIMPLE BITMAP INDEX: Size of each bitmap (bytes): ϊ\ T\/8] =\ T\18

Number of bitmaps to access: f<| + f2 + f3 + ... . Number of index blocks to be read: ((|7|)/8B)( f-j + f + f3 + ... ) =(|T|/(8B))( |Tk1|/|Akj1 | + |Tk2|/|Akj2|+....+|Tkd|/|Akjd|)

ENCODED BITMAP INDEX: Size of each bitmap(bytes): (~| 71/81 =| 7|/8

Number of bitmaps to access: log |Akj1|+ log |Akj2|+... log |A jd| Number of index blocks to be read: =(|7|/(8B))( log lAkj1l+ log2|A j2|+... Iog2|Akj d|)

EMBODIMENT OF PRESENT INVENTION:

Remembering the size of the FRC file is 10 bytes.

Number of documents that satisfy the FACT table for the i* query attribute is (|T|/|T '|) * fj. Replacing fj with

|T '|/(A ji|, we arrive at the following equation. Number of index blocks to be read: s (10|T|/B)( 1/|A j1| + 1/|A j |+....+

1/|Akj°D)

ANALYSIS:

It can be seen that the simple bitmap index performs least efficiently. The embodiment of the present invention would work better than the encoded bitmap index if the average cardinality of the query attributes is greater than 19. If however, we assume FRC key compression (typically around 0.3) this number is reduced to 8.

Bitmap compression processes would reduce the disk space requirements for both simple indexes as well as encoded indexes. However, the impact vis-a-vis the embodiment of the present invention would be still not significant. The downside would be a significant overhead on compression and decompression.

The compression technologies like ExpGol, BCC, LZ provide high compression ratios of around 0.1 when the bit density is close to 0.0001. For bit density between 0.2 and 0.98 the best is LZ offering ratios between 0.2 and 1.0. Only BCC and LZ offers any compression at bit densities of > 0.5. These results are published in Performance Measurements of Compressed Bitmap Indices, Proceedings of VLDB Conference, Edinburgh, Scotland, 1999 by Theodore Johnson. So the above magic number on cardinality would improve to 80 with compression (at 0.3) and bitmap indices (at 0.1 ).

In Improved query performance with variant indices, ACM SIGMOD '97, 1997 by P.O'Neil the aggregate function SUM is discussed with respect to other schemes like projection index, bit-sliced index and value-list index. No INU access is required for NR attributes. Once the foundset (stored as a bitmap in the embodiment according to the present invention) is generated, the CRF access becomes the same as that of the Value-List Index. This has the same disk complexity as the bit-sliced index assuming that we are using the full 32 bits to store the values of F attribute.

BUILT-IN FAULT TOLERANCE

The four B+ Trees of the present invention are designed to perform increasingly better with increasing number of records, as well as increasing number of attributes or dimensions. With larger databases, the issue of recovery in case of storage failures, or other system malfunction assumes a greater importance.

The B+ Trees are designed such that there is an automatic built-in fault tolerance capability. For example: 1. The UNI file can be created from the INU and the FRC file. INU and

FRC file are the partner files for UNI file.

2. The FRC file can be re-created from the CRF file.

3. The CRF file can be re-created from the FRC file.

4. The INU file can be re-created from the UNI file. This facilitates two advantages:

1. Quick recovery in case of storage problems with any of the files. Facilities development of efficient data integrity checking programs, which can be run to validate the data. APPLICATION AREAS

Document content management systems (DCMS) require a highly indexed environment, as documents need to be accesses through different access path. A DCMS environment like most OODBMS domains seldom requires a projection over the associated attributes, but needs to access the document content or objects, which are, accessed though the OBJECT-ID stored in the Record ID.

Knowledge management systems, digital asset management systems, medical databases, data warehousing, OLAP databases which requires multidimensional access through arbitrary queries are ideal environments for implementation of the present invention.

Object Oriented Databases, XML databases which require storing unnormalised data and setting data types can effectively use the structures and techniques described herein. These applications also deal with arbitrary queries and have indexing needs in high dimensions (> 3).

This does not preclude the usage of the present invention along with relational databases. While relational systems typically use B-Trees, hashed index or bit-mapped index for their index requirement, in a highly indexed environment the present invention can be used. The result of the searches would be a list of ROW-IDs very similar to the traditional index structures.

In the three-tier architecture of system design, we can finally achieve total separation of the business middle-layer from the lower data-storage layer. This is because all access paths are optimized (as all paths are "indexed"). Hence, no fresh paths need to be added because of any change or addition to the business layer. This will reduce deployment time in case of new applications, as well as changes to existing applications.

It should be understood by those skilled in the art that the forgoing description is only of the preferred embodiments and that various changes and modifications can be made in the invention without departing from the scope thereof. The scope of the invention is defined by the appended claims.

Claims

Claims:
1. A data structure for storing data records, each data record including a plurality of data fields capable of storing one or more data items, the data structure comprising: a plurality of data stores arranged, in combination, to store and index said data records when reduced to a single dimension.
2. A data structure according to claim 1, wherein each data store comprises a B+ tree structure.
3. A data structure according to claim 1 or 2, wherein the data stores includes a first data store including a generic record of a first type for each unique data item of each of said data fields.
4. A data structure according to claim 3, wherein said first data store includes a generic record of said first type for each of said data fields.
5. A data structure according to claim 3, further comprising a further data store including a generic record of said first type for each of said data fields, said further data store being associated with said first data store.
6. A data structure according to claim 4 or 5, wherein for a data field each generic record of said first type includes: an attribute identifier, the attribute identifier for each data field being unique; a value ID defining a data type of the data field's data items; and, a value defining the data field.
7. A data structure according to claim 6, wherein for a data item each generic record of said first type includes: an attribute identifier comprising the attribute identifier of its respective data field; a value ID comprising a unique identifier with respect to its respective data field; and, a value comprising the data item.
8. A data structure according to claim 6 or 7, wherein each generic record of said first type includes a frequency count, the frequency count for a data field comprising the number of data items stored for said data field.
9. A data structure according to claim 8, wherein each generic record of said first type includes a frequency count, the frequency count for a data item comprising the number of instances of said data item occurring in said stored data items for the respective data field.
10. A data structure according to claim 9, wherein if the frequency count for a data item is 1 , the frequency count stored in the generic record of said first type is replaced with an encoded unique record ID corresponding to said data record.
11. A data structure according to any of claims 6 to 10, wherein the first data store is indexed on a composite key of the attribute identifier and the value.
12. A data structure according to any of claims 6 to 11, wherein the second data store includes a generic record of a second type for each data item, each generic record of said second type including said attribute identifier, said value ID and said value, the second data store being indexed on a composite key of the attribute identifier and value ID.
13. A data structure according to any of claims 6 to 12, wherein the third data store includes a generic record of a third type for each data item, each generic record of said third type including a unique Record ID identifying the data item's respective data record, said attribute identifier and said value ID, the third data store being indexed on a composite key of the Record ID and the attribute identifier.
14. A data structure according to claim 13, wherein the fourth data store includes a generic record of said third type for each data item, the fourth data store being indexed on a composite key of attribute identifier and value ID.
15. A data structure according to any of claims 2 to 14, wherein records in the leaves of one or more of the B+ Trees are compressed.
16. A data structure according to any preceding claim, wherein data items of multi-dimensional data fields are mapped in a numerical and order- preserving manner, said mapped values being stored in the data structure.
17. A method of storing data records including a plurality of data fields storing one or more data items comprising: reducing the data records to a plurality of records of a single dimension; storing and indexing said plurality of records across a plurality of data stores.
18. A method claimed in claim 17, wherein the plurality of data stores comprises 4 B+ trees.
19. A method according to claim 18, further comprising the steps of: Creating a unique record identifier for every data record; Generating a unique attribute identifier for each data field; Assigning said unique attribute identifier to each unique data item of said data field;
Assigning a value ID to each data field, the value ID defining a data type of the respective data item(s);
Generating a value ID for each unique data item, the value ID comprising a unique identifier for said data item with respect to said respective data field;
Maintaining a frequency count for each data field or data item, the frequency count for a data field comprising the number of data items stored for said data field, the frequency count for a data item comprising the number of instances of said data item occurring in said stored data items for the respective data field; and,
Storing at least selected parts of the data fields and/or data items and associated unique record identifier, attribute identifier, value ID and frequency count in each of the four B+ Trees.
20. A method according to claim 19, wherein for a new data record to be inserted, the method comprising:
Creating a unique record identifier for the data record; For each data item of each data field of the data record:
Identifying the unique attribute identifier for said data field; Determining if the data item is stored with respect to said unique attribute identifier;
If said data item is stored, incrementing the frequency count for the data item;
If said data item is not stored, generating a value ID for the data item, the value ID comprising a unique identifier for said data item with respect to said respective data field, assigning a frequency count to said data item and incrementing the frequency count of the data field; and, Storing at least selected parts of the data item and associated unique identifier, attribute identifier, value ID and frequency count in each of the four B+ Trees.
21. A method according to claim 19 or 20, wherein if a data record is to be deleted, the method comprising:
For each data item of each data field of the data record: Identifying the unique attribute identifier for said data field; If the frequency count of the data item stored with respect to said unique attribute identifier is greater than one, decrementing the frequency count;
If the frequency count is one, deleting at least selected parts of the data item and associated unique identifier, attribute identifier, value ID and frequency count from the four B+ Trees.
22. A data storage system for storing data record, each data record including a plurality of data fields storing one or more data item, the data storage system comprising: an interface and a database; the database comprising four data stores encoded in a computer readable memory, the data stores being arranged, in combination, to store and index said data records when reduced to a plurality of records of a single dimension; the interface being programmed to accept at least selected ones of: input of a new data record to be stored, input of changes to a stored data record, input of a deletion request for a stored data record, input of a query on a data record; wherein the interface is arranged to communicate with the database and operate in dependence on said input.
PCT/IN2003/000297 2002-09-05 2003-09-05 Indexed data storage system, method and data structure WO2004023328A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
IN524/CAL/02 2002-09-05
IN524KO2002 2002-09-05

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
AU2003278589A AU2003278589A1 (en) 2002-09-05 2003-09-05 Indexed data storage system, method and data structure

Publications (1)

Publication Number Publication Date
WO2004023328A1 true WO2004023328A1 (en) 2004-03-18

Family

ID=31972127

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IN2003/000297 WO2004023328A1 (en) 2002-09-05 2003-09-05 Indexed data storage system, method and data structure

Country Status (1)

Country Link
WO (1) WO2004023328A1 (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100383786C (en) * 2004-11-25 2008-04-23 金诚国际信用管理有限公司 Expandable data storage method
EP1926030A3 (en) * 2006-11-23 2009-12-30 Samsung Electronics Co., Ltd. Apparatus and method for optimized index search
US7698258B2 (en) 2006-06-02 2010-04-13 Microsoft Corporation Searchable storage system
US20130006969A1 (en) * 2009-11-11 2013-01-03 International Business Machines Corporation Supporting set-level slice and dice in data warehouses
FR3043814A1 (en) * 2015-11-18 2017-05-19 Bull Sas Method for generating a data set defining a message for a constrained application
CN107038202A (en) * 2016-12-28 2017-08-11 阿里巴巴集团控股有限公司 Data processing method, device and equipment, computer-readable recording medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5600596A (en) * 1993-12-28 1997-02-04 Kabushiki Kaisha Toshiba Data access scheme with simplified fast data writing
US5799308A (en) * 1993-10-04 1998-08-25 Dixon; Robert Method and apparatus for data storage and retrieval
US5974412A (en) * 1997-09-24 1999-10-26 Sapient Health Network Intelligent query system for automatically indexing information in a database and automatically categorizing users
US5978791A (en) * 1995-04-11 1999-11-02 Kinetech, Inc. Data processing system using substantially unique identifiers to identify data items, whereby identical data items have the same identifiers
US6243720B1 (en) * 1998-07-08 2001-06-05 Nortel Networks Limited Address translation method and system having a forwarding table data structure
US6282544B1 (en) * 1999-05-24 2001-08-28 Computer Associates Think, Inc. Method and apparatus for populating multiple data marts in a single aggregation process

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5799308A (en) * 1993-10-04 1998-08-25 Dixon; Robert Method and apparatus for data storage and retrieval
US5600596A (en) * 1993-12-28 1997-02-04 Kabushiki Kaisha Toshiba Data access scheme with simplified fast data writing
US5978791A (en) * 1995-04-11 1999-11-02 Kinetech, Inc. Data processing system using substantially unique identifiers to identify data items, whereby identical data items have the same identifiers
US5974412A (en) * 1997-09-24 1999-10-26 Sapient Health Network Intelligent query system for automatically indexing information in a database and automatically categorizing users
US6243720B1 (en) * 1998-07-08 2001-06-05 Nortel Networks Limited Address translation method and system having a forwarding table data structure
US6282544B1 (en) * 1999-05-24 2001-08-28 Computer Associates Think, Inc. Method and apparatus for populating multiple data marts in a single aggregation process

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100383786C (en) * 2004-11-25 2008-04-23 金诚国际信用管理有限公司 Expandable data storage method
US7698258B2 (en) 2006-06-02 2010-04-13 Microsoft Corporation Searchable storage system
EP1926030A3 (en) * 2006-11-23 2009-12-30 Samsung Electronics Co., Ltd. Apparatus and method for optimized index search
US20130006969A1 (en) * 2009-11-11 2013-01-03 International Business Machines Corporation Supporting set-level slice and dice in data warehouses
US9646071B2 (en) * 2009-11-11 2017-05-09 International Business Machines Corporation Supporting set-level slice and dice in data warehouses
FR3043814A1 (en) * 2015-11-18 2017-05-19 Bull Sas Method for generating a data set defining a message for a constrained application
EP3171575A1 (en) * 2015-11-18 2017-05-24 Bull Sas Method of generation of one volume of data defining a message for an application with constraints
US10567317B2 (en) 2015-11-18 2020-02-18 Bull Sas Method for generating a dataset defining a message for a constrained application
CN107038202A (en) * 2016-12-28 2017-08-11 阿里巴巴集团控股有限公司 Data processing method, device and equipment, computer-readable recording medium
CN107038202B (en) * 2016-12-28 2020-05-05 阿里巴巴集团控股有限公司 Data processing method, device and equipment and readable medium

Similar Documents

Publication Publication Date Title
US9015150B2 (en) Displaying results of keyword search over enterprise data
US10019284B2 (en) Method for performing transactions on data and a transactional database
US9171065B2 (en) Mechanisms for searching enterprise data graphs
Dong et al. Indexing dataspaces
Hellerstein et al. On the analysis of indexing schemes
Ozcan et al. A framework for using materialized XPath views in XML query processing
Faye et al. A survey of RDF storage approaches
Schenkel et al. HOPI: An efficient connection index for complex XML document collections
Lian et al. An efficient and scalable algorithm for clustering XML documents by structure
US7366708B2 (en) Mechanism to efficiently index structured data that provides hierarchical access in a relational database system
Deppisch S-tree: a dynamic balanced signature index for office retrieval
Wang et al. Condensed cube: An effective approach to reducing data cube size
US6694323B2 (en) System and methodology for providing compact B-Tree
Gravano et al. Text joins in an RDBMS for web data integration
EP1101172B1 (en) Perspective transformations on relational database tables
US5257365A (en) Database system with multi-dimensional summary search tree nodes for reducing the necessity to access records
Feng et al. Trie-join: a trie-based method for efficient string similarity joins
US6701317B1 (en) Web page connectivity server construction
Bawa et al. LSH forest: self-tuning indexes for similarity search
CA2337079C (en) A search system and method for retrieval of data, and the use thereof in a search engine
US5781906A (en) System and method for construction of a data structure for indexing multidimensional objects
Tatarinov et al. Storing and querying ordered XML using a relational database system
Harman et al. Inverted Files.
Stockinger et al. Bitmap indices for data warehouses
US6598051B1 (en) Web page connectivity server

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ OM PH PL PT RO RU SC SD SE SG SK SL TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LU MC NL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase in:

Ref country code: JP

WWW Wipo information: withdrawn in national office

Country of ref document: JP