JP2009537906A - System and method for storing and retrieving data - Google Patents

Abstract

Systems and methods for storing and accessing data. Query language statements can reference multiple data sets, and multiple algebraic relationships between data sets can be assembled from the query language statements. Algebraic relationships can be used to define new datasets and optimize access to datasets. A memory of algebraic relationships can be accumulated over time. Alternative collections of algebraic functions can be generated and evaluated to determine the optimal collection of optimized algebraic relations to use in calculating and providing the required data set. Optimization can be performed using algebraic relationships rather than retrieving the underlying data set from storage. As a result, optimization can be performed at the processor speed to minimize the amount of time required to retrieve data from slower storage devices.

Description

Cross-reference This application is based on the following co-pending patent applications: US patent application 11 / 383,476 filed May 15, 2006, US patent application filed May 15, 2006 No. 11 / 383,477, U.S. Patent Application No. 11 / 383,478, filed May 15, 2006, U.S. Patent Application No. 11/383, filed May 15, 2006. , 479, US patent application Ser. No. 11 / 383,480 filed on May 15, 2006, and US Patent Application Ser. No. 11 / 383,482 filed on May 15, 2006. In connection with the specification, each of these US patent applications is hereby incorporated by reference.

BACKGROUND OF THE INVENTION The field of the invention relates to systems and methods for storing and accessing data, and more particularly to data storage, database queries, and data retrieval.

II. BACKGROUND Many databases and data storage systems have a predetermined schema that provides structure to the data as it is received. The schema may not capture information about the structure of the data as it is originally provided. In addition, the schema may be designed according to predefined relationships that are not optimized for the way the data is actually provided or queried. Schema-specific logical relationships can also lead to database structures that are not optimized for the way data is actually stored. Furthermore, logical relationships inherent in the schema and / or associated database structure can limit the types of logical relationships that can be specified in a data query. A single query may need to access the storage device multiple times, especially because of the wide spread between processing speed and storage access speed. A great deal of effort has been devoted to improving relational and other traditional database access methods, but such methods are inherently attached to data as a result of predefined relationships. Limited by the structure. Tight coupling of these relationships and structures in many databases can be done in a variety of different formats such as flat files, comma-separated value (CSV) files, and data defined using extensible markup language (XML). It also makes it difficult to efficiently capture, convert, and process data provided in (format).

SUMMARY OF THE INVENTION Aspects of the present invention provide systems and methods for storing and accessing data. Example embodiments include a data store that stores data sets, a data set information store that stores information about data sets, an algebraic relationship store that stores algebraic relationships between data sets, a data store using algebraic relationships An optimizer that optimizes storage and access of the data set from and a set processor that computes algebraic relationships and provides the data set. In example embodiments, modules may be provided by a combination of hardware, firmware, and / or software, and in some example embodiments, parallel processing and distributed storage may be utilized.

  One aspect of the present invention provides a method for assembling algebraic relationships between data sets from query language statements. Another aspect provides a method for providing a request data set. Query language statements can be presented to the system. For example, the query language statement may be in a structured query language (SQL) format that uses a relational data model, or may be in an XQuery format that uses a markup language format. A plurality of algebraic relationships can then be assembled from the query language statements and stored in the algebraic relationship store. In this way, each time a statement is presented to the system, algebraic relationships between data sets can be accumulated over time in the relationship store. In some example embodiments, query language statements do not require a requested data set, but may still be used to construct algebraic relationships that are useful in providing the requested data set. At least some of these algebraic relationships can be retrieved from the relationship store and used to provide a request data set.

  In a further aspect, algebraic relationships between data sets can accumulate in the relationship store over time each time a statement is presented to the system. Alternative collections of algebraic relationships (collections) can be generated and evaluated to determine the optimal collection of algebraic relationships for use in calculating and providing the requested data set. This optimization can be performed using algebraic relationships rather than retrieving the underlying data set from storage. As a result, optimization can be performed at processor speed, minimizing the amount of time required to retrieve data from a slow storage device.

  In another aspect, the query language statement requires a data set to be provided and the relationship store includes other algebraic relationships of the data set that are not assembled from the query language statement. In some examples, both algebraic relationships constructed from query language statements and other algebraic relationships in the relationship store may be used to provide the request data set. In a further aspect, the optimizer can be used to generate multiple collections of algebraic relationships that define results equal to the requested data set, and an optimization criterion is applied and used to provide the requested data set. , One of a set of algebraic relationships can be selected. In some example embodiments, the optimization criterion is an estimate of the amount of data that needs to be transferred from the storage device and / or the amount of time required to transfer the data set from the storage device to calculate a collection of algebraic relationships. Can be based. In another example, the optimization criteria may identify equivalent data sets that contain the same logical data in different physical formats or at different locations in the data store.

  Another aspect provides a method for providing a request data set that can assemble at least two alternative algebraic relationships that each define an equal result to the request data set. The data set can include the same logical data stored in different physical formats and / or at different locations within the data store. For example, the data set can be a comma separated value (CSV) format, a binary string encoding (BSTR) format, a fixed offset (FIXED) format, a type encoded data (TED) format, and / or an XML or other markup language format. Can be stored in a storage medium. Type-encoded data (TED) is a file format that includes data and associated values that point to the format of such data. These are merely examples, and other physical formats may be used in other embodiments. Data sets may be stored in different locations in the data store, such as different disk drives in a distributed storage system, and accessible via different data channels with different data rates and / or different available bandwidth It is good also as being. Selecting one of the algebraic relationships to be used to provide the requested data set based at least in part on the physical format and / or location of the data set referenced within the algebraic relationship. Can do. In other examples, the algebraic relationship may be selected based at least in part on the speed and available bandwidth of the channels used to search for the datasets referenced within the algebraic relationship.

  Another aspect provides a method for providing a request data set using functions that operate on operands of different physical formats. The data set can be in comma separated value (CSV) format, binary string encoding (BSTR) format, fixed offset (FIXED) format, type encoded data (TED) format, and / or XML or other markup language format, etc. Can be stored in multiple physical formats. A function is defined that uses a data set as an operand. Logically equivalent functions can be defined for different combinations of physical formats that can be used for operands. To provide the request data set, an algebraic relationship can be constructed that defines a result equal to the request data set. Algebraic relationships can refer to data sets in storage. In order to calculate the requested data set from the algebraic relationship, the reference data set is retrieved from storage and a function is applied to the data set to perform the operations specified in the algebraic relationship. The function used to calculate the algebraic relationship can be selected to correspond to the physical format in which the data set is retrieved. In this way, the function best suited for the physical format from which the data set was retrieved can be used without the need for conversion to a separate format.

  In a further aspect, a plurality of algebraic relationships are assembled that define a result equal to the requested data set. Some of the algebraic relationships can refer to the same data but with different physical formats. Optimization criteria can be applied to the algebraic relationships that take into account the physical format of the data sets, the functions available for operation on the data sets of those formats, and / or any format conversion that may be required for the calculation. Algebraic relationships are selected based on optimization criteria and can be used to provide the requested data set. The format-specific function is then used to calculate the selected algebraic relationship. At least some of the functions are selected based on the physical format of the data set referenced within the algebraic relationship.

  In another aspect, algebraic relationships can be used to define new data sets. In one example embodiment, a data set information store may be provided that stores information about the data set. A new data set can be created by associating a data set identifier with the data set and storing the data set identifier in the data information store. In some examples, the new data set may be an explicit data set that is presented to the system as part of the query language statement.

  In another aspect, the query language statement may specify one or more of a data set not yet stored in the data store upon receipt of the query language statement. In some embodiments, a data set can be defined by an algebraic relationship without realizing the data set in storage.

  In another aspect, temporary information indicating when the data set is created is stored in the data set information store. In a further aspect, the data set information store can be temporarily redefined by removing the data set associated with the temporary information from the data set information store prior to the specified time. If an unimplemented data set refers to data with temporary information before a specified time, the data set is realized and stored in the data store before the referenced data set is removed Can do.

  Another aspect provides a method for providing a request data set using a mapping between schemas. Mappings can be provided between multiple schemas based on different data models. Statements can be presented to the system based on different schemas and data models. For example, the statement may be presented to the system as a query language in a structured query language (SQL) format based on a relational data model and / or an XQuery format based on an extensible markup language (XML) data model. These statements and data models are merely examples, and in other examples, other statements and data models may be supported. Algebraic relationships between data sets can be assembled from statements presented to the system based on different schemas and data models. If a data set is requested based on a particular schema and data model, the mapping can use algebraic relationships based on other schemas and data models to provide the requested data.

  In a further aspect, a plurality of algebraic relationships that define a result equal to the requested data set can be assembled. An optimization criterion can be used to select one of the algebraic relationships for calculating the required data set. Algebraic relationships can be assembled from statements based on different schemas and data models. Mappings can be provided between schemas based on different data models. As a result, optimization can be performed over a wider set of algebraic relationship candidates to provide a requested data set. Algebraic relationships can be considered even when assembled from statements based on different schemas using different data models. For example, algebraic relationships can be assembled from query statements presented to the system in both a structured query language (SQL) format based on a relational data model and an XQuery format based on an extensible markup language (XML) model. These algebraic relationships can then be used for optimization in response to subsequent query statements being presented to the system. For example, an algebraic relationship assembled from SQL statements can be used in response to an XQuery statement. Similarly, algebraic relationships constructed from XQuery can be used in response to SQL statements. These are merely examples, and other types of statements and data models may be used in other examples.

  Another aspect provides a method for storing a data set using virtualization. Data sets can be removed from the data store and defined by algebraic relationships in the relationship store. The data set information can include information identifying whether each data set is implemented in a data store. Criteria can be established for determining when a data set should be virtualized. For example, the criteria can be based on the size of the data set, the number of times it was referenced, and / or the frequency at which the data set was accessed in the data store. Data sets that are implemented in the data store and that meet this criteria can be considered from removal from the data store. In an example embodiment, an algebraic relationship that defines a data set based on other data sets realized in the data store (direct or other based directly or indirectly on the realized data set) These data sets can be removed if they are in the relationship store (whether indirect by referring to their algebraic relationships). After a data set is removed, information about that data set in the data set information store can be changed to indicate that the identified data set is not implemented in the data store.

  In a further aspect, the data set for optimization can be selected by dividing the data set into subsets and then virtualizing the data set by removing it from the data storage device. . For example, a data set that is a subset of the selected data set can be added to the data store. In some examples, the subset may be a portion of the selected data set that has equal density, or may be defined based on a scalar value range of data items in the selected data set. These are merely examples, and other subsets may be defined in other examples. Based on the collection of subsets added to the data store, an algebraic relationship defining the selected data set can be assembled. The selected data set can then be removed from the data store and the information in the data set information store can be changed to indicate that the selected data set is not implemented in the data set.

  In a further aspect, an algebraic relationship that references a virtual data set can be used to retrieve the requested data set from the system. For example, the selected data set can be removed from the data store and replaced with an algebraic relationship that defines the selected data set. Algebraic relationships can be provided in the relation store for use in providing other requested data sets, even if the selected data set is no longer implemented in the data store. . For example, multiple collections of algebraic relationships that define the request data set can be assembled. Some of these algebraic relationships can be assembled by performing substitutions of references to the (virtual) selected dataset using the algebraic relationship that defines the selected dataset. it can. For example, an expression that references a selected data set can be replaced with an expression that references one or more subsets implemented in the data store. The optimization criteria can then be applied to select one of the algebraic collections that compute the required data set.

  In another aspect, a computer system is provided having one or more processors programmed to perform one or more of the above aspects of the invention. The computer system can include volatile storage and / or non-volatile storage that provides a data set store. In another aspect, one or more hardware accelerators or other circuits are configured to perform one or more of the above aspects of the invention. In another aspect, a computer readable medium having executable instructions for performing one or more of the above aspects of the invention is provided.

  It will be appreciated that each of the above aspects of the invention may be used alone or in combination with other aspects of the invention as described above or in the following description.

INCORPORATION BY REFERENCE All publications and patent applications mentioned in this specification are herein incorporated by reference as if each individual publication or patent application was specifically indicated to be specifically incorporated herein by reference. To the same extent in the book.

  The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings of which:

FIG. 2 is a block diagram illustrating a first example structure of a computer system that can be used in connection with example embodiments of the present invention. FIG. 2 is a block diagram illustrating a computer network that can be used in connection with example embodiments of the present invention. FIG. 6 is a block diagram illustrating a second example structure of a computer system that can be used in connection with example embodiments of the invention. It is a block diagram which shows the logical structure of one example embodiment of this invention. It is a block diagram which shows the information memorize | stored in the set manager module of the example of 1 embodiment of this invention. 4 is a flowchart of a method for submitting a data set according to an example embodiment of the present invention. 4 is a flowchart of a method for submitting a statement, according to an example embodiment of the invention. FIG. 7 shows an example of an XSN tree of statements and the method of FIG. 2 is a flowchart of a method for implementing a data set, according to an example embodiment of the present invention. 4 is a flowchart of a method for algebraic and computational optimization, according to an example embodiment of the invention. 6 is a flow chart of a method for algebraic and computational optimization according to an alternative example embodiment of the present invention. Fig. 4 illustrates a method for comprehensive optimization according to an exemplary embodiment of the present invention. Fig. 4 illustrates a method for comprehensive optimization according to an exemplary embodiment of the present invention. Fig. 4 illustrates a method for comprehensive optimization according to an exemplary embodiment of the present invention. Fig. 4 illustrates a method for comprehensive optimization according to an exemplary embodiment of the present invention. Fig. 4 illustrates a method for comprehensive optimization according to an exemplary embodiment of the present invention. Fig. 4 illustrates a method for comprehensive optimization according to an exemplary embodiment of the present invention. An example field of an OptoNode structure is shown. FIG. 4 is a block diagram of an example OptoNode structure according to an example embodiment of the present invention. 4 is a flowchart of a method for calculating a data set from an algebraic relationship, according to an example embodiment of the invention. FIG. 4 is a block diagram of an example XSN tree according to an example embodiment of the present invention. FIG. 4 is a block diagram of an example XSN tree according to an example embodiment of the present invention. FIG. 6 is a block diagram illustrating an example implementation of a buffer chain that can be used in an example embodiment of a storage manager. FIG. 6 is a block diagram illustrating an example implementation of a buffer chain that can be used in an example embodiment of a storage manager. FIG. 6 is a block diagram illustrating an example implementation of a buffer chain that can be used in an example embodiment of a storage manager. FIG. 6 is a block diagram illustrating an example implementation of a buffer chain that can be used in an example embodiment of a storage manager. FIG. 6 is a block diagram of conversion from relational data to XML according to an example embodiment. FIG. 6 is a block diagram of conversion from related data to a directed graph according to an example embodiment.

DETAILED DESCRIPTION While the present invention is open to various modifications and alternative constructions, the embodiments shown in the drawings will now be described in detail. However, it should be understood that the invention is not intended to be limited to the particular forms disclosed. On the contrary, the invention is intended to cover all modifications, equivalents, and alternative constructions falling within the spirit and scope of the invention as expressed in the appended claims.

  The example embodiments of the present invention provide systems and methods for storing and processing data using extended set processing and algebraic optimization. In one example, a universal data model based on extended set theory can be used to capture scalar information, structural information, and temporal information from data provided in a wide variety of heterogeneous formats. For example, fixed format, comma separated value (CSV) format, extensible markup language (XML) and other format data can be captured and processed without loss of information. These encodings are called physical formats. The same logical data can be stored in any number of different physical formats. Embodiments can be seamlessly converted between these formats while retaining the same logical data.

  By using a rigorous mathematical data model, example embodiments maintain algebraic integrity of data and data interrelationships, provide temporal invariance, and allow adaptive data reconstruction Can be.

  Algebraic completeness allows the manipulation of information that an algebraic relationship models to be replaced by the operation of that algebraic relationship. For example, queries can be processed by evaluating algebraic expressions at processor speed without having to retrieve and examine various data sets from storage at a much slower speed.

  Time invariance can be provided by maintaining a constant value, structure, and location of information until the information is discarded from the system. New data defined in part as algebraic representations with references to data already identified in the system by standard database operations such as "insert", "update", and "delete" functions Is created. Since such operations do not change the original data, the example embodiments provide the ability to examine the information contained in the system at any time in the form as it exists in the recorded history.

  Adaptive data reconstruction combined with algebraic integrity allows the logical and physical structure of information to be changed while maintaining a strict mathematical mapping between logical and physical structures To do. In example embodiments, adaptive data reconstruction can be used to speed up query processing and minimize data transfer between non-volatile and volatile storage.

Example embodiments use these features to access dynamically changing data regardless of whether it is provided in XML format, in a related format, or in other data formats. , Integration, and processing can provide dramatic efficiencies. In particular, example embodiments can provide:
Independence from information structures that allow all types of corporate information to be mathematically modeled and processed with equal equipment and without extensive programming. Data pre-construction and database extraction. Conversion and load operations, and elimination of most database index structures and associated storage capacity-Eliminate redundant operations and reduce data transfer across non-volatile / volatile storage boundary performance barriers with adaptive rebuild work data sets High-speed query processing via adaptive optimization ・ Scalable, highly asynchronous and parallel internal operation that fully utilizes massively parallel computing and storage systems ・ Performance by minimizing stateless entity recording and hence reusable resources Improvement in fault tolerance and fault tolerance The ability to query the database as it is was

  The mathematical data model allows the example embodiments to be used in a wide range of computer structures and systems, and is useful for massively parallel computing and storage systems. Several examples of computer structures and systems that can be used in connection with example embodiments will now be described.

  FIG. 1 is a block diagram that illustrates a first example structure of a computer system 100 that can be used in connection with example embodiments of the invention. As shown in FIG. 1, an example computer system may include a processor 102 that processes instructions, such as an Intel Xeon ™ processor, an AMD Opteron ™ processor, or other processor. Multiple execution threads can be used for parallel processing. Some embodiments use processors with multiple processors or multiple cores regardless of whether they are in a single computer system, in a cluster, or distributed over the network through the system. May be.

  As shown in FIG. 1, a high speed cache 104 may be connected to or incorporated within the processor 102 to provide a high speed memory of recently used or frequently used instructions or data by the processor 102. . The processor 102 is connected to the north bridge 106 by a processor bus 108. The north bridge 106 is connected to a random access memory (RAM) 110 by a memory bus 112 and manages access to the RAM 110 by the processor 102. The north bridge 106 is also connected to the south bridge 114 by a chipset bus 116. The south bridge 114 is connected to the peripheral bus 118. The peripheral bus may be, for example, PCI, PCI-X, PCI Express, or other peripheral bus. The North Bridge and South Bridge are often referred to as processor chipsets and manage data transfer between the processor, RAM, and peripheral components on the peripheral bus 118. Depending on the alternative structure, instead of using a separate Northbridge chip, Northbridge functionality may be incorporated into the processor.

  In some embodiments, the system 100 can include an accelerator card 122 attached to the peripheral bus 118. The accelerator can include a field programmable gate array (FPGA) or other hardware that accelerates certain processes. For example, accelerators can be used to evaluate algebraic expressions used for adaptive data reconstruction or extended set processing.

  Software and data are stored in the external storage device 124 and can be loaded into the RAM 110 and / or the cache 104 for use by the processor. System 100 includes an operating system for managing system resources, such as Linux or other operating systems, and application software running on the operating system for managing data storage and optimization according to example embodiments of the present invention. including.

  In this example, system 100 is connected to a peripheral bus and provides a network interface card (NIC) 120 that provides a network interface to external storage devices such as network attached storage (NAS) and other computer systems that can be used for distributed parallel processing. And 121 are also included.

  FIG. 2 is a block diagram illustrating a network 200 having multiple computer systems 202a, 202b, and 202c and network attached storage (NAS) 204a, 204b, and 204c. In the example embodiment, computer systems 202a, 202b, and 202c manage data storage to data stored in network attached storage (NAS) 204a, 204b, 204c and data to NAS 204a, 204b, and 204c. Access can be optimized. The mathematical model can be used for data and can be used for evaluation using distributed parallel processing across computer systems 202a, 202b, and 202c. Computer systems 202a, 202b, and 202c may also provide parallel processing for adaptive data reconstruction of data stored in network attached storage (NAS) 204a, 204b, and 204c. This is merely an example, and a wide range of other computer structures and systems can be used. For example, a blade server may be used to provide parallel processing. Processor blades can be connected via the backplane to provide parallel processing. The storage device may also be connected to the backplane or may be network attached storage (NAS) via a separate network interface.

  In example embodiments, the processor may maintain a separate memory space and send data through a network interface, backplane, or other connector for parallel processing by other processors. In other embodiments, some or all of the processors may use a shared virtual address memory space.

  FIG. 3 is a block diagram of a multiprocessor computer system 300 that uses a shared virtual address memory space according to an example embodiment. The system includes a plurality of processors 302 a-302 f that are accessible to the shared memory subsystem 304. The system incorporates a plurality of programmable hardware memory algorithm processors (MAPs) 306 a-306 f in the memory subsystem 304. Each MAP 306a-306f may include memory 308a-308f and one or more field programmable gate arrays (FPGAs) 310a-310f. The MAP provides configurable functional units and can provide FPGAs 310a-310f with specific algorithms or portions of algorithms for processing in close cooperation with each processor. For example, in an example embodiment, MAP can be used to evaluate an algebraic representation for a data model and perform adaptive data reconstruction. In this example, each MAP is globally accessible from all processors for these purposes. In one configuration, each MAP uses direct memory access (DAM) to access the associated memory 308a-308f, thereby allowing tasks independent of each microprocessor 302a-302f and asynchronously with each processor 302a-302f. Can be performed. In this configuration, the MAP can feed the results directly to another MAP for algorithmic pipelining and parallel execution.

  The above computer structures and systems are merely examples, and general purpose processors, coprocessors, FPGAs, and other programmable logic elements, system on a chip (SOC), application specific integrated circuits (ASICs), other processing elements and logic elements A wide range of other computer structures and systems can be used in connection with the example embodiments, including systems that use any combination of the above. All or part of a data management and optimization system may be implemented in software or hardware, random access memory, hard drives, flash memory, tape drives, disk arrays, network attached storage (NAS), and other local or distributed It will be appreciated that any of a variety of data storage media may be used in connection with example embodiments, including data storage devices and systems.

  In an example embodiment, the data management and optimization system may be implemented using software modules that execute on any of the above or other computer structures and systems. In other embodiments, the functions of the system include firmware, programmable logic elements such as field programmable gate arrays (FPGAs) as referenced in FIG. 3, system on chip (SOC), application specific integrated circuits (ASIC), Or it can be partially or fully implemented in other processing and logic elements. For example, the set processor and optimizer can be implemented with a hardware accelerator through the use of a hardware accelerator card such as the accelerator card 122 shown in FIG.

  FIG. 4A is a block diagram illustrating an example of the logical structure of the software module 400. The software is component-based and is organized into modules encapsulating specific functions as shown in FIG. 4A. This is only an example, and other software structures can be used as well.

  In this example embodiment, data stored internally in one or more various physical formats can be presented to the system. The system can create a mathematical representation of the data based on the extended set theory and assign a globally unique identifier (GUID) to the mathematical representation for unique identification within the system. In this example embodiment, the data is represented internally in the form of an algebraic representation applied to one or more data sets, where the data may be defined at the time of creation of the algebraic representation, Or it may not be defined. A data set includes a set of data elements called members of the data set. In one example embodiment, an element can be a data value or an algebraic representation formed from a combination of operators, values, and / or other data sets. In this example, the data set is an algebraic operand. Algebraic relationships that define the relationships between the various data sets are stored and managed by the set manager 402 software module. In this embodiment, algebraic integrity is preserved because all data sets are related through specific algebraic relationships. A particular data set may or may not be stored in the system. Some data sets are defined solely by algebraic relationships with other data sets and need to be calculated to retrieve the data set from the system. Some data sets are even defined by algebraic relationships that reference data sets that have not yet been provided to the system, and cannot even be calculated until sometime in the future.

  In one example embodiment, GUIDs referenced in algebraic relationships and algebraic representations are not changed after they are created and stored in set manager 402. This provides a time invariance that allows data management without worrying about locking other concurrent management devices and without the associated overhead. The algebraic relationship and GUID of the corresponding data set are attached only in the set manager 402 and are not removed or changed as a result of the new operation. This creates a world in which operands and algebraic relationships continue to expand, and any state of information in the record history can be reproduced. In this embodiment, a separate external identifier can be used to point to the same logical data even if the logical data changes over time, but a unique GUID can be used for a raw data set that exists at a particular time. Used to refer to each instance. The set manager 402 can associate a GUID with an external identifier and a time stamp indicating the time when the GUID was added to the system. The set manager 402 can also associate specific information about a specific data set with a GUID. This information can be stored in the set manager 402 in a list, table, or other data structure (referred to in this example embodiment as a set universe). Algebraic relationships between data sets can also be stored in a list, table, or other data structure in the set manager 402 (referred to in this example embodiment as an algebraic cache).

  In some embodiments, the set manager 402 can purge (erase) unnecessary or redundant information and can be redefined in time to limit the time range of the recording history. For example, unnecessary or redundant information can be automatically purged, and temporal information can be periodically collapsed based on user settings or commands. This can be accomplished by removing from the set manager 402 all GUIDs that have a time stamp prior to the specified time. All algebraic relationships that reference these GUIDs are also removed from the set manager 402. If other data sets are defined by algebraic relationships that reference these GUIDs, these data sets may need to be calculated and stored before the algebraic relationships are removed from the set manager 402.

  In one example embodiment, the data set can be purged from storage, and the system can later recreate the data set if necessary, relying on algebraic relationships. This process is called virtualization. When an actual data set is purged, the storage capacity associated with such a data set can be released, but the system has the ability to identify the data set based on the algebraic relationship stored in the system. Hold. In one example embodiment, data sets that are large or referenced less than a certain number of times threshold can be automatically virtualized. Other embodiments may include virtualization of datasets that have been used little or no recently, virtualization of datasets to free up faster memory or storage capacity, or dataset virtualization for enhanced security (virtualization). After virtualization, other virtualization criteria can be used, including because it is more difficult to access the data set without accessing algebraic relationships. These settings can be user configurable or system configurable. For example, if set manager 402 includes an algebraic relationship in which data set A and A are equal to the intersection of data sets B and C, the system purges data set A from set manager 402 and, if necessary, Data sets A can be configured to rely on data sets B and C and algebraic relationships. In another example embodiment, if two or more data sets are equal to each other, all can be deleted from the set manager 402, leaving one of the data sets. This can occur when the sets are logically equal but in different physical formats. In such cases, all but one of the data sets can be removed to save physical storage space.

  If the values of the data set need to be calculated or provided by the system, the optimizer 418 can retrieve the algebraic relationship that defines the data set from the set manager 402. The optimizer 418 may also use the algebraic relationship from the set manager 402 to generate an equivalent additional algebraic relationship that defines the data set. In this case, the most efficient algebraic relationship can then be selected to calculate the data set.

  The set processor 404 software module provides an engine that calculates the values of a data set represented in an algebraic representation and performs the arithmetic and logical operations and functions necessary to evaluate algebraic relationships. The set processor 404 also allows adaptive data reconstruction. As the data set is manipulated by the operations and functions of the set processor 404, it is physically and logically processed to speed up subsequent operations and functions. The operations and functions of the set processor 404 are implemented as software routines in one example embodiment. However, such operations and functions may be implemented in firmware, programmable logic elements such as field programmable gate arrays (FPGAs) as referenced in FIG. 3, system on chip (SOC), application specific integrated circuits (ASIC), or It can be partially or fully implemented on other hardware or a combination thereof.

  The software module shown in FIG. 4A will be described in further detail. As shown in FIG. 4A, the software includes a set manager 402 and a set processor 404 and an SQL connector 406, an SQL translator 408, an XSN connector 410, an XML connector 412, an XML translator 414, an XSN interface 416, an optimizer 418, a storage manager 420, an executive 422 and an administrator interface 424.

  In the example embodiment of FIG. 4A, queries and other statements for the data set are provided through one of three connectors: SQL connector 406, XSN connector 410, or XML connector 412. Each connector receives and provides statements in a specific format. In one example, the SQL connector 406 provides a standard SQL92 compliant ODBC connector for use with user applications and an ODBC compliant third party relational database system, and the XML connector 412 includes a user application, a compatible third party XML system, and A standard web service W3C XQuery compliant connector for using other instances of the software 400 on the same or other systems is provided. SQL and XQuery are example formats for providing query language statements to the system, but other formats may be used. Query language statements provided in these formats are converted by SQL translator 408 and XML translator 414 into extended set notation (XSN) used by the system. XSN connector 410 provides a connector for receiving statements directly in XSN format. An example of extended set notation is described below at the end of this specification. This example of extended set notation includes a syntax that allows statements regarding the extended data set to be presented to the system. This example of extended set notation is merely an example, and other notations may be used in other embodiments. Other embodiments may use different types and formats of data sets and algebraic relationships to capture information from statements provided to the system.

  The XSN interface 416 provides a single entry point for all statements from the connector. Statements are provided in XSN format from SQL translator 408, XML translator 414, or XSN connector 410. Statements are provided using a text-based description of extended set notation. The XSN interface 416 provides a parser that converts the text description into an internal representation used by the system. In one example, the internal representation uses an XSN tree data structure as described further below. When the XSN statement is parsed, the XSN interface 416 can call the set manager 402 to assign a GUID to the data set referenced in the statement. The overall algebraic relationship that represents the XSN statement can also be parsed into elements that are each an algebraic relationship. In one example embodiment, these elements may be algebraic relationships having a representation composed of a single operation that references from one to three data sets. Each algebraic relationship can be stored in an algebraic cache within the set manager 402. For each new algebraic representation, a GUID representing the data set defined by the algebraic representation can be added to the set universe. This causes the XSN interface 416 to assemble multiple algebraic relationships that reference the data set specified in the statement presented to the system as well as a new data set that can be created when parsing the statement. In this way, XSN interface 416 and set manager 402 capture information from statements presented to the system. And if the system needs to compute data sets, these data sets and algebraic relationships can be used for algebraic optimization.

  The set manager 402 provides a data set information store that stores information about data sets known to the system, in this example called a set universe. The set manager 402 also provides a relationship store that stores relationships between data sets known to the system, in this example called an algebraic cache. FIG. 4B illustrates information held in the set universe 450 and the algebraic cache 452 according to an example embodiment. Other embodiments may use different data set information stores to store information about data sets, or use different related stores to store information about algebraic relationships known to the system. Good.

  As shown in FIG. 4B, the set universe 450 can maintain a GUID list of data sets known to the system. Each GUID is a unique identifier for a data set in the system. The set universe 450 can also associate each GUID with information about a particular data set. This information can be, for example, an external identifier used to reference a data set in a statement presented through the connector (which may or may not be unique for a particular data set), It may include a set type having a date / time indicator that indicates when the data set is known to the system, a format field that indicates the format of the data set, and a flag that indicates the type of the data set. The format field can indicate a logical-to-physical conversion model of the data set in the system. For example, the same logical data can be stored in different physical formats on a storage medium in the system. As used herein, a physical format refers to a format for encoding logical data when data is stored on a storage medium, and is a specific type of physical storage medium used (e.g., disk , RAM, flash memory, etc.). The format field indicates how the logical data is mapped to the physical format on the storage medium. For example, the data set may be stored on a storage medium in comma separated value (CSV) format, binary string encoding (BSTR) format, fixed offset (FIXED) format, type encoded data (TED) format, and / or markup language format. Can be remembered. Type encoded data (TED) is a file format that includes data and associated values that indicate the format of such data. These are merely examples, and other physical formats may be used in other embodiments. The set universe stores information about the data set, but the underlying data is in this example embodiment anywhere else, such as storage device 124 of FIG. 1, network attached storage 204a, 204b, and 204c of FIG. The data can be stored in the memories 308a to 308f in FIG. 3 or other storage devices. Some data sets do not exist in physical storage but are calculated from algebraic relationships known to the system. In some cases, a data set is defined by an algebraic relationship that references a data set that has not yet been provided to the system, and may not be calculated until sometime in the future. The set type is called realization, whether the data set is available in storage or whether the data set is defined by an algebraic relationship with other data sets, called virtualization. Can show. Depending on the embodiment, other types may be supported, such as a transition type that indicates a data set that is being created or removed from the system. These are merely examples, and in other embodiments, other information about the dataset may be stored in the dataset information store.

  As shown in FIG. 4B, the algebraic cache 452 can maintain a list of algebraic relationships that associate one data set with another data set. In the example shown in FIG. 4B, the algebraic relationship can specify that the data set is equal to an operation or function performed on one to three other data sets (in FIG. 4B, “guid OP GUID shown as “guid guide”). Examples of operations and functions include projection functions, inverse functions, density functions, binding functions, and constraint functions. Additional examples are described at the end of this specification as part of an extended set notation example. An algebraic relationship can also specify that a data set has a specific relationship to another data set (shown as “guid REL guide” in FIG. 4B). Examples of relational operators include equal, subset, and disjoint and their logical negation, as further described at the end of this specification as part of an extended set notation example. These are only examples, and in other embodiments, other operations, functions, and relational operators may be used, including functions that operate on more than three data sets.

  Other modules can access the set manager 402 to add new GUIDs for the data set, and the set manager 402 can use the known between data sets for use in optimizing and evaluating other algebraic relationships. You can search for relationships. For example, the system may receive a query language statement that specifies a data set that is the intersection of a first data set A and a second data set B. The system can determine and return the resulting data set C. In this example, the module processing this request calls the set manager 402 to algebraize the known relationships of datasets A and B that may be useful in evaluating the intersection of dataset A and dataset B. Can be obtained from the cache. It may be possible to identify results using known relationships without actually retrieving the underlying data for data sets A and B from the storage system. The set manager 402 can also create a new GUID for data set C and store the relationship in the algebraic cache (ie, data set C is equal to the intersection of data set A and data set B). Once this relationship is added to the algebraic cache, it can be used for future optimizations and calculations. All data sets and algebraic relationships can be maintained in the set manager 402 and can provide temporal invariance. Existing data sets and algebraic relationships are not deleted or changed when the system receives a new statement. If a new statement is received without modification, a new data set and algebraic relationship is assembled and added to the set manager 402. For example, if data is required to be removed from the data set, a new GUID can be added to the set universe and defined in the algebraic cache as the difference between the original data set and the data to be removed.

  The optimizer 418 receives the algebraic representation from the XSN interface 416 and optimizes it for computation. When a dataset needs to be computed (eg, to be implemented in a storage system or returned in response to a request from a user), the optimizer 418 algebraizes the algebraic relationship that defines the dataset. Search from the cache. The optimizer 418 can then generate multiple collections of other algebraic relationships that define equivalent data sets. Algebraic replacement can be done using other algebraic relationships from the algebraic cache, and algebraic operations can be used to generate algebraically equivalent relationships. In one example embodiment, all possible algebraic collections are generated from information in an algebraic cache that defines a data set equal to the specified data set.

  The optimizer 418 can then determine an estimated cost for calculating the data set from each collection of algebraic relationships. Cost can be determined by applying a cost function to each set of algebraic relationships, and the lowest cost set of algebraic relationships can be used to calculate a specified data set. In one example embodiment, the cost function retrieves the data set needed to calculate each collection of algebraic relationships from the storage device and determines the estimated time required to store the result in the storage device. If the same data set is referenced more than once in a collection of algebraic relationships, the cost of searching the data set will be available in memory after it is first searched, so the assignment is only once It's okay. In this example, the collection of algebraic relationships that requires the least data transfer time is selected to calculate the requested data set.

  The optimizer 418 can generate different collections of algebraic relationships that reference the same logical data stored at different physical locations and / or in different physical formats via different data channels. Although the data is logically the same, different data sets with different GUIDs can be used to distinguish data that is logically the same in different locations or formats. Different collections of algebraic relationships may have different costs because the time taken to retrieve data sets from different locations and / or different formats may differ. For example, logically the same data may be available in different formats via the same data channel. Examples of formats include comma separated value (CSV) format, binary string encoding (BSTR) format, fixed offset (FIXED) format, type encoded data (TED) format, and markup language format. Other formats may be used. If the data channels are the same, the physical format having the smallest size (and thus the smallest number of bytes to transfer from the storage device) can be selected. For example, the comma separated value (CSV) format is often smaller than the fixed offset (FIXED) format. However, if a larger format is available via a faster data channel, a larger format can be selected over a smaller format. In particular, a larger format that is available in a high-speed non-volatile memory such as DRAM is generally selected than a smaller format that is available in a slower non-volatile storage device such as a disk drive or flash memory.

  In this way, the optimizer 418 optimizes algebraic relationships by taking advantage of the fast processor speed without accessing the underlying data of the data set from the data storage device. The processor speed for executing instructions is often higher than the data access speed from the storage device. By optimizing the algebraic relationship before calculation, unnecessary data access from the storage device can be avoided. The optimizer 418 can consider a number of equivalent algebraic relations and optimization techniques at processor speed, and can take into account the efficiency of data access required to actually evaluate the representation. For example, the system may receive a query that requests data that is the intersection of data sets A, B, and D. The optimizer 418 can obtain known relationships for these data sets from the set manager 402 and optimize the representation prior to evaluation. For example, an existing relationship can be obtained from the algebraic cache indicating that data set C is equal to the intersection of data sets A and B. Rather than calculating the intersection of datasets A, B, and D, optimizer 418 determines that it is more efficient to calculate the intersection of datasets C and D to obtain equivalent results. Can do. In making this determination, the optimizer 418 considers that the data set C is smaller than the data sets A and B and can be obtained from the storage device faster, or the data set C is used for recent operations, It can be considered that it is already loaded into a faster memory or cache.

  The optimizer 418 may also continually enrich the information in the set manager 402 through the presentation of additional relationships and sets discovered through set and algebraic cache analysis. This process is called comprehensive optimization. For example, the optimizer 418 utilizes unused processor cycles to analyze relationships and data sets and adds new relationships that are expected to be useful in optimizing the evaluation of future requirements to the algebraic cache. A set can be added to the set universe. Once the relationship is entered into the algebraic cache, the optimizer 418 can utilize the computations while processing subsequent statements, even if the computations being performed by the set processor 404 are not complete. . There are many algorithms for comprehensive optimization that can be useful. These algorithms can be based on the discovery of calculations that are repeated over a limited number of sets that indicate usage patterns or usage trends during recent time periods.

  The set processor 404 actually calculates the selected collection of algebraic relationships after optimization. The set processor 404 provides the arithmetic and logical processing necessary to implement a data set specified by an algebraic extended set representation. In one example embodiment, the set processor 404 provides a collection of functions that can be used to compute operations and functions referenced in algebraic relationships. The collection of functions may include functions configured to receive a data set in a particular physical format. In this example, the set processor 404 can provide multiple different algebraically equivalent functions that operate on the data set and provide the results in different physical formats. The function selected to calculate the algebraic relationship (which can be selected during optimization by the optimizer 418) corresponds to the format of the data set referenced within the algebraic relationship. In the example embodiment, the set processor 404 is capable of parallel processing of a plurality of simultaneous operations, and it is necessary to pipeline data input / output via the storage manager 420 to cross the non-volatile / volatile storage device boundary. A certain amount of data can be minimized. In particular, algebraic relationships from selected collections can be assigned to various processing resources for parallel processing. These processing resources include multi-thread execution by the processor 102 and accelerator 122 shown in FIG. 1, the distributed computer system shown in FIG. 2, the plurality of processors 302 and MAP 306 shown in FIG. 3, or any of the above. Can be included. These are merely examples, and other processing resources may be used in other embodiments.

  The executive 422 performs overall scheduling of execution, management and allocation of computing resources, and appropriate startup and shutdown.

  The administrator interface 424 provides an interface for managing the system. In example embodiments, this may include an interface for importing or exporting a data set. Although datasets can be added through the connector, the administrator interface 424 provides an alternative mechanism for importing multiple datasets or very large datasets. Data sets can be imported by specifying the location of the data set through the interface. The set manager 402 can then assign a GUID to the data set. However, it is not necessary to access the underlying data until a request is received that requires access to the data. This allows for very fast system initialization without having to import and reformat the data into a specific structure. Rather, when the data is actually queried, the relationship between the data sets is defined and added to the algebra cache in the set manager 402. As a result, optimization is based on the actual way in which the data is used (rather than a predefined relationship built into a table set or other predefined data structure).

  The example embodiments can be used to manage large amounts of data. For example, a data store may contain more than 1 terabyte, 100 terabytes, or 1 petabyte or more of data. The data store can be provided by a storage array or a distributed storage system having a large storage capacity. The data set information storage store can define a large number of data sets. In some cases, there may be more than one million defined data sets in the data information storage device. In one example embodiment, the software is a 264 dataset scale, but other embodiments may manage smaller or larger dataset universes. Many of these data sets may be virtual and others can be implemented in a data store. Occasionally scan entries in the dataset information store to determine if additional datasets should be virtualized or remove datasets and redefine the datasets captured in the dataset information store in time It can be determined whether or not. The relationship store can also contain a number of algebraic relationships between data sets. In some cases, more than 1 million and over 10 million algebraic relationships may be included in the relationship store. In some cases, the number of algebraic relationships may be greater than the number of data sets. Numerous datasets and algebraic relationships represent a vast amount of information that can be captured about the datasets in the data store, and use extended set processing and algebraic optimization to efficiently handle very large amounts of data. Enable management. The above are merely examples, and other embodiments may manage different numbers of data sets and algebraic relationships.

  FIG. 5 is a block diagram illustrating an example embodiment of a software module implemented to facilitate importing information into the system. Unlike conventional database systems, the system of this example does not work immediately on presented data sets. Rather, the system stores a reference to the new dataset in the dataset information store. In one example embodiment, this is accomplished by adding a new GUID to the set universe. After the data set is known to the set universe, the system can use the data set.

  As previously mentioned, information can be added to the system through functions contained within the administrator interface 424, which will be described in further detail below. One such method of adding information to the system is by issuing a command 501 to import function 502 for importing information set 506. In one embodiment, the command includes a value that indicates the physical location of the dataset to be imported, the external identifier, and the logical-to-physical mapping that the dataset uses to encode the data for storage. Various physical formats can be supported, including comma separated value (CSV) files, extensible markup language (XML) files, fixed length files (FIXED), XSN format files, and others. In addition, the information set may be located on various non-volatile or volatile storage media, connected locally, or remotely accessed via a network or other communication method. The information set may be distributed over several different physical storage media, or real-time data such as data packets received over the network or input from the user (eg, input in real time by the end user) It may be provided from a stream. After the command is issued, the import function 502 parses the command and causes a data set to be created with an external identifier and physical format value associated with the set manager 503. Next, the set manager 503 creates a GUID for the associated data set and enters various information into the set universe, including physical format type value, external identifier, associated GUID, and GUID realized. . The import function 502 then calls the storage manager 504 to create an association between the data set's physical location identifier and the GUID assigned by the set manager 503. Specifically, the storage manager 504 adds an index record to the storage map 505 that includes the physical path between the data and the associated GUID. Data set 506 is now imported into the system and control is returned to the caller. Even if the data set is not implemented on a storage device (ie, virtualized), the system can capture information about the data set. For example, data set C can be defined as the intersection of data sets A and B. Data sets A and B are implemented in storage, but data set C is defined only by the relationship “C = A UNION B” in the algebraic cache, and the GUID of data set C is added to the set universe. May not be realized in the storage device.

  Statement submission is the process of providing assignments or relationships to the system. Statements can be submitted to the system through various interfaces. In one example embodiment, three interfaces are provided. An SQL connector for submitting standard SQL92 compliant statements, an XSN connector for submitting statements using XSN, and an XML connector for submitting web server W3C XQuery compliant statements and other XML-based statements is there.

  FIG. 6 is a block diagram illustrating an example embodiment of how a software module that facilitates the submission of statements to the system may be implemented. In this example, standard SQL commands are submitted to the system through SQL connector 601. The SQL command can include another standard SQL92 compliant SQL statement. The SQL connector 601 first captures the submission time and establishes a time value for every set included in the submitted statement. The command is then parsed to verify that the syntax of the SQL statement is correct. If there is a syntax error or compliance error, an error message is returned to the caller and submission is stopped. If there are no errors, the SQL connector 601 builds an internal navigable representation of the SQL command, which is output to the SQL translator 602. The SQL translator 602 then converts the internal navigable representation of the SQL command into an appropriate equivalent XSN statement. After conversion, the resulting XSN statement is passed to the XSN interface 603 for future processing. Each statement is then translated from its text XSN representation into an internal structure called an XSN tree. The XSN tree provides a means for programmatically examining members of an XSN statement and for navigating the elements of the statement.

  Next, the XSN tree is examined to determine whether the statement represents an assignment or relationship. If the statement is an assignment, the GUID is assigned by the set manager 402 to the algebraic expression specified in the statement. The XSN tree is then examined to determine whether the GUID is assigned to all datasets and operations in the representation and whether the representation includes any explicit set or any redundant assignment. An explicit set is a set that is entered into the system as part of a statement, such as may occur within the context of a standard SQL “insert” statement. A redundant assignment is an assignment that includes operations and arguments already in the algebraic cache. For explicit collections, the set manager 402 assigns new GUIDs to these collections, which are immediately realized by the set processor 404. In the case of redundant assignments, the GUI of the left value (lvalue) of an existing assignment entry in the algebraic cache is found by searching the algebraic cache looking for expressions containing the same operation and right value (rvalue). Retrieved from 402 and assigned to the lvalue of the facilitative assignment in the representation. If the assignment is not redundant, a new GUID is provided for the assignment from the set manager 402 and assigned to the lvalue of the assignment in the representation. Complex algebraic relationships specified by statements can also be broken down into primitive (atomic) relationships and collections of assignments. GUIDs can be provided for these relationships and assignments, and corresponding algebraic relationships can be added to the algebraic cache.

  Once a GUID has been assigned to all explicit sets and assignments lvalue, control is returned to the SQL connector 601. If necessary, a second call is then made to the XSN interface 603 to realize any set expected to be returned to the caller. The realized set is then returned to the caller.

  FIG. 7 shows an example of a statement that can be submitted to the system with the method of FIG. In this example, the user is querying the database looking for specific information about standard commercial transactions. The request is represented by a standard SQL statement 701. The ORDERKEY required in this case is “12345”. Specifically, the user in this example requests a discount on the item, the date of shipment, and a comment on the item with a specific customer order number “12345”. Information is obtained from two tables: LINEITEM and ORDERS. The two tables are joined based on the L_ORDERKEY field being equal to O_ORDERKEY. The SQL statement 701 is passed to the SQL connector 601 by the user. The SQL translator 602 converts the internal navigable representation of the SQL statement into an appropriate equivalent XSN statement 702. Note that the columns or fields of the LINEITEM table and the ORDERS table have been converted to a representation that is not unique to the relational database. In particular, the columns or fields of the LINEITEM table are here represented by domains “1” to “16”, and the columns or fields of the ORDERS table are here represented by domains “17” or higher. . Starting from the innermost function in the representation, the join operation of the SQL statement 701 is converted to an rdmJoin operation, and LINEITEM, ORDERS, and NULL are passed as three parameters. Next, the result of rdmJoin is passed to rdmRest, and rdmRest uses the L_ORDERKEY domain in the LINEITEM data set equal to the constant “12345” and the L_ORDERKEY from the LINEITEM data set. The field domain “1” is also limited to only data equal to domain “17” of the O_ORDERKEY domain from the ORDERS data set. The XSN statement 702 is then passed to the XSN interface for future processing.

  The XSN interface 603 records the submission time and establishes a temporal value for the set included in the submitted statement. The statement is then translated from the XSN statement 702 to the XSN tree 703. The structure of the XSN tree will be further described later with reference to FIGS. 12A and 12B. As part of the conventional process, a GUID is created or retrieved from the set manager 402 and inserted into the corresponding set of XSN trees 703. Next, the control right is returned to the SQL connector 601.

  Since this example embodiment requested a result set, a second call is then made to the XSN interface 603 to realize any set expected to be returned to the caller. The XSN tree 703 is then returned to the optimizer 604, where the XSN tree 703 is optimized for efficiency and an optimized XSN tree 704 is generated (here, for illustrative purposes, not a tree format) Shown in the format). Note that in this case, the optimizer merges rdmRest with rdmJoin for efficiency. The optimized XSN tree 704 is then passed to the set processor 605 where a set of algebraic relationships in the XSN tree is calculated. The realized set is then returned to the caller.

  FIG. 8 is a block diagram illustrating an example embodiment of software modules implemented to facilitate set realization. Set realization is the process of calculating set membership and realizing such a physical representation of the set in a storage device. A collective realization can be invoked from an external interface of the system that supports the realization, such as an SQL connector or an XML connector, or from an executive software module as part of a set export procedure. In this example embodiment, an export command is issued to executive 801. The command can identify the external identifier or GUID to be exported along with the storage path. The executive 801 passes the external identifier or GUID to the XSN interface 802. If the external identifier is identified in the command, the XSN interface 802 passes the external identifier to the set manager 803. The set manager 803 determines the GUID associated with the external identifier and returns the GUID to the XSN interface 802. This reference is made relative to the time value associated with the GUID. Unless otherwise specified by the user, the example embodiment uses the latest GUID associated with the external identifier. Once the associated GUID is identified, the external identifier is replaced with the associated GUID. Regardless of whether the GUID to be realized is directly specified in the command or obtained from the external identifier, it is next passed to the set manager 803 to determine whether or not to realize it. If the data set associated with the GUID has already been realized, control is returned to the executive 801. If the data set associated with the GUID has not been realized, the GUID is submitted to the optimizer 804 and realized. Next, the optimizer 804 determines the optimal collection of algebraic relationships that represent the data set associated with the GUID. This collection of algebraic relationships is then passed to the set processor 805 where it is calculated. When the collection of algebraic relationships is submitted to the set processor 805, control is returned to the executive 801. The executive 801 then requests that the storage manager provide the data from the data set to the executive 801, which in turn uses the pathname specified in the export command to place the data on the storage device. Remember.

  FIG. 9A is a block diagram illustrating an example embodiment of an algebraic and arithmetic optimizer software module. Prior to submission to the set processor 909, the optimizer processes the collection of algebraic relationships to optimize algebraically and computationally. There are many methods that can be used to determine which collection of algebraic relationships is most efficient based on the system environment and various limitations or performance weaknesses associated with the system.

  In the example embodiment of FIG. 9A, the optimizer operates on two basic principles. First, there is no data set realization alternative plan that has a lower cost than the cost of simply reusing a previously realized data set. Second, the amount of data retrieved across the storage device boundary performance barrier should be minimized. In other example embodiments, especially the state of the art changes, so other principles can be applied. The basic principles are implemented in the exemplary embodiment through three optimization routines: a findAltOps routine 904, a findMetaGuids routine 905, and a findAltGuids routine 906. It is important to note that other optimization routines may be used and that more or fewer optimization routines may exist in the system. In the example embodiment, the optimization routines are executed in a specific order that is designed to attempt optimizations that are likely to result in a collection of algebraic relationships that are as fast as possible and sufficiently low in cost.

  As will be described later, the findLeastCost routine 903 is executed before the optimization routine is executed and after each optimization routine is executed. The cost of executing a particular collection of algebraic relationships is determined by estimating the time it takes for the system to retrieve from the storage device the data set needed to compute the collection of algebraic relationships. The estimated retrieval time can be calculated based on the speed at which information can be retrieved across each I / O storage barrier and the estimated amount of information that needs to be retrieved across such storage barrier. Cost determination includes whether information is read through the same I / O channel or different I / O channels, and whether specific information is used in multiple subparts of the representation, etc. Other factors can also be considered, both of which can affect performance. These optimization techniques can give different optimization results depending on the state of the system when the optimization routine is executed. For example, different data sets having the same logical data can be provided in different data formats having different sizes. If they are available via the same I / O channel, the data set with the smallest format can be selected. However, for recently accessed data sets already available in high speed memory or cache, a larger format may be selected.

  In the example embodiment, the XSN interface 901 calls an optimizer software module to implement a set associated with a collection of algebraic relationships. The XSN interface 901 passes the set of GUIDs to be realized to the buildExpressions routine 902 in the optimizer software module. The buildExpressions routine 902 retrieves one or more original algebraic relationships that define one or more sets identified by a GUID from the algebraic cache. These algebraic relationships can be called genesis expressions. Next, the buildExpressions routine 902 constructs an OptoNode tree representation of such a primitive representation. The OptoNode tree is described in more detail below and can be used to represent algebraic relationships as a collection of more primitive algebraic relationships. Next, the optimizer software module executes a findLeastCost routine 903 to determine the lowest cost source expression. If the findLeastCost routine 903 determines that the primitive expression found to have the lowest cost is sufficiently cheap to execute, further optimization is stopped and the algebraic relationship of such primitive expression is the realizeNode described below. Submitted to routine 908.

  If the findLeastCost routine 903 determines that the primitive representation found to have the lowest cost is not cheap enough to execute, the findAltOps routine 904 is executed to find an alternative operation. This routine synthesizes an alternative version of the primitive expression using extended set theory algebra. The synthesized alternative representation is constructed so that it is potentially less expensive to execute and is easily identified in the algebraic cache. Representation synthesis is based on the perception of the “form” of the representation and substitution of other forms that are algebraically equivalent but are less computationally expensive and / or more likely to be recognized in the algebraic cache. Is called. A simple example is a restriction on two connected sets. Using some notation abbreviation, this can be expressed as SETA = R (J (a, b, c), d). However, the join operation can also be restricted, and the equivalent expression is SETA = J (a, b, CP (c, d)). Both of these forms require the same amount of input data for calculation, while the second form produces less output data. This means that the second form requires less computation and I / O resources. Whether the second form is preferred over the first form depends on what is available from the algebraic cache and which sets have already been implemented in non-volatile storage. However, exploring both forms within the optimizer 418 can increase the probability of finding a more efficient alternative.

  If the findAltOps routine 904 indicates that an alternative representation has been found, the findLeastCost routine 903 is executed again to find the minimum cost representation based on the minimum cost source representation and the alternative representation. Again, if the findLeastCost routine 903 determines that an expression found to have the lowest cost is sufficiently cheap to execute, further optimization is stopped and the algebraic relationship of such an expression is described below. Submitted to the realizeNode routine 908. The threshold for aborting the optimization can be determined based on the relative speed of the processing resources and the data channel and / or other system characteristics. In one example, the threshold is set for 10 MB data transfer. In this example, typically 10 MB of data can be transferred in about 1/10 second, so further optimization is stopped and the set is simply computed from the representation.

  If neither the original expression nor the alternative of the original expression identified by the findAltOps routine 904 was determined to be by the execution of the findLeastCost routine 903, then the next optimization routine is executed. . In the example embodiment, the next optimization routine is a findMetaGuids routine 905. The findMetaGuids routine 905 finds all expressions with incrementally small costs and submits them to the set processor for execution. Expressions with incrementally small costs often include only metadata. Examples of low cost operations include predicate cross product (CP operation), output scope transform (OST operation), and left and right (rdmSFL and rdmSFR operations) relational data model sort domains Can be mentioned. These operations typically operate on the metadata in the user data model to generate additional metadata. The physical set size is typically less than about 500 bytes, making it a prime candidate for fast computation that is well below the execution threshold of the optimizer 418. Thus, these operations can simply be performed immediately from the optimizer 418 without testing whether the minimum threshold is met. Next, the findLeastCost routine 903 is called again and the least cost expression (least-costly expression) is between the least expensive expression determined from the previous call to the findLeastCost routine 903 and the expression from the findMetaGuids routine 905. ) Is selected. Again, if the findLeastCost routine 903 determines that an expression that is found to have the lowest cost is sufficiently cheap to execute, further optimization is stopped and the algebraic relationship of such an expression is described below. Submitted to the realizeNode routine 908.

  If the lowest cost representation identified by the findLeastCost routine 903 is not yet sufficiently cheap to execute, the findAltGuid routine 906 is executed. The findAltGuids routine 906 determines whether one or more subexpressions can be replaced with an alternative expression describing a previously realized set. This routine can be used to provide further cost savings because the cost of reusing a realized set is always lower than the cost of performing the representation necessary to realize such a set. An example of such a substitution can be described using a relational data model. Assume that a particular field (called SIZE, the third field of the table) in the table (called BOXES) has a value in the range of 1-100. Next, the user issues a query (Q1) that requests all boxes of size less than 50. This is expressed in XSN as Q1 = rdmREST (BOXES, {{{“LT”. <“3”, “CONST”. “50”}}}). After some time, the user requests all boxes that are less than 25 in size. This is submitted as Q2 = rdmREST (BOXES, {{{"LT". <"3", "CONST". "25">}}}). In both these queries, if executed as submitted, the entire box data set must be read to identify results Q1 and Q2. However, the metadata set {{{“LT”. <“3”, “CONST”. "50"}}} and {{{"LT". <“3”, “CONST”. A mathematical examination of “25”>}}} indicates that any set limited by the second set is a first subset (subset). Thus, algebraic substitutions could be made and the following expression was generated: Q2 = rdmREST (Q1, {{{"LT". <"3", "CONST". "25">}}}). If Q1 is already implemented with non-volatile storage, this means that the size of Q1 must be less than the size of the box, and therefore the I / O cost required for the transfer is lower. Can show. And if Q1 has already been realized, this provides an overall cheaper means of evaluating Q2 than the originally submitted expression.

  After the subexpression has been replaced with any suitable alternative expression, the findLeastCost routine 903 is executed again to minimize the minimum cost expression determined from the previous call to the findLeastCost routine 903 and the expression from the findAltGuids routine 906. A cost expression is selected. If the findLeastCost routine 903 determines that an expression that is found to have the lowest cost is sufficiently cheap to execute, further optimization is aborted and such expression is submitted to the realizeNode routine 908 described below. The

  After the optimization work described above is completed, the optimizer calls the realizeNode routine 908. The realizeNode routine 908 converts the optoNode tree to an XSN tree, calls the spProcessXsnTree routine, submits the XSN tree to the set processor 909 for execution, deletes the XSN tree, returns control to the optimizer software module, and The optimizer software module returns control to the XSN interface 901.

  FIG. 9B is a block diagram illustrating another example embodiment of an algebraic and arithmetic optimizer software module. Unlike the example embodiment presented in FIG. 9A, the optimization routine in this example embodiment is applied to each OptoNode tree, going from leaf to root. This approach provides the result of each optimization routine as an argument to the expression and creates further opportunities for optimization, but at the cost of increased execution time. This approach would be preferred in situations where further optimization can be performed.

  The implementation in this example embodiment uses only two optimization routines, a findOperational routine 913 and a findAlgebraic routine 914. Unlike the previous example embodiment, the findLeastCost routine 903 is executed only after both the findOperational routine 913 and the findAlgebraic routine 914 are executed. The functions of the findLeastCost routine 903 are the same as those described in the previous embodiment.

  As in the previous embodiment, the XSN interface 901 calls the optimizer software module and passes the set of GUIDs to be realized to the buildExpressions routine 902. The buildExpressions routine 902 is the same as described in the previous example embodiment. After the buildExpressions routine 902 builds the expression's OptoNode tree, the findOperational routine 913 is executed to find alternative operations. This routine performs the same function as the findAltOps routine 904 described in the previous embodiment.

  After the findOperational routine 913 is complete, the modified OptoNode tree is passed to the findAlgebraic routine 914 to find additional alternative expressions. The findAlgebraic routine 914 is repeated on the OptoNode tree from right to left and from the innermost representation to the outermost representation. This repetition order maximizes the potential to find additional alternative expressions. If each representation includes one operation and one to three arguments, each combination of argument and operation is presented to the findExpressions routine 915, one at a time. A findExpressions routine 915 then executes code specific to the operation of the representation, with the intention of finding or synthesizing alternative representations. Code specific to the operation performs an algebraic substitution of the arguments from the algebraic cache, performs a calculation of the low-cost expression contained within the expression, calculates the expression itself, and an alternative form of the expression or argument of any expression Can be synthesized. Next, any alternative representations are added to the appropriate place in the OptoNode tree with code specific to the operation.

  After the optimization work described above is completed, the optimizer calls the same realizeNode routine 908 as the realizeNode routine in the previous example embodiment. Next, the control right is returned to the XSN interface 901.

  The system can also perform overall optimization. Global optimization analyzes relationships and data sets to add new relationships to the algebraic cache that are expected to be useful in optimizing the evaluation of future requirements, and adds sets to the set universe. This can be performed based on a pattern of past requests to the system, which can be used to perform optimization in anticipation of similar future requests. This overall optimization can be performed in the background by using spare processor cycles. 9C, 9D, 9E, 9F, 9G, and 9H show examples of the overall optimization method. However, various other overall optimizations are possible, and these example embodiments are only a few examples within the present invention.

  FIG. 9C shows an example where individual scalar values or free range scalar values identify membership of a subset. Queries of this nature can benefit from creating subsets that divide the data into equal concentration sets, each containing a specific range of values. For example, the data set may have a data distribution indicated by 950 in FIG. 9C. This data set can be divided into multiple data sets of equal density, such as subsets 1-6 shown at 950 in FIG. 9C. An example of this is requesting all transactions that occur after or before specific data. This optimization has the advantage that the set processor reduces the amount of data that must be examined to compute a future subset of similar properties. The overall optimization routine identifies this situation by detecting a significant number of relational constraints from a particular set that uses a range of scalar values by checking the algebraic cache. From these entries, the optimizer determines the maximum and minimum scalar values for establishing the range of scalar values to be divided. The optimizer then determines the number of split subsets to be equal to the average number of available I / O channels. Finally, for each split subset, the optimizer inserts the appropriate relationship into the algebra cache and inserts the set into the set universe. The optimizer can also insert a relationship indicating that the union of the subsets is equal to the set and call the set processor to compute each split subset.

  FIG. 9D shows an example of an overall optimization similar to the optimization of FIG. 9C, except that the membership criteria in the split subset is based on scalar values that fall within a certain range. An example of this is determining a desired subset for a desired range of customer ages. For example, the data in the data set may fall within a specific range shown at 954 in FIG. 9D. This data set can be divided into subsets 1-5 that include each of these ranges, as shown at 956 in FIG. 9D. As with other global optimization examples, this type of partitioning can reduce the data to be examined by the set processor, resulting in improvements through the reduction of required computation time and resources. .

  FIG. 9E shows another form of global optimization, but this optimization example is based on the domain of the members of the set rather than a scalar value. In this example, the optimizer determines that only a particular domain is needed to generate a useful subset and no other domains are needed. For example, the data set 958 of FIG. 9E has columns 1-5, but the optimizer may determine that many requests require the use of only columns 1, 3, and 4. Next, the optimizer creates an entry in the set manager for generating a subset having members including only the target domain, and calls the set processor to generate this subset. For example, a data set with only columns 1, 3, and 4 shown at 960 in FIG. 9E can be created.

  FIG. 9F shows an example in which the scalar value of the target domain is determined to have a relatively low density. An example is a binary domain with scalar values TRUE and FALSE as shown at 962. In this case, the optimizer creates a subset relationship where this domain is monotonic for each value present in the domain while removing the domain from the resulting subset. For example, as shown in 964, a subset can be created for all members of the original data set whose domain value is false, and for all members whose domain value is true. A separate subset can be created. This optimization can have a significant performance benefit because it provides a 100% average performance improvement even in the binary field.

  FIG. 9G shows an example in which the set is composed of a relational combination of two sets. The optimizer performs the join under conditions where the join produces a data set whose cost is less than or equal to the original two sets. An example of this is a relational inner join where there are primary keys and foreign keys that do not correspond between relation sets. For example, the first data set 966 includes three columns (shown as columns 1, 2, and 3 of data set 966 in FIG. 9G), and the second data set 968 includes four columns (of data set 968 in FIG. 9G). Columns 1, 2, 3, and 4). These two data sets are combined to create a third data set 970 having seven columns (shown as columns 1, 2, 3, 4, 5, 6, and 7 of data set 970 in FIG. 9G). be able to.

  FIG. 9H shows an example of vectored multipaging. If the user often accesses information in a specific way (eg, a phone number is used to look up a person's name and address information), the optimizer automatically defines a new data set, By creating multiple pages of one-dimensional arrays (vectored multi-pages), new relationships (eg, defining a data set containing only phone numbers, names, and addresses) can be added to the algebraic cache to make these requests Can be more efficient. For example, the optimizer may determine that a three-digit area code, a three-digit prefix, and a four-digit postfix that are elements of a telephone number should be used for vectored multi-page processing. Next, the optimizer creates a set 972 including 1,000 subsets 974 corresponding to each of 1,000 possible area codes (000 to 999). Each of these subsets contains 1,000 GUIDs that reference a subset of each possible prefix value (000-999), and each of these subsets includes a person's nomination and address for each 4-digit postfix. Contains 10,000 members with information. If all codes are buried, 100,000 subsets can be created based on the area code and phone number prefix. However, since many area code and prefix combinations are not in use, these entries are simply referred to as a NULL set. Once these sets have been created, the set processor uses these sets to simply use the area code as an offset (vector) within the area code set, retrieve the GUID representing the appropriate prefix subset, and then retrieve the prefix. By using it as an offset to determine the GUID of the appropriate postfix subset, individuals can be quickly found based on phone numbers. Finally, the telephone number postfix is used as an offset to find personal data.

  FIG. 10A is a diagram of an OptoNode tree structure. The OptoNode tree is used to keep track of the relationships, expressions, and arguments that the optimizer is processing. There is an OptoNode 1001 at the root of the tree, which is a list of a plurality of OptoExpressions 1002. Each OptoExpression 1002 in the list contains information about mathematically equivalent variants of other expressions in the same list. Specifically, in the example embodiment, each OptoExpression 1002 includes an operation type, a GUID that identifies the expression, and various flags (these Boolean flags indicate whether the OptoExpression has a GUID, and whether the expression is in the algebraic cache. , And whether the OptoExpression is used as part of the alternative representation of the GUID, cost information (value indicating the cost to be used to evaluate the cost of this OptoExpression, and the rest of the contained representation) And a value indicating the cost of representation when implemented independently), and up to three OptoNode arguments. The optimizer creates one or more OptoExpression 1002 to determine the most efficient way to evaluate the desired expression. As described above, the optimizer analyzes each OptoExpression 1002 and determines the cost associated with the expression evaluation. The optimizer can then determine which OptoExpression 1002 to use for efficiency.

  FIG. 10B shows an example of an OptoNode tree. At the root of the tree is an OptoNode 1004 which is a list of OptoExpressions that represent mathematically equivalent expressions. Each OptoExpression contains a list of arguments for that expression. For example, OptoExpression 1006 includes three arguments Arg [0], Arg [1], Arg [2]. Each argument refers to an OptoNode that lists alternative expressions that can be used for the specific argument. For example, the OptoNode 1008 refers to an expression list (List [0], List [1], List [2],...) That can be used for Arg [2] of the OptoExpression 1006. These expressions are represented by OptoExpression 1010, 1012, and 1014. Each of these representations, when used for the argument Arg [2] of the representation represented by OptoNode 1006, provides a mathematically equivalent result. In this OptoNode tree structure, multiple equivalent expressions can be listed at each level of the tree. For example, the findAlgebraic routine 914 (see FIG. 9B) in the optimizer can be repeated on the OptoNode tree to find additional alternative expressions and add them to the OptoNode tree. The FindLeastCost routine 915 can then traverse the OptoNode tree to identify a specific collection of expressions that can be used to calculate the overall result at the lowest cost. The selected collection of expressions is then transformed into an XSN tree and sent to the set processor for computation.

  The set processor is responsible for all calculations and logical value comparisons that the system performs on the data set. In one example embodiment, the set processor is multi-threaded designed to utilize a system that includes multiple processors and multiple independent non-contentional I / O channels between system memory and non-volatile storage. It may be the reentrant software body. Set processors can also be designed to take advantage of data pipelining between operations. In other words, the result of a certain operation can be used as the next input, and the result can be passed as it is without being stored in the nonvolatile storage device. Data pipelining reduces the amount of data that crosses (crosses) the I / O performance barrier and reduces the load on the storage manager responsible for acquiring data from non-volatile storage, thereby improving the efficiency of the set processor Can be greatly improved.

  The execution of various operations is monitored by objects known as thread pools. The thread pool is responsible for starting execution threads for each operation required by the ProcessOp routine, monitoring their execution, and reporting their success or failure. The thread pool also works with executives to limit the current number of threads running in the engine that are needed to manage system resources. Threads may be implemented using different numbers of hardware and software platforms. For example, a conventional single core processor such as processor 102 of FIG. 1 may be used in conjunction with an operating system such as Microsoft Windows® that simulates multiprocessing. In alternative embodiments, multi-processor or multi-core processors may be used, and one or more threads are assigned to each processor. In another embodiment, a multiprocessor system as shown in FIG. 3 may be used, and an execution thread is assigned to each MAP 306a-306f. Regardless of the physical implementation of the system, the set processor in the example embodiment allows a list, tree, or other so that output from one thread becomes input to another thread to increase performance. A structure can be used to chain operations together.

  Each operation in the set processor is an individual routine designed to perform calculations on one or more input data and generate an output data set. These operations are equivalent to extended set operations and functions that have been found useful for data processing. The set processor can also have multiple implementations of algorithms for each operation to support a wide range of physical-logical format mappings. By adapting the arithmetic routine for a physical data format, higher efficiency and performance can be achieved than converting all data into a single physical representation for processing. An example embodiment may include, for example, data in comma separated value (CSV) format, binary string encoding (BSTR) format, fixed offset (FIXED) format, type encoded data (TED) format, and / or markup language format. Supports logical-to-physical mapping between different formats so that they can be mapped between. This allows the system to process the data without having to convert all the data to a common format. For example, if the system needs to calculate the result of a combination of a first data set in CSV format and a second data set in XML format, the mapping can be done without having to convert any data to another format. Can be used to calculate results and return such results in CSV format, XML format, or another selected format. In addition, one example embodiment provides several logical-physical mappings for atomic values such as strings, 32-bit integers, 64-bit integers, floating point numbers, currency, Boolean values, datetime values, and interval values. As well. These mappings can be used in the same manner as the data format mapping. The system may include all potential mappings to the various supported data formats and atomic formats, or may include only selected mappings. For example, if the example embodiment supports 5 data formats, there are 5 inputs and 5 outputs for each mapping routine, resulting in 125 potential versions of the software routine. In the example embodiment, software routines for mapping between various formats are included only if there is a substantial increase in efficiency. If no substantial increase results, the example embodiment converts the data to a common format instead of using a mapping function.

  Another function of the set processor is to provide an object-oriented data model instance of a common schema that is used throughout the program. This includes predicate sets, domain sets, concentration sets, and others that satisfy certain definitions and are useful structures within algebraic and computational processes performed by the program.

  FIG. 11 illustrates an example embodiment of a set processor software module. In the example, the optimizer presents the XSN tree to the set processor for evaluation through the spProcessXsnTree routine 1102. The spProcessXsnTree routine 1102 examines the XSN tree and determines whether the XSN tree represents an assignment or relational statement, or whether the XSN tree represents an explicit set.

  In the case of an assignment statement, the ProcessXsnAssignment routine 1105 determines whether the left value (lvalue) of the statement is confirmed to be an XSN set. If lvalue is not a set, the routine returns a failure code. Next, the right value (rvalue) is examined to determine if it is an operation or an explicit set. If the rvalue is an explicit set, the rvalue's GUID is associated with the external identifier associated with the lvalue. If rvalue is neither an operation nor an explicit set, the routine returns a failure code. If rvalue is an operation, the ProcessXSN routine 1107 is called and processing continues.

  In the case of a relational statement, ProcessXSNRlation routine 1106 examines and confirms that lvalue and rvalue are operations. If either or both are operations, the ProcessXSN routine 1106 is called to continue processing for either or both. If lvalue or rvalue is not an operation, it is simply ignored. The goal is to realize an arbitrary set referenced in a relational statement, typically but without limitation, so that the relation can be evaluated to support the optimizer.

  In the case of a request to realize an explicit set, the spProcessXsnTree routine 1102 immediately realizes the set in routine 1103 and returns a GUID that identifies the realized set.

  The ProcessXSN routine 1107 examines all members of the XSN tree starting from the current operation at the root of the XSN tree and repeatedly calls itself for all operations. Each operation to be performed is passed to the ProcessOP routine 1108 in an order that ensures that the root operation of the XSN tree is started before the subordinate operations to establish the proper data pipelining.

  The ProcessOp routine 1108 takes each operation and inserts all sets of appropriate GUIDs into the thread pool 1109 in a state associated with the operation to be executed. The thread pool 1109 then starts an individual execution thread for each operation in the statement presented to the ProcessXSN routine 1107. These execution threads are then executed independently by calling the appropriate operation 1110 until the operation is complete. As each of these threads completes, the thread pool 1109 is notified of the thread completion, and the thread pool 1109 provides appropriate cleanup and error handling including removing the thread from the active thread list.

  The set processor also includes a function known as an spLogical routine that is designed to perform logical operations on the set. These logical operations are essentially different from the computational operations performed by the set processor's spProcessXsnTree routine 1102. The spLogical routine includes spLogicalEqual, spLogicalPredicateEqual, and spLogicalPredicateSubSet, and is typically designed to compare two data sets stored in binary XSN notation to determine the logical relationship between them. These relationships include equivalence, subsets, supersets, and disjoint. These functions are used by the optimizer in determining alternative representations.

  FIG. 12A is a diagram of an XSN tree structure that can be used to represent an example XSN representation in the system. The XSN tree provides a convenient format for processing XSN representations in the system. FIG. 12A shows an XSN tree of the expression A REL OP (B, C, D). This representation associates data set A with operations (OP) performed on data sets B, C, and D by relation (REL). The XSN tree is a double linked list composed of a relation node 1201, an operation node 1205, a plurality of member nodes 1202, 1203, 1206, 1207, and 1208, and a plurality of data sets 1204, 1209, 1210, and 1211. . The relation node 1201 designates a relation of expression such as equal sign, less than, and super. Relation node 1201 is linked to member node 1202, which has a link to dataset A 1204 (the left value of the statement) as its left child and a link to member node 1203 as its right child. . The member node 1203 is linked to the operation node 1205 as a child on the left side. The action node 1205 identifies operations to be performed such as projection, restriction, and combination. The compute node 1205 is linked to a member node 1206, which has a link to dataset B 1209 as its left child and a link to another member node 1207 as its right child. Member node 1207 has a link to dataset C 1210 as its left child and member node 1208 as its right child. Member node 1208 is linked to data set D 1211.

  FIG. 12B is a diagram of an XSN tree structure that can be used to represent an example XSN assignment statement in the system. FIG. 12B shows the XSN tree for the assignment statement SQL1 = rdmPROJ (rdmREST (A, C1), C2). This statement assigns the alphanumeric identifier SQL1 to the expression rdmPROJ (rdmREST (A, C1), C2). The XSN tree is composed of an allocation node 1251, an alphanumeric identifier 1254, a plurality of member nodes 1252, 1253, 1256, 1257, 1260, and 1261, operation nodes 1255 and 1258, and a plurality of data sets 1259, 1262, and 1263. This is a double linked list. Allocation node 1251 is linked to member node 1252, which has a link to alphanumeric identifier SQL1 1254 as its left child and a link to member node 1253 as its right child. Member node 1253 is linked to operation node 1255 (rdmPROJ) as a child on its left side. The operation node 1255 identifies an operation to be executed (in this case, projection). The compute node 1255 is linked to a member node 1256, which has a link to the compute node 1258 (in this case, the limit computation rdmREST) as its left child and to another member node 1257 as its right child. With links. Member node 1257 has a link to data set C2 1259 as its left child. The compute node 1258 is linked to a member node 1260, which has a link to data set A 1262 as its left child and a link to another member node 1261 as its right child. Member node 1261 is linked to data set D 1263. In an example embodiment, these XSN trees can be stored inside the system as an array.

  A storage manager 420 maintains the actual data, including each collection, and provides efficient transfer between non-volatile and volatile storage devices.

  13A, 13B, 13C, and 13D illustrate how a buffer chain can be used in the storage manager 420 to enable data sharing through data pipeline transfer and buffer chaining. Show. Note that this is just one example embodiment, and there are various ways in which the storage manager 420 can be implemented with or without a buffer chain. The storage manager 420 provides access to set data via a simple mechanism in the form of a SetReader class and a SetWriter class (called reader and writer for short), which are separate subclasses of a class called SetBase. The reader reads data from the storage device, the writer writes data to the storage device, and the reader and writer together encapsulate the more complex functions of the storage manager 420.

  This encapsulation allows a flexible storage manager 420 implementation that can be different for different platforms or storage systems. In addition, the underlying storage manager 420 can provide pipelining between operations to minimize the amount of data that must be transferred from a physical storage device. In this sense, pipelining is the sharing of the underlying data buffer, regardless of whether data is being written or read. As an example, consider operation A (OpA) and operation B (OpB). However, OpA generates (and thus stores) data, and OpB needs to read the data. In the non-pipeline approach, OpA simply writes data and OpB reads the data from the storage device in separate operations. Instead, in the design of the storage manager 420, when OpA writes data and OpB creates data, in fact, it is often the case before the data is actually written to the storage device. You can access the data. Since OpB only knows the SetReader interface, there is no need to know that the data actually comes from the output of OpA, not from the storage device. As a second example, consider OpC and OpD, both of which need to read data from the same set. The pipelined storage manager 420 reads the data only once for both operations.

  This mechanism is shown in FIGS. 13A, 13B, 13C, and 13D. Data sets are generated by the operation of a set processor or retrieved from disk via a storage manager. In either case, a writer is used to place data sequentially in a linked list of RAM buffers known as buffer chains. When the set processor operation requests data from the data set, the reader is used to sequentially retrieve the data from the linked list of RAM buffers for use in computations. In one example embodiment, the data set has only one writer, but may have any number of readers. This is shown in FIG. 13A, which shows a buffer chain 1302 that includes four serial buffers Dbuf1, Dbuf2, Dbuf3, and Dbuf4. A writer 1304 is used to refer to a buffer in which data in the buffer chain 1302 is to be written. The writer 1304 proceeds sequentially through the buffer chain, and if additional data is attached to the buffer chain by the writer, a new buffer is created. Readers 1306 and 1308 are used to refer to buffers from which data can be read from the buffer chain 1302.

  Due to the nature of operation within the set processor, a data set being read by more than one reader is likely to have readers that travel at different paces through the data. For example, as shown in FIG. 13A, the low-speed reader 1308 is reading Dbuf1, while another reader 1306 has already read Dbuf3. As the writer and reader progress through the buffer chain, the writer creates additional buffers, and the reader is free to proceed through the data at that pace whatever the pace required by the operation in the set processor. FIG. 13B shows the same combination of reader and writer shown in FIG. 13A, but the writer 1304 has advanced to Dbuf7, the reader 1306 has advanced to Dbuf6, and the low speed reader 1308 remains Dbuf1.

  As the set processor continues, a long series of buffers may be created between the slow reader 1308 and the writer 1304 and reader 1306 ahead, as shown in FIG. 13C. As the buffer chain 1302 grows, the free space in the RAM consumed to hold data in memory increases. At some point, the amount of RAM in use becomes excessive due to the needs of other routines that require additional RAM, and some of the RAM must be freed so that these other routines can be used. No longer. If this situation is detected, a buffer chain break can be started.

  A buffer chain break is achieved by creating additional buffer chains associated with the data set. In the example shown in FIG. 13D, the low speed reader 1308 that has now advanced to Dbuf2 is copied to a new buffer chain 1310. This new buffer chain 1310 is also assigned a new writer 1312 to provide serial data from the disk. The existing buffer chain 1302 includes Dbuf3 to Dbuf12 here, and includes only the writer 1304. Since there are no more readers behind the writer 1304, Dbuf3 to Dbuf11 are removed by the DoCleanup routine, which is a separate asynchronous routine that frees RAM buffers that are no longer used by the storage manager. This provides a substantial amount of RAM that can be used by other routines that require additional RAM, since the number of buffers can be quite large.

  In addition to providing optimized data storage and retrieval, example embodiments can also be used for conversion and mapping of requests and statements between different schemas using different data models. For example, the system can include mappings between schemas that use different data models, such as an SQL data model, an XML data model, an XSN data model, or other data model. Statements can be provided based on schemas that use different data models. For example, a number of query language statements based on a first schema can be provided in a first format, such as SQL format. As described above, these statements can be converted to XSN format, and data sets and algebraic relationships from these statements can be assembled and stored in set manager 402. Later, the statement may be received in a second format, such as an XQuery format. This can also be converted to XSN format and the data set and algebraic relationships from this statement can be assembled and stored in the set manager 402. In particular, this statement may require provision of a data set based on a schema that uses the second data model. Since all statements are converted to a unified XSN data model, the optimizer 418 was requested in the second format using a data set and algebraic relationships assembled from the statements received in the first format. A collection of algebraic relationships optimized to compute the data set can be determined. Mapping between algebraic relationships and schemas stored in the algebraic cache uses the datasets and relationships captured from the statements in the first format to optimize and calculate the datasets required in the second format. It becomes possible. This allows multiple different data models to be supported on a single system. All information from the statement is captured by the set manager as data sets and algebraic relationships, so the system can convert between models. In addition, this information can be used to replace the subexpressions and other optimization techniques used by the optimizer as described above for the algebraic values used to calculate the data set with respect to other data models. The relationship can also be optimized. The data model may be a relational data model, a markup language data model, a set notation data model, or other data model. The format of the statements submitted to the system can include standard query language statements, XQuery statements, set notation statements, or other formats.

  As an example, consider the relationship table and XML document presented in FIG. 14A. The relational table can be expressed mathematically as an extended set. Members of the extended set that represent the relationship table are commonly referred to as rows in the relationship table. Rows in the relationship table can also be expressed mathematically as an extended set. Members of the extended set that represent rows in the relationship table are commonly referred to as fields. Fields that are common to rows are called columns. Therefore, the relationship table can be expressed by an extended set of the form << f11, f12, f13,..., F1c >>,..., <Fr1, fr2, fr3,. Here, f represents a field value, and subscripts r and c represent unique row and column count values.

  An XML document can also be expressed mathematically as an extended set. Members of an extended set that represent an XML document are commonly referred to as XML fragments and include tags and values that represent data. The value of these XML fragments can be a string or another XML fragment. Therefore, an XML document is <t1. {V1}, ..., tn. It can be represented by an extended set of the form {vn}>. Here, t represents the tag of the XML fragment, and v represents the value of the XML fragment.

  Using an appropriately defined extension set, members of the extension set conversion function gRX () representing the relation table can be mapped to members of the extension set representing the XML document, and the data in the relation format or XML format. Enables transparent representation of The transformation function provides a structural relationship between the fields in the relationship table and the fragments in the XML document and operates on the extended set representation of the relationship table. As a result of this information, a functional mapping is provided between the XML representation values and structures of the same data as the relation representation values and structures.

  The transformation function can be stored in the algebraic cache as a collection of relationships between the relationship table and the collection of XML fragments. To map from the XML document to the relational table, the respect of the function fXR () shown as gRX () in FIG. 14A is used. In order for these functions to provide proper mapping, constraints on terms and term relationships must be valid. These constraints are listed as a where clause in FIG. 14A. With membership constraints that x and z are in B and B and D are in C, a is s. The constraint that it must be equal to {x} indicates that the XML fragment must contain only one value. Furthermore, with the membership constraint that x and y are in A, b is s. The constraint that it must be equal to x indicates that the relationship field in a particular row must have only one value. Together, these constraints ensure a unique mapping from XML fragments to fields in the relationship table.

  Another example is the mapping of a vector representation of a directed graph to a relational data table. The directed graph shown in FIG. 14B includes paths and junction points. At each junction point, one or more paths enter and exit the junction point, with the exception being the start point of a directed graph that has only paths that exit from it and the end point of a directed graph that has only paths that enter it. Each junction point of the directed graph and the path that enters and exits each junction point are {from. {P1, p2,..., Pm}, to. It can be expressed as an extended set of the form {pm + 1, pm + 2,..., Pn}}. Values p1-pm uniquely identify the path from the junction point, and values pm + 1-pn uniquely identify the path to the junction point. Therefore, the directed graph is an extended set {j1. {From. {P11, p12,..., P1m}, to. {P1m + 1, p1m + 2,..., P1n}}, j2. {From. {P21, p22,..., P2m}, to. {P2m + 1, p2m + 2,..., P2n}},. {From. {Pk1, pk2,..., Pkm}, to. {Pkm + 1, pkm + 2,..., Pkn}}. In this case, the conversion function is fNR (). The transformation function that completely maps the directed graph to the relational table is explicitly defined as presented in FIG. 14B. As with the relationship-to-XML mapping, constraints are needed to enforce the rules for each model and provide a mapping of values and structures between models. The directed graph is completely represented by the extended set N. The extended set N is a term nk.d that represents the path of all k joints in the graph. This is the union of Jk. Path nk. Jk is the from path f. Fk and two-pass t. Defined with respect to Tk. The relation table is represented by an extended set R. The extended set R is a union of terms Rijk representing each row of the relation table including fields from to (from) to (to) and the path. The remaining constraints define the relationship between terms and constraints on the terms themselves. This is because the constraint that f, t, and p must exist and must not be equal to each other, Fk is set to {xi} to define the relationship between the relationship table field and the directed graph path. The constraint that Tk must be equal to {yj}, the constraint that the pair of Fk and Tk representing the path must be unique, and the range f and range of each path represented by Jk It includes the constraint that there is one unique value for t.

  It will be appreciated that the above formats, schemas, and mappings are merely exemplary, and that other formats, schemas, and mappings may be used in other embodiments.

Example of Extended Set Notation As described above, extended set notation (XSN) can be used in example embodiments. In the following, an example of an extended set notation (XSN) that can be used will be described. This is just one possible embodiment of extended set notation, and other embodiments may use different terms, set types, syntax, parameters, operations, and functions than the following. The extended set notation example provides a simple and easy-to-use syntax for specifying and manipulating expressions based on extended set mathematics within the environment of modern computing systems. This notation can be expressed in standard ASCII characters and provides a standard syntax for representing values, sets, operations, relationships, and representations suitable for computer-based operations and processing. This notation provides the ability for standard ASCII characters to specify an algebraic extended set representation in machine-readable form.

  Terms used to describe and identify the major components of XSN are defined in Table 1 below.

  Syntax: The XSN system includes a symbolic means of specifying a set and a grammar for formulating expressions and statements. In the following description, terms enclosed in parentheses ([]) indicate optional syntax. For example, if no range is required, the element is [Range]. Expressed as a component. The ellipsis (...) indicates a series of arbitrary length expressions. For example, <“1”, “2”, “3”,.

  symbol. The syntax utilizes certain common symbols listed in Table 2 below. For ease of reading, an optional space may be inserted between punctuation marks if desired. For clarity, line breaks can occur anywhere in a statement, expression, or set.

  Value: The value is specified by explicitly describing the value within double quotes. Examples of values include “Curly”, “123”, and “$ 2,343.76”. If the value contains a double quote (“), it can be delimited by inserting a double quote before it, eg“ John said “” shoot ”” when he saw the moose. ”Null value Is specified by two consecutive double quotes such as "".

  An alphanumeric identifier. The set identified by the alphanumeric identifier is specified by the assignment statement. If specified, an alphanumeric identifier can be used synonymously with the expression to which the alphanumeric identifier is assigned. For example, if an alphanumeric identifier NDCENSUS1960 is assigned to a set, NDCENSUS1960 can be used in any representation to refer to the set to which NDCENSUS1960 is assigned.

  Ranges, Constituents, and Elements: Ranges and components can be represented by values, alphanumeric identifiers, elements, or collections. The syntax of the element is the [Range.] Component. Ranges are separated from components through the use of a period, the term to the left of the period representing the range, and the right side of the period representing the component. For example, an element whose range has the value “1” and whose component has the value “Bob” is represented in the appropriate notation as “1”. ”Bob”.

  An element is a range or component having a composite structure that requires at least one range and one component. The component must be explicitly described, but the null value for the range is not explicitly described but is implied. In the above example, the element “1”. “Bob” has the range “1” and the component “Bob”. However, both ranges and components can be alphanumeric identifiers, elements, and collections, which can result in potentially complex expressions.

  One problem that arises from these potentially complex expressions is the priority with respect to scope and components. For example, when the element “integer”. ”Sum”. ”5” is given, a problem occurs in the division between the range and the component. That is, whether the range is "integer" or "integer". "Sum". The problem is whether the component is “5” or “sum”. ”5”. By convention in this XSN example, the left term of the first period is a range and the right term is a component. Then, this indicates that “integer” is a range and “sum”. ”5” is a component. However, if you want “integer”. ”Sum” to be a range and “5” to be a component, this is specified through the use of parentheses like element (“integer”. ”Sum”). ”5” be able to.

  Members and sets. A member is an element, set, or representation contained within a set. A set is specified by expression or by enumerating individual members, and some or all of the members can be elements, sets, or expressions. Any sets that contain the same members listed in any order are the same set.

  In many cases, members of a set include a range that belongs to a set of natural numbers. In some cases, these ranges are contiguous and unique and include the value 1. In such cases, the set can be called an ordered set. All sets that do not meet these criteria can be referred to as unordered sets.

  A set is represented as {member [, member [, ...]]}. The members of the unordered set are enclosed in curly braces, such as {"a", "x", "b", "g"} or {"Groucho", "Harpo", "Gummo"}. The members of the ordered set are enclosed in angle brackets, such as <"a", "b", "x", "g">. The members of the ordered set have an implicit order listed in the specification. The range of each consecutive member of the ordered set is the corresponding member of the natural number set. Thus, <″ a ″, ″ b ″, ″ x ″, ″ g ″> is {″ 1 ″. "A", "2". "B", "3". ″ X ″, ″ 4 ″. It is equivalent to "g"}.

  For example, an ordered set is a data record having an arbitrary number of data fields, where the members of the set represent the fields of the record, and the range of members is the order position within the record of the corresponding field. Can be represented. The comma-separated values in the first row of the following table can be specified as a set for processing. Data can be grouped into a hierarchy in many different ways. Table 3 below shows some possibilities.

  The original comma-separated value includes four value sequences, each value sequence having three values.

  Set 1 is designated as an unordered set of four members, each member comprising an unordered set of three members.

  Set 2 is designated as an ordered set of four members, each member containing an unordered set of three members.

  Set 3 is designated as an unordered set of four members, each member comprising an ordered set of three members.

  Set 4 is designated as unordered. Set 4 uses a range to indicate the position of each member of the set relative to the other members of the set.

  The contents and structure of a set are sometimes determined by the purpose of the set, especially when the set is used as an argument for functions and operations. Some of these determined structures frequently occur when specifying XSN examples to describe relational data operations. Some such common sets are commonly referred to as predicate sets, mapping sets, transformation sets, or aggregate sets, and are discussed in more detail below.

  Predicate set: A predicate set provides a specification of the mapping between members of one set and members of another set. A predicate set describes a nested conditional expression to determine true. In the case of a conditional expression as used in the RDMREST function, the basic condition is expressed as “condition”. <Element1, element2>.

  The element is "column value" or "const". Can be specified as "scalar value". Conditions are equal ("EQ"), not equal ("NEQ"), less than ("LT"), less ("LE"), more than ("GT"), more ("GE"), and so on ( "LK") or not ("NLK"). For the RDMREST function, each element specifies a constant scalar value indicated by the column or range “const” to be compared in the condition.

  For example, the condition is “EQ”, where the condition is EQ, the first element is the name of the column, and the second element provides a constant value. <“2”, “const”. "MI"> indicates that all members (rows) whose second column is equal to the value "MI" are included in the output set.

In the following example, a single condition is specified for the RDMREST function predicate set. The resulting set includes only members (rows) that contain the value “IN” in the third column from the set zipcitystate. Note the two additional sets of braces.
RDMREST (zipcitystate, {{{″ EQ ″. <″ 3 ″, ″ const ″. ″ IN ″>}}})

  These are necessary to support the construction of AND and OR operations described below.

AND statement: The condition set is an AND statement, and all conditions in the list are ANDed together. If all are true, the overall condition is true. This is an example of an AND structure.
{{"EQ". <“2”, “const”. "MI">}, {"GE". <″ 5 ″, ″ const ″. "49000">}, {"LT". <″ 5 ″, ″ const ″. "51000">}}

  Three conditional sections are enclosed in a set of braces to delimit the AND statement.

OR statement: An OR statement is created by combining two or more AND statements. If the result of any AND statement is true, the entire statement is true. Here is an example
{{{"GE". <“1”, “const”. "10000">}}, {{"GT". <“3”, “const”. "AK">}, {"LT". <“3”, “const”. "CA">}}, {{"EQ". <“2”, “const”. "Pasadena">}}})
It is.

  In this example, there are three OR statements. The first OR statement contains one conditional phrase, the second OR statement contains two AND conditional phrases, and the last OR statement contains a single conditional phrase. In this way, complex conditional expressions can be constructed to define operations.

Mapping set: Some operations and functions require the set to provide a mapping. In most cases, ranges and components are used to provide a relationship between input and output sets. For example, in the RDMPROJ operation, the set provides a mapping between the columns of the input set and the columns of the output set. The range value indicates an output set column, and the component indicates an input set column. For example,
<″ 3 ″, ″ 5 ″, ″ 1 ″>
It is.

  This predicate set indicates that the third, fifth, and first columns of the input set should be mapped to the first, second, and third columns of the output set.

Transformation set: A transformation representation is used to transform one or more input values from a set to values in an output set. The transformation includes operations such as subtraction (“SUB”), addition (“ADD”), division (“DIV”), and multiplication (“MUL”). The additional conversion operation is a constant (“CONST”). Transform expressions are typically used with relational operations such as RDMATH to define the members of the output set. For example, if the first column of the output set is designed as the sum of the first column and the second column of the input set, the following transformation set is used to specify this:
<“ADD”. <″ 1 ″, ″ 2 ″ >>

This indicates that the first and second columns of the input set should be used as the first and second arguments of the addition transformation to generate the value of the first column of outputs. Transformations can be deeply nested to provide specifications. For example, if the calculation (COL1 + COL2) / (COL3-1) is desired to represent column 1 in the output set, and column 5 and column 6 of the input set are mapped to column 2 and column 3, the transformation set is
<"DIV". <“ADD”. <″ 1 ″, ″ 2 ″>, ″ SUB ″. <“3”, “CONST”. ″ 1 ″ >>, ″ 5 ″, ″ 6 ″>
It is.

The transformation set can also include specific scalar domain modifiers. For example, if mathematics is performed in the integer domain, <"ADD". An example of <“1”, “2” >>
<("Int64". "ADD"). <″ 1 ″, ″ 2 ″ >>
Is expressed as

  This indicates that the scalar values in columns 1 and 2 are added together as if they were integer values. Results are also generated in the integer scalar domain. Like function names and operation names, scalar domain identifiers are not case sensitive.

Aggregate sets: Aggregates are also used in the RDMGROUP function to provide aggregates. Aggregation operations include addition (“SUM”), average (“AVG”), count (“CNT”), minimum (“MIN”), and maximum (“MAX”). These functions specify the operations to be performed on the members of the set in each group created by the RDMGROUP function. For example,
<“1”, “3”, “COUNT”. "1", "AVG". ″ 1 ″>
It is.

  This indicates that the first and third columns of inputs provide the basis for the group and are included as the first and second columns of outputs. The third column of output is the count of members from column 1 in the group and the fourth column is the average of members in column 1 of the group.

Like a transform set, an average set can specify a scalar domain on which to perform an operation. For example, if the above is to be performed in a string domain, the specified set is
<"1", "3", "(" STRING "." COUNT ")." 1 ", (" STRING "." AVG ")." 1 ">
It is.

  Functions and operations: Functions and operations are explicitly specified, and the function or operation defines a set that is specified in combination with one to three sets that provide arguments to the function or operation. In other embodiments, a different number of arguments may be possible. The operation is atomic and is specified in extended set mathematics. A function is a combination of one or more operations, and is a convenient notation for a combination of frequently executed operations.

  Functions and operations are expressed via predefined alphanumeric identifiers, parentheses, and 1-3 set arguments. An example is CRD ({"1", "2", "3"}) representing a set that is a density set {"1", "2", "3"}.

  In general, a function is specified as a function (expression 1 [, expression 2 [, expression 3 [, ...]]). However, the number of arguments depends on the function. In particular, unary functions require one argument, binary functions require two arguments, and ternary functions require three arguments. For some functions, the last argument is a set that is used to specify mappings and transformations. Unlike alphanumeric identifiers used in sets, function names and operation names are not case sensitive.

  The following are some examples of functions.

  RDMPROJ (ASet, <"7", "1", "2", "3">)-RDMPROJ is a relational data model (RDM) projection function. A set with the name ASet is an argument to the operation representing the relation table. The second set is a set that specifies the mapping of members (columns) from the ASet to be used as columns in the resulting set.

  INV (OldSet)-INV is an inversion function, which exchanges the range of members of a set and its constituent elements. The set named OldSet is the argument of the operation and is inverted to produce the output.

  CRD (MySet)-CRD is a density function and generates a density set of the input argument set. A set named MySet is an input and is used to generate an output set.

  RDMJOIN (cities_and_states, states_and_zips, {{{"EQ". <"2", "3">}}})-RDMJOIN is a relational data model (RDM) binding function. The first two sets named cities_and_states and states_and_zips are objects to be combined by operation. The explicit predicate set provided in the third set specifies the conditions used to select members of the resulting combined set. In this case, the predicate set specifies that the second column of the first set is equal to the first column (state column) of the second set and that the rows should be joined in the output set.

  RDMREST (zips, {{{"GE". <"1", "const". "10000">}, {"LE". <"1", "const". "14999">}}, {{" GT ". <" 3 "," const "." AK ">}, {" LT ". <" 3 "," const "." CA ">}}})-RDRMEST is a relational data model (RDM ) Limit function. The first set named zips is an operation argument representing a relation table. The second argument is a predicate set that specifies which members (rows) should be included in the restricted output set.

  In this example, functions with names that begin with RDM (Relational Data Model) are specifically designed to operate on relational data as a set. For example, RDMSORT is a binary function that sorts a first set of arguments using members in the second set of arguments that indicate the sort order and procedure.

Expression: An expression is a symbolic expression that specifies a set. An alphanumeric identifier that represents a collection is the simplest form of representation. An expression can also consist of many functions, operations, and sets. Some examples of expressions include
CRD (SetA)
rdmPROJ (SetA, <″ 1 ″, ″ 5 ″, ″ 23 ″>)
CRD (rdmPROJ (SetA, <″ 1 ″, ″ 5 ″, ″ 23 ″>))
Is mentioned.

Relations and relational operators: Relational operators are symbolic expressions that specify the relationship between two expressions. Relational operators include equality, subsets, and disjointness, and their negation. These are specified using the values “EQ”, “SUB”, “DIS”, “NEQ”, “NSB”, and “NDS”, and some example sentences using relational operators include:
SetA EQ CRD (SetB)
SetC SUB SetB
Is mentioned.

Assignment: An assignment is a statement that assigns an alphanumeric identifier to an expression. Syntactically, the assignment is specified as an alphanumeric identifier = expression. For example,
NewSet = <"1", "2", "12", "4">
SetA = SS (SETB)
SetC = <″ b ″, ″ c ″, ″ a ″, ″ x ″>
SetD = {"Larry", "Moe", "Curly"}
SetG = NULL
It is.

Relational data model: The relational data model (RDM) is a subset of the extended set data model that can be described using XSN. A relational table is considered to be a set of ordered sets, and the rows of the table are represented by these ordered sets. A member of a set representing a row is a column (field) value in the row. A relational table with 3 rows, each row containing 4 columns, has the structure:
<<< a1, b1, c1, d1>, <a2, b2, c2, d2>, <a3, b3, c3, d3 >>
Represented by a set with

Both tables and individual rows are represented as ordered sets, but relational tables
{<A1, b1, c1, d1>, <a2, b2, c2, d2>, <a3, b3, c3, d3 >>
It is also possible to express as a set including members that are not ordered.

Concentration set: When a set is presented as ordered, information indicating the order of the set must be presented. To retain some of the additional features of the relational data model and to provide concentration information useful for optimizing the processing of the XSN representation, the concentration set is usually It is specified. The concentration set of the unordered set is
<"3", <"4", <Ca, Cb, Cc, Cd >>>
It is.

The concentration set is a nested set. The outermost set contains the density of the set (in this example, 3 because the table contains 3 rows), followed by the density set of members that represent the rows. Ca to Cd are values representing the density of values constituting the members of the set representing the row. Each value of Cn represents the maximum concentration of that particular member. The concentration set is the concentration function:
CardinalityOfSetA ＝ CRD （SetA）
Is generated through.

  RDM function: The standard relational data model consists of 8 operations. However, it can be seen that only five are needed to implement the entire relational model, and usually only four are used in the actual implementation. XSN provides a notation for these functions within the framework of extended set mathematics.

  These functions are an extended set of relational data models sum (RDMUNION), projection (RDMPROJ), restriction (RDMREST), join (RDMJOIN), difference (RDDMDIF), and quotient (RDDMIV). In addition to these functions, three additional functions are available under XSN. These include RDMSORT, RDMPIVOT, and RDMGROUP.

RDDMIFF function. RDDMIFF defines an unordered set equivalent to a relational AB operation. The resulting set contains all members of A that are not in B. The following is an example format and description of this function.
RDMDIFF (A, B) == {}

argument:
A-Unordered set B-Unordered set in which members are excluded from A to produce a result

  Result: unordered set containing members of A that are not members of B specified by different function conditions

  Note: As an extension to the standard relationship difference where all values of column members need to be equivalent, the XSN version can specify a set of predicates that define an equivalence relationship. If a null is provided in the conditional predicate set, a standard RDM function is executed. For A EQ B, the result is a null set. If the product set of A and B is a null set, the result is A.

  Requirement: Set A must be an RDM set. The result when these conditions are not met is a null set. A and B must have the same member column density.

Example:
A = {<″ a ″, ″ b ″, ″ c ″>, <″ d ″, ″ b ″, ″ r ″>}
B = {<"3", "c", "8">}
RDMDIFF (A, B) == {<"a", "b", "c">}

RDMGROUP function. RDMGROUP defines an unordered set in which columns are aggregated based on a specified group identified by one or more column members. In conjunction with the aggregate predicate set, this function provides the ability to generate sum, count, average, minimum, and maximum (SCAMM) values. The following is an example format and description of this function.
RDMGROUP (A, Z)

  Argument: A-ordered or unordered set

  Result: an unordered set containing members generated based on the aggregate functions of the members of the columns of the set A and the specified aggregate predicate set Z

  Note: RDMGROUP generates one member row for each unique combination of member column values specified in the predicate set. The member columns to be aggregated are specified by enumerating in the predicate set without a range. Other members to be included in the output set should indicate which SCAMM values should be performed to generate the output set.

  Requirement: Set A must be an RDM set. Set Z must be an aggregate predicate set. The result when these conditions are not met is a null set.

Example: A = <<"3","Tom","a">,
<″ 2 ″, ″ Sam ″, ″ c ″>,
<"6", "Harry", "a">,
<″ 7 ″, ″ Harry ″, ″ a ″ >>
Z = <“3”,
"COUNT". ″ 2 ″,
"SUM". ″ 1 ″>

RDMGROUP (A, Z)->{<″ a ″, ″ 3 ″, ″ 16 ″>, <″ c ″, ″ 1 ″, ″ 2 ″>}

RDMJOIN function. RDMJOIN is a concatenation of one member row from set A and one member row from set B when the conditional predicate set Z is satisfied between these two member rows. Define an unordered set. The following is an example format and description of this function.
RDMJOIN (A, B, Z) == {}

argument:
A-Unordered set joined as the left side of the resulting member row B-Unordered set joined as the right side of the resulting member row Z-Condition set for determining the members to join A predicate set containing

  Result: An unordered set in which members that match the conditions specified in the conditional predicate set Z are created from one member row from A and one member row from B. When a member row from set A and a member row from set B satisfying the conditions specified in predicate set Z are found, the members of the resulting set are generated. The resulting member row is an ordered member that includes the member columns from the member row of set B after the member columns of the member row from set A.

  Remark: The conditional predicate set Z specifies a condition that must be maintained between a member row of set A and a member row of set B.

  Requirement: Set A and Set B must be RDM sets. Set Z must be a conditional predicate set. The result when these conditions are not met is a null set. The predicate set Z must have a schema defined for the condition. The range of members in the predicate set specifies a member column of the set A, and the constituent elements of the members in the predicate set specify a member column from the set B.

Example:
A = {<"sales", "Tom">,
<"Sales", "Sam">,
<"Shipping", "Bill">,
<"Shipping", "Sally">}
B = {<"Bldg 1", "sales">,
<"Bldg 2", "shipping">}
Z = {{{"EQ". <″ 1 ″, ″ 2 ″>}}}

RDMJOIN (A, B, Z)-> {
<"Sales", "Tom", "Bldg 1", "sales">,
<"Sales", "Sam", "Bldg 1", "sales">,
<"Shipping", "Bill", "Bldg 2", "shipping">,
<"Shipping", "Sally", "Bldg 2", "shipping">}

RDMPIVOT function. The RDMPIVOT function defines an ordered set that exchanges member columns and member rows of a specified set. The following is an example format and description of this function.
RDMPIVOT (A) == <>

argument:
A-ordered set

  Result: The resulting set includes member rows composed of the member columns of set A. The sets are ordered by the order of member columns in set A.

  Note: Rotating very large sets can be costly and time consuming and should only be done if no other means of processing the set can be found.

  Requirement: Set A must be an RDM set. The result when these conditions are not met is a null set.

Example:
A = {<"3", "Tom", "a">,
<″ 2 ″, ″ Sam ″, ″ c ″>,
<"6", "Harry", "a">,
<″ 7 ″, ″ Harry ″, ″ a ″>}

RDMPIVOT (A)-><
<″ 3 ″, ″ 2 ″, ″ 6 ″, ″ 7 ″>,
<"Tom", "Sam", "Harry", "Harry">,
<″ A ″, ″ c ″, ″ a ″, ″ a ″ >>

RDMPROJ function. RDMPROJ defines an unordered set consisting of members from all member rows of the argument set, excluding the member columns specified through the conversion predicate set. The following is an example format and description of this function.
RDMPROJ (A, Z) == {}

argument:
A-Unordered set to be projected Z-Transformation predicate set for projection

  Result: The resulting set includes the member rows of each member row of A, except for the member columns specified by the transformation predicate set.

  Note: For information on how to specify the set Z as appropriate, see the specification for the conversion predicate set.

  Requirement: Set A must be an RDM set. Set Z must be a transformation predicate set. The result when these conditions are not met is a null set.

Example:
A = {<"3", "Tom", "a", "b", "s">,
<″ 2 ″, ″ Sam ″, ″ c ″, ″ b ″, ″ s ″>,
<"6", "Harry", "a", "z", "s">}
Z = <"3", "2">

RDMPROJ (A, Z)->{<"a","Tom">,
<″ C ″, ″ Sam ″>,
<″ A ″, ″ Harry ″>}

  RDMREST function. The RDMREST function defines an unordered set in which member rows are restricted to those that satisfy the conditions specified in the conditional predicate set. The following is an example format and description of this function.

    RDMREST (A, Z) == {}

argument:
A-Unordered set to be restricted Z-Conditional predicate set that specifies conditions for restriction

  Result: The resulting set includes only member rows from set A that satisfy the condition specified by conditional predicate set Z.

  Note: For information on how to specify the set Z, see the specification of the conditional predicate set.

  Requirement: Set A must meet the requirements of the RDM set. Set Z must be a conditional predicate set. If these conditions are not met, the result is a null set.

Example:
A = {<"3", "Tom", "a", "b", "s">,
<″ 2 ″, ″ Sam ″, ″ c ″, ″ f ″, ″ s ″>,
<"6", "Harry", "a", "z", "s">}
Z = {{{"EQ". <“2”, “const”. "Tom">}},
{{"EQ". <“2”, “const”. "Harry">}, {"EQ". <“4”, “const”. "F">}}}

RDMREST (A, Z)->{<"3","Tom","a","b","s">}

RDMSORT function. RDMSORT defines an ordered set based on the order specified by the unordered set A and the predicate set Z. The following is an example format and description of this function.
RDMSORT (A, Z) == <>

argument:
A-unordered set Z-mapping set describing the sort order of the result set

  Result: ordered set containing all member rows of set A sorted by the order specified in mapping set Z

  Remarks: Z is a mapping set that includes a range of member columns that specify the highest order member to the lowest order member that determines the sort order in ascending order.

  Requirement: The predicate set Z must be an ordered set whose elements are each NAT members less than the concentration of set A. Set A must be an RDM set. If these conditions are not met, the result is a null set.

Example:
A = {<"3", "Tom", a "," b "," s ">,
<″ 2 ″, ″ Sam ″, ″ c ″, ″ b ″, ″ s ″>,
<"6", "Harry", "a", "z", "s">}
Z = <"3", "2">

RDMSORT (A, Z)-><<"6","Harry","a","z","s">,
<"3", "Tom", "a", "b", "s">,
<"2", "Sam", "c", "b", "s">>

RDMUNION function. RDMUNION defines an unordered set that includes all member rows of sets A and B. The following is an example format and description of this function.
RDMUNION (A, B) == {}

argument:
A-Unordered set B-Unordered set

  Result: unordered set containing member rows of A or B

  Remarks: None

  Requirement: A and B must be RDM sets and have the same member column concentration. If these conditions are not met, the result is a null set.

Example:
A = {<″ a ″, ″ b ″, ″ c ″>}
B = {<"3", "c", "8">}

RDMUNION (A, B)->{<"a","b","c">,
<″ 3 ″, ″ c ″, ″ 8 ″>}

  The above functions, formats, and arguments are merely examples and may be different in other embodiments. For example, in other embodiments, different or additional functions may be used.

  While preferred embodiments of the present invention have been illustrated and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. From now on, those skilled in the art will be able to conceive many variations, modifications and substitutions without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be utilized in practicing the invention. It is intended that the following claims define the scope of the invention and that methods and structures that fall within the scope of these claims and their equivalents be covered by the claims.

Claims (145)

A method for establishing an algebraic relationship from a query language statement comprising:
Providing a relationship store for storing algebraic relationships between datasets;
Receiving query language statements that reference multiple datasets;
Assembling a plurality of algebraic relationships that reference the data set specified in the query language statement;
Storing the plurality of algebraic relationships in the relationship store;
Including methods. Defining at least one new data set by assembling an algebraic relationship that references at least one of the data sets specified in the query language statement;
Storing the algebraic relationship of the new data set in the relationship store;
The method of claim 1, further comprising: The data set specified in the query language statement comprises at least one explicit data set;
Providing a data set information store for storing information about the data set;
Associating a dataset identifier with the explicit dataset;
Storing the data set identifier in the data set information store;
The method of claim 1, further comprising:   A data store for storing at least some of the data sets, wherein at least one of the data sets specified in the query language statement is stored in the data store upon receipt of the query language statement; The method of claim 1, which is not stored.   Further comprising providing a data set information store for storing information about the data set, wherein temporal information indicating a time at which information about the data set was added to the data set information store is within the data set information store. The method of claim 1, associated with each data set.   Further comprising providing a data set information store that stores information about the data set, the data set information store including a unique identifier and an external identifier associated with each data set in the data information store; The method of claim 1, wherein an external identifier is the same for at least some of the data sets.   The data set information store includes a unique identifier and an external identifier associated with each data set in the data information store, the external identifier being the same for at least some of the data sets. The method of claim 5.   6. The method of claim 5, further comprising redefining the data set information store in time by removing a data set associated with temporal information prior to a specified time from the data set information store. The method described.   9. The method of claim 8, further comprising removing an algebraic relationship that references the data set removed from the data set information store from the relationship store.   A data store for storing at least some of the data sets, wherein the data set information store is defined by an algebraic relationship in the relationship store and is not stored in the data store; The method of claim 1, comprising information about the set.   Further comprising a data store for storing at least some of the data sets, wherein the data set information store is not stored in the data set store and has temporal information prior to the specified time. The step of redefining the data set information store in time, including information about at least one unrealized data set defined by an algebraic relationship in the relation store that references the data set, is specified. 10. The method further comprises storing the unrealized data set in the data set store prior to removing the data set having temporal information prior to a previous time from the data set information store. The method described in 1.   The method of claim 1, wherein the plurality of algebraic relationships assembled from the query language statement have a single operator and a number of operands in the range of 1-3.   The method of claim 2, wherein the algebraic relationship of the new data set has a single operator and several operands in the range of 1-3.   The method of claim 1, wherein the query language statement is based on a relational data model.   The method of claim 1, wherein the query language statement is received in a structured query language (SQL) format.   The method of claim 1, wherein the query language statement is based on a markup language model.   The method of claim 1, wherein the query language statement is received in an XQuery format.   The query language statement requests a data set to be provided, and the method provides the request data set using at least some of the plurality of algebraic relationships assembled from the query language statement. The method of claim 1 further comprising:   The query language statement requests a dataset to be provided, the relationship store includes other algebraic relationships of the dataset not assembled from the query language statement, and the method includes the other algebraic relationships; The method of claim 1, further comprising providing the request data set using at least some of them.   The query language statement requests a data set to be provided, the relationship store includes other algebraic relationships of the data set not assembled from the query language statement, and the method includes the other algebraic relationships. The method of claim 18, further comprising providing the request data set using at least some of them. The query language statement requests a data set to be provided, and the method includes:
Providing a plurality of collections of algebraic relationships defining results equal to the request data set;
Applying an optimization criterion to select one of the collection of algebraic relationships;
Providing the request data set using the selected set of algebraic relationships;
The method of claim 1, further comprising: Providing a plurality of collections of algebraic relationships defining results equal to the request data set;
Applying an optimization criterion to select one of the collection of algebraic relationships;
Providing the request data set using the selected set of algebraic relationships;
21. The method of claim 20, further comprising:   The method of claim 21, further comprising storing each collection of algebraic relationships in a tree data structure.   The method of claim 21, further comprising assigning the algebraic relationship from the selected collection to processing resources for parallel processing.   The method of claim 21, wherein the optimization criteria includes a cost algorithm.   23. The method of claim 22, wherein the optimization criteria includes a cost algorithm, and the selected set of algebraic relationships has a minimum cost according to the cost algorithm.   The method of claim 21, wherein the optimization criteria is based at least in part on an estimate of the time required to retrieve from a storage device a data set required to calculate each collection of algebraic relationships.   The method of claim 21, wherein the optimization criteria is based at least in part on an amount of data to be retrieved from storage to calculate each collection of algebraic relationships.   23. The method of claim 22, wherein the optimization criteria is based at least in part on an estimate of a transfer rate of the data channel that retrieves the data set required to calculate each collection of algebraic relationships. The optimization criterion is based, at least in part, on the cost of retrieving the data set needed to compute each collection of algebraic relationships from storage.
22. The optimization criteria assigns a cost to retrieve an individual data set from storage only once if the individual data set is referenced more than once in a collection of algebraic relationships. The method described. Providing the collection of a plurality of algebraic relationships comprises:
Retrieving an algebraic relationship defining the request data set from the relationship store;
Generating a collection of algebraic relationships that are algebraically equivalent to the retrieved algebraic relationships;
The method of claim 21, comprising:   22. The method of claim 21, wherein the plurality of algebraic relation collections further comprises at least two algebraic relation collections that are not algebraically equivalent to each other but both provide a result equal to the requested data set. .   22. The providing a plurality of algebraic relation collections includes generating a collection of algebraic relations that distinguish between equivalent data sets that differ in physical format but logically contain the same data. the method of.   34. The method of claim 33, wherein the different physical formats are selected from the group consisting of a comma separated value format, a tab separated value format, a binary string format, a fixed offset format, and a markup language format.   22. The providing a plurality of algebraic relation collections includes generating a collection of algebraic relations that distinguish between equivalent data sets that differ in storage location but logically contain the same data. the method of.   36. The method of claim 35, wherein at least some of the equivalent data sets are available via different data channels. The collection of algebraic relationships refers to at least two equivalent data sets with different physical formats but logically the same data;
The step of applying the optimization criterion includes estimating a transfer time required to acquire the equivalent data set from a storage device, and a data set having a minimum estimated transfer time from the equivalent data set. The method of claim 21, further comprising: selecting The collection of algebraic relationships refers to at least two equivalent data sets containing different data but logically the same data;
The step of applying the optimization criterion includes estimating a transfer time required to acquire the equivalent data set from a storage device, and a data set having a minimum estimated transfer time from the equivalent data set. The method of claim 21, further comprising: selecting Providing a plurality of functions for calculating algebraic relationships, including at least two algebraically equivalent functions operating on data sets of different physical formats;
Providing the request data set using at least some of the functions;
The method of claim 21, further comprising: Providing a plurality of functions for calculating algebraic relationships, including at least two algebraically equivalent functions operating on data sets of different physical formats;
Selecting one of the algebraically equivalent functions based on the format of the selected equivalent data set to be retrieved from storage;
Implementing the request data set using at least some of the functions, including the selected function from the algebraically equivalent function;
38. The method of claim 37, further comprising: A method for providing a request data set comprising:
Receiving a plurality of query language statements not requesting the request data set;
Assembling a plurality of algebraic relationships in the data set based on the query language statement;
Providing the request data set based at least in part on the plurality of algebraic relationships assembled from the query language statement;
Including methods. The query language statement specifies at least one explicit data set, the method comprising:
Providing a data set information store for storing information about the data set;
Associating a dataset identifier with the explicit dataset;
Storing the data set identifier in the data set information store;
42. The method of claim 41, further comprising:   Further comprising providing a data store that stores at least some of the data sets, wherein the query language statement is at least one data that was not stored in the data store upon receipt of a request for the request data set. 42. The method of claim 41, wherein the set is specified.   Calculating the request data set using at least some of the algebraic relationships assembled from the query language statement was not stored in the data store upon receipt of a request for the request data set 44. The method of claim 43, further comprising using at least one data set.   42. The method of claim 41, wherein each algebraic relationship has a single operator and several operands in the range of 1-3.   42. The method of claim 41, wherein the query language statement is based on a relational data model.   42. The method of claim 41, wherein the query language statement is based on a markup language model.   42. The method of claim 41, wherein at least one of the query language statements is based on a relational data model and at least one of the query language statements is based on a markup language model. A method for providing a request data set comprising:
Providing a relationship store containing multiple algebraic relationships assembled from multiple query language statements;
Assembling a collection of algebraic relationships that define a result equal to the requested data set using at least some of the algebraic relationships from the relationship store;
Computing the request data set from the collection of algebraic relationships;
Including methods. Receiving a query language statement requesting the request data set;
Assembling a plurality of additional algebraic relationships based on the query language statement requesting the request data set;
Assembling the collection of algebraic relationships using at least some of the additional algebraic relationships;
50. The method of claim 49, further comprising:   Defining at least one new data set by assembling an algebraic relationship based on the portion of the data set referenced in the query language statement; and transferring the algebraic relationship of the new data set to the relation store. 51. The method of claim 50, further comprising storing. A method for providing a request data set comprising:
Providing a relationship store containing multiple algebraic relationships assembled from multiple query language statements;
Assembling a plurality of collections of algebraic relationships that define a result equal to the requested data set using at least some of the algebraic relationships from the relationship store;
Applying an optimization criterion to select one of the collection of algebraic relationships;
Calculating the request data set using the selected set of algebraic relationships;
Including methods.   53. The method of claim 52, further comprising storing each collection of algebraic relationships in a tree data structure.   53. The method of claim 52, wherein the optimization criteria is based at least in part on an estimate of the time required to retrieve from a storage device a data set required to calculate each collection of algebraic relationships. The optimization criterion is based, at least in part, on the cost of retrieving the data set needed to compute each collection of algebraic relationships from storage.
53. The optimization criteria allocates a cost for retrieving an individual data set from storage only once if the individual data set is referenced more than once in a collection of algebraic relationships. The method described.   53. The method of claim 52, further comprising assigning the algebraic relationship from the selected collection to processing resources for parallel processing.   53. Providing the plurality of algebraic relation collections includes generating a collection of algebraic relations that distinguish between equivalent data sets that differ in physical format but contain logically the same data. the method of. The collection of algebraic relationships refers to at least two equivalent data sets with different physical formats but logically the same data;
The step of applying the optimization criterion includes estimating a transfer time required to obtain the equivalent data set from a storage device, and a data set having a minimum estimated transfer time from the equivalent data set. 53. The method of claim 52, further comprising: selecting. Providing a plurality of functions, including at least two algebraically equivalent functions operating on data sets of different physical formats;
Providing the request data set using at least some of the functions;
57. The method of claim 56, further comprising: Providing a plurality of functions, including at least two algebraically equivalent functions operating on data sets of different physical formats;
Selecting one of the algebraically equivalent functions based on the format of a selected data set of the equivalent data sets;
Realizing the request data set using at least some of the functions including selected functions of the algebraically equivalent functions;
59. The method of claim 58, further comprising: A method for providing a request data set, comprising:
Providing multiple algebraic relationships that define relationships between multiple datasets;
Assembling a collection of algebraic relationships defining at least some of the plurality of algebraic relationships defining a result equal to the requested data set;
A respective cost of each of the plurality of algebraic relations, and at least in part, a transfer time required to retrieve a data set required to calculate the requested data set from the collection of algebraic relations Determining a cost based on an estimate of
Selecting the collection of algebraic relationships having the lowest cost;
Providing the request data set using the selected set of algebraic relationships;
Including methods.   Assembling the collection of algebraic relationships was not previously available when the requested data set was first requested using at least some of the algebraic relationships. 62. The method of claim 61, further comprising generating a new algebraic relationship.   Assembling the collection of a plurality of algebraic relationships includes assembling at least one algebraic relationship defining the request data set and algebraically equivalent to the at least one algebraic relationship defining the request data set. 62. The method of claim 61, further comprising: generating a collection of a plurality of algebraic relationships that are   62. The method of claim 61, wherein the plurality of algebraic relation collections further comprises at least two algebraic relation collections that are not algebraically equivalent to each other but both provide a result equal to the requested data set. .   62. The method of claim 61, further comprising storing each collection of algebraic relationships in a tree data structure.   62. The method of claim 61, further comprising assigning the algebraic relationship from the selected collection to processing resources for parallel processing.   62. The step of determining the cost further comprises assigning the cost of retrieving an individual data set only once if the individual data set is referenced more than once in a collection of algebraic relationships. The method described in 1.   62. Assembling the plurality of algebraic relation sets further comprises generating a set of algebraic relations that distinguish between equivalent data sets that have different physical formats but that logically contain the same data. the method of.   69. The method of claim 68, wherein the different physical formats are selected from the group consisting of a comma separated value format, a tab separated value format, a binary string format, a fixed offset format, and a markup language format.   62. Assembling the plurality of algebraic relation collections further comprises generating a collection of algebraic relations that distinguish between equivalent data sets that differ in storage location but contain logically the same data. the method of.   71. The method of claim 70, wherein at least some of the equivalent data sets are available over different data channels. The collection of algebraic relationships refers to at least two equivalent data sets with different physical formats but logically the same data;
Determining the cost further comprises estimating a transfer time required to obtain the equivalent data set from a storage device;
62. Selecting the algebraic relation collection having the lowest cost further comprises selecting the algebraic relation collection referencing the equivalent data having a minimum estimated transfer time. the method of. The collection of algebraic relationships refers to at least two equivalent data sets containing different data but logically the same data;
Determining the cost further comprises estimating a transfer time required to obtain the equivalent data set from a storage device;
62. Selecting the algebraic relation collection having the lowest cost further comprises selecting the algebraic relation collection referencing the equivalent data set having a minimum estimated transfer time. The method described. Providing a plurality of functions for calculating algebraic relationships, including at least two algebraically equivalent functions operating on data sets of different physical formats;
Providing the request data set using at least some of the functions;
62. The method of claim 61, further comprising: Providing a plurality of functions for calculating algebraic relationships, including at least two algebraically equivalent functions operating on data sets of different physical formats;
Selecting a function from among the algebraically equivalent functions based at least in part on the format of the data set referenced in the selected set of algebraic relationships;
Implementing the requested data set using at least some of the functions, including a function selected from the algebraically equivalent functions;
69. The method of claim 68, further comprising:   The equivalent data set includes a first data set in a first format and a second data set in a second format having a size larger than the first format, the algebraic having the lowest cost. The step of selecting a collection of relationships refers to the second data set when the second data set is available via a data channel having a higher transfer rate than the first data set. 69. The method of claim 68, further comprising selecting the collection of data.   The equivalent data set is a first data set in a non-volatile storage device in a first format and a second format in a volatile memory in a second format having a size larger than the first format. 69. Selecting the set of algebraic relationships that includes a data set and has the lowest cost further comprises selecting the set of algebraic relationships that reference the second data set. the method of. A method for storing a specified data set, comprising:
Providing multiple algebraic relationships that define relationships between multiple datasets;
Assembling a collection of algebraic relationships that define a result equal to the specified data set using at least some of the plurality of algebraic relationships;
A respective cost of each of the plurality of algebraic relations, and at least in part, a transfer time required to retrieve a data set required to calculate the requested data set from the collection of algebraic relations Determining a cost based on an estimate of
Selecting the collection of algebraic relationships having the lowest cost;
Computing the specified data set using the selected set of algebraic relationships;
Storing the specified data set calculated from the selected set of algebraic relationships;
Including methods.   79. The method of claim 78, wherein assembling the plurality of algebraic relation collections further comprises generating new algebraic relations that were not previously available when the designated data set was first designated. .   Assembling the collection of a plurality of algebraic relationships includes assembling at least one algebraic relationship defining the specified data set and algebraically equivalent to the at least one algebraic relationship defining the specified data set. 79. The method of claim 78, further comprising: generating a collection of algebraic relationships that are A method for providing a request data set, comprising:
Providing a storage system for storing data sets;
A first algebraic relationship based on a set of first data sets stored in the storage system and at least two alternative algebraic relationships each defining an equal result to the requested data set; and the storage system Assembling an algebraic relationship comprising a second algebraic relationship based on a collection of second data sets stored in
An estimate of the transfer time required to retrieve the first set of data sets from the storage system to calculate the requested data set using the first algebraic relationship, at least in part, and Based on an estimate of the transfer time required to retrieve the second collection of data sets from the storage system to calculate the requested data set using the second algebraic relationship, Selecting an algebraic relationship or the second algebraic relationship;
Calculating the request data set using the selected algebraic relationship;
Including methods. A method for providing a request data set, comprising:
Providing a storage system for storing data sets;
At least two alternative algebraic relationships, each defining an equal result to the requested data set, the first algebraic relationship based on at least a first data set stored in the storage system and the storage system; Logically the same data that includes a second algebraic relationship based on at least a stored second data set, and wherein the first data set and the second data set are stored in different physical formats Assembling algebraic relationships, including
Selecting the first algebraic relationship or the second algebraic relationship based at least in part on the physical format of the first data set and the physical format of the second data set. When,
Calculating the request data set using the selected algebraic relationship;
Including methods.   83. The method of claim 82, wherein the physical format of the first data set is selected from the group consisting of a comma separated value format, a tab separated value format, a binary string format, a fixed offset format, and a markup language format. .   84. The method of claim 83, wherein the physical format of the second data set is selected from the group consisting of a comma separated value format, a tab separated value format, a binary string format, a fixed offset format, and a markup language format. . A first function operating on the physical format data set of the first data set and a second function operating on the physical format data set of the second data set; and Providing a plurality of functions for calculating an algebraic relationship, wherein the first function and the second function are algebraically equivalent;
Using the selected algebraic relationship to calculate the request data set uses the first function when the first algebraic relationship is selected and uses the second algebra. 83. The method of claim 82, further comprising using the second function when a general relationship is selected.   83. The method of claim 82, wherein the storage system includes a plurality of data channels, and the first data set and the second data set are available via different data channels. The first data set of the first physical format has a first size and is available via a first data channel having a first data rate, and the second physical format The second data set has a second size and is available via a second data channel having a second data rate, the method comprising:
Determining a first cost based at least in part on the first size and the first data rate;
Determining a second cost based at least in part on the second size and the second data rate;
If the first cost is lower than the second cost, selecting the first algebraic relationship to calculate the request data set;
90. The method of claim 86, further comprising: A method for providing a request data set, comprising:
Providing a storage system for storing a data set, comprising a volatile memory and a non-volatile storage medium;
At least two alternative algebraic relationships, each defining an equal result to the requested data set, the first algebraic relationship based on at least a first data set stored in the storage system and the storage system; Including a second algebraic relationship based on at least a second stored data set, and wherein the first data set and the second data set include the same logical data stored in different physical formats And the first data set is stored in a first physical format having a first size, and the second data has a second size smaller than the first size. Assembling algebraic relationships, stored in the physical format of
Determining that the first data set is available in the volatile memory;
Selecting the first algebraic relationship to calculate the request data set;
Including methods. A method for providing a request data set, comprising:
Providing a storage system for storing data sets in multiple physical formats;
Providing a plurality of functions that operate on the data sets of the plurality of physical formats, each function configured to operate on at least one operand of each one of the physical formats; When,
Providing an algebraic relationship defining a result equal to the requested data set, wherein the algebraic relationship references at least a first data stored in the storage system;
Retrieving the first data set in each of the physical formats;
Selecting a function configured to act on an operand of each physical format of the first data set of the plurality of functions;
Calculating the request data set from algebraic relationships using the selected function;
Including methods.   90. The method of claim 89, wherein the algebraic relationship specifies at least one operation and the selected function corresponds to the specified operation.   90. The method of claim 89, wherein the plurality of functions include algebraically equivalent functions configured to operate on operands of different physical formats.   90. The method of claim 89, wherein providing the algebraic relationship further comprises selecting the algebraic relationship from a plurality of algebraic relationships that each define a result equal to the requested data set.   92. The method of claim 91, wherein providing the algebraic relationship further comprises selecting the algebraic relationship from a plurality of algebraic relationships that each define an equal result to the requested data set.   Selecting the algebraic relationship further comprises selecting the algebraic relationship based at least in part on an estimate of the time required to retrieve the first data set from the storage system; 94. The method of claim 93.   The algebraic relationship refers to at least a second data set stored in the storage system, and the method further comprises retrieving the second data set individually in the physical format. 90. The method according to 89.   96. The method of claim 95, wherein the selected function is configured to operate on at least a second operand of each physical format of the second data set.   96. The method of claim 95, wherein the algebraic relationship specifies at least one operation and the selected function corresponds to the specified operation.   96. The method of claim 95, wherein the plurality of functions include algebraically equivalent functions configured to operate on operands of different physical formats.   96. The method of claim 95, wherein providing the algebraic relationship further comprises selecting the algebraic relationship from a plurality of algebraic relationships that each define an equal result to the requested data set.   Selecting the algebraic relationship further comprises selecting the algebraic relationship based at least in part on an estimate of the time required to retrieve the second data set from the storage system; 100. The method of claim 99.   99. The method of claim 98, wherein providing the algebraic relationship further comprises selecting the algebraic relationship from a plurality of algebraic relationships that each define a result equal to the requested data set. A method for providing a request data set, comprising:
Providing a mapping between a first schema based on a first data model and a second schema based on a second data model;
Receiving a first statement based on the first schema;
Assembling a plurality of algebraic relationships between a plurality of data sets based on the first statement;
Receiving a second statement based on the second schema requesting the request data set;
Providing the request data set using at least one of the algebraic relationships assembled from the first statement and a mapping between the first schema and the second schema; When,
Including methods.   103. The method of claim 102, wherein the first data model is a relational data model.   104. The method of claim 103, wherein the second data model is a markup language data model.   104. The method of claim 102, wherein the first data model is a markup language data model.   106. The method of claim 105, wherein the second data model is a relational data model.   103. The method of claim 102, wherein the first statement is a structured query language (SQL) format query statement.   108. The method of claim 107, wherein the second statement is a query language statement in XQuery format.   103. The method of claim 102, wherein the first statement is a query language statement in XQuery format.   110. The method of claim 109, wherein the second statement is a structured query language (SQL) format query language statement.   103. The method of claim 102, wherein the first statement is a set notation statement.   112. The method of claim 111, wherein the second statement is a structured query language (SQL) format query statement.   108. The method of claim 107, wherein the second statement is a set notation statement.   112. The method of claim 111, wherein the second statement is a query language statement in XQuery format.   110. The method of claim 109, wherein the second statement is a set notation statement. A method for providing a request data set based on a first schema comprising:
Providing a mapping between a first schema based on a first data model and a second schema based on a second data model;
Providing a first plurality of algebraic relationships between a plurality of data sets based on the second schema;
Assembling an algebraic relationship defining a result equal to the requested data set based at least in part on the first plurality of algebraic relationships;
Providing the request data set using the algebraic relationship defining a result equal to the request data set and the mapping between the first schema and the second schema;
Including methods.   117. The method of claim 116, wherein the first plurality of algebraic relationships is assembled from a first plurality of statements based on the second schema.   Further comprising providing a second plurality of algebraic relationships between a plurality of data sets based on the first schema, wherein the algebraic relationship defining a result equal to the requested data set is at least partially 117. The method of claim 116, based on the second plurality of algebraic relationships.   119. The method of claim 118, wherein the first plurality of algebraic relationships is assembled from a first plurality of statements based on the second schema.   119. The method of claim 118, wherein the second plurality of algebraic relationships is assembled from a second plurality of statements based on the first schema.   120. The method of claim 119, wherein the second plurality of algebraic relationships is assembled from a second plurality of statements based on the first schema.   117. The method of claim 116, wherein the first data model is a relational data model.   The method of claim 122, wherein the second data model is a markup language data model.   117. The method of claim 116, wherein the first data model is a markup language data model.   129. The method of claim 124, wherein the second data model is a relational data model. A method for storing a dataset, comprising:
Providing a data store;
Providing a data set information store for storing information about a plurality of data sets, including information identifying whether each data set is implemented in the data store;
Providing a relationship store for storing a plurality of algebraic relationships between the data sets;
Establishing criteria for virtualizing datasets in the dataset information store;
Identifying at least one data set implemented in the data store and meeting the virtualization criteria;
Determining that the relationship store includes at least one algebraic relationship defining the identified dataset based on at least one other dataset implemented in the data store;
Removing the identified data set from the data store;
Changing the information about the identified data set in the data set information store to indicate that the identified data set is not realized in the data store;
Including methods.   127. The method of claim 126, wherein the virtualization criteria includes an individual size of the data set.   127. The method of claim 126, wherein the virtualization criteria includes the number of times the data set has been referenced.   127. The method of claim 126, wherein the virtualization criteria includes the frequency with which the individual data sets are accessed in the data store.   127. The method of claim 126, wherein the algebraic relationship defining the identified data set refers to at least two other data sets implemented in the data store.   127. The method of claim 126, wherein the at least one other data set used to define the identified data set comprises data that is different in physical format but logically the same.   132. The method of claim 131, wherein the physical format in which the identified data set is stored is larger than the physical format of the at least one other data set.   135. The physical format in which the identified data set is stored is selected from the group consisting of a comma separated value format, a tab separated value format, a binary string format, a fixed offset format, and a markup language format. The method described.   The physical format in which the at least one other data set is stored is selected from the group consisting of a comma separated value format, a tab separated value format, a binary string format, a fixed offset format, and a markup language format. 132. The method according to 132.   129. The method of claim 126, further comprising providing a requested data set without realizing the identified data set using the algebraic relationship defining the identified data set. Providing the requested data set using the algebraic relationship defining the identified data set comprises:
A set of algebraic relationships defining the requested data set, wherein the algebraic relationships defining the identified data are used to assemble at least one of the collection of algebraic relationships. Assembling a collection of the plurality of algebraic relationships;
Applying one or more optimization criteria to select one of the collection of algebraic relationships for calculating the request data set;
Calculating the request data set using the selected set of algebraic relationships;
136. The method of claim 135, further comprising:   137. The algebraic relationship defining the identified data is used as an alternative to a reference to the identified data set in at least one of the collection of algebraic relationships. Method. A method for storing a dataset, comprising:
Providing a data store;
Providing a data set information store for storing information about a plurality of data sets, including information identifying whether each data set is implemented in the data store;
Providing a relationship store for storing a plurality of algebraic relationships between the data sets;
Selecting at least one data set implemented in the data store from the data set information store;
Adding a data set that is a subset of the selected data set to the data set information store, and realizing the added data set in the data store;
Adding an algebraic relationship to the relationship store that defines the selected dataset based on the added dataset;
Removing the selected data set from the data store;
Changing the information about the selected data set in the data set information store to indicate that the selected data set is not implemented in the data store;
Including methods. Receiving a statement requesting a request data set including an expression that references the selected data set;
An algebraic relationship having a result equal to the requested data set, comprising an expression referring to at least one of the added subsets as an alternative to the expression referring to the selected data set Building a relationship,
Calculating the request data set using the algebraic relationship;
140. The method of claim 138, further comprising:   139. The method of claim 138, wherein the added data set is a portion of the selected data set that has the same concentration.   138. The method of claim 138, wherein the added data set is a subset defined by a scalar value range of data items included in the selected data set. A method for storing a dataset, comprising:
Providing a relationship store for storing algebraic relationships between datasets;
Providing a data store for storing a data set including at least a first data set, a second data set, and a third data set;
Assembling an algebraic relationship defining the first data set using at least the second data set and the third data set;
Adding the algebraic relationship to the relationship store;
Removing the first data set from the data store after the algebraic relation has been added to the relation store;
Including methods.   At least one of the second data set and the third data set is stored in the data store in a physical format different from the physical format in which the first data set is stored in the data store. 143. The method of claim 142.   145. The physical format in which the first data set is stored is selected from the group consisting of a comma separated value format, a tab separated value format, a binary string format, a fixed offset format, and a markup language format. The method described. A data set that stores information about a plurality of data sets, including information indicating that the first data set, the second data set, and the third data set are realized in the data store. Providing an information store;
Changing the information about the first data set in the data set information store to indicate that the first data set is not realized in the data store;
143. The method of claim 142, further comprising:

Images