WO2012012968A1 - 一种分布式并行数据库系统的数据分区方法 - Google Patents

一种分布式并行数据库系统的数据分区方法 Download PDF

Info

Publication number
WO2012012968A1
WO2012012968A1 PCT/CN2010/077565 CN2010077565W WO2012012968A1 WO 2012012968 A1 WO2012012968 A1 WO 2012012968A1 CN 2010077565 W CN2010077565 W CN 2010077565W WO 2012012968 A1 WO2012012968 A1 WO 2012012968A1
Authority
WO
WIPO (PCT)
Prior art keywords
node
data
record
dimension table
database system
Prior art date
Application number
PCT/CN2010/077565
Other languages
English (en)
French (fr)
Inventor
张卫平
张松波
刘为怀
Original Assignee
北京播思软件技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京播思软件技术有限公司 filed Critical 北京播思软件技术有限公司
Priority to US13/325,810 priority Critical patent/US20120109888A1/en
Publication of WO2012012968A1 publication Critical patent/WO2012012968A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning

Definitions

  • the invention relates to a distributed parallel database system, in particular to a data partitioning method of a distributed parallel database system.
  • DBMS database management system
  • SQL DDL standard data definition languages
  • definition contains data table (Table) or relationship (Relation), data structure, index, primary key (Primary Key) and foreign key (Foreign Key)
  • Database data management scheme Database Schema
  • the application operates according to the data provided by the DBMS (such as SQL DML), you can perform data operations such as insert, query, update, import and export.
  • Sensing Data Data sets such as Financial Transaction Data, Goods Data, and Sales Data (Data) Set.
  • These data sets may reach a massive scale of hundreds of terabytes (TeraBytes) or PBs (PetaBytes), and as time goes on and the business grows, the speed of data generation may also increase. Higher operational requirements for such massive data, such as query speed, are raised.
  • Multi-node shared-nothing cluster The (Shared-nothing Cluster) architecture has proven to have the advantage of managing large-scale data.
  • the shared-nothing multi-node distributed parallel database system architecture diagram is shown in Figure 1.
  • the front-end server implements a global partitioner (Partitioner), which will follow each rule according to certain rules (such as specific attributes according to each data table).
  • Partitioner partitioned or sharded, and the data is distributed on a plurality of different storage and processing nodes (node 1 to node N in the figure), and each Local database instance running on each node (Local Database Instance) to manage the data partitions or shards assigned to the node based on the partition; meanwhile, a global optimized querier running on the front-end server (Global Querier), analyzes the specific query (Query) initiated by the application, and sends (Dispatch) to each node database system instance, by the local querier on each node (Local Querier) to process, and then return the result to the global querier for further processing, such as merge (Merge) and sort (Sort) operations, and finally
  • the partitioner When the partitioner divides each data table, it uses round robin (Round Robin). Partitioning), Hash Partitioning, Range Partitioning, and List Partitioning Partitioning) and other partitioning methods to send data to the corresponding node. Since the partitioning method employed acts on each data table separately, for a more complex relational query for multiple data tables, especially for a query involving a join operation between multiple tables, the global querier judges according to the Join query. The partition information of any table involved in the (Predicate), when the query is sent to the local querier on the node corresponding to each partition, for each table involved in the Join judgment, each node must be from other nodes. Partition copy handling data. Inter-node data handling during such queries is also called dynamic re-partitioning (Dynamic) Repartitioning) not only consumes network bandwidth, but also generates transmission time, greatly increases the response time of the query and affects the query efficiency.
  • Round Robin Round Robin
  • Partitioning Hash Partitioning
  • the present invention aims to provide a data partitioning method for a distributed parallel database system, which eliminates copying and handling of data between nodes during query, and improves query response speed and efficiency.
  • the present invention provides a data partitioning method for a distributed parallel database system, the method comprising the following steps:
  • each node is Data, with partial completeness of the data.
  • the primary-foreign key constraint condition since the data of each node has local completeness for such a query, there is no need to dynamically re-partition the data between nodes, thereby avoiding the time-consuming network transmission of data. Reduce query response time and improve query efficiency.
  • FIG. 1 is a structural diagram of a shared-nothing multi-node distributed parallel database system in the prior art
  • FIG. 2 is a flow chart of a data partitioning method of a distributed parallel database system in accordance with the present invention
  • Figure 3 is a fact table and dimension table association diagram in accordance with the present invention.
  • Figure 5 is a data distribution diagram of a dimension table record insertion according to the present invention.
  • FIG. 6 is a schematic diagram showing the distribution of data after the fact table record is inserted according to the present invention.
  • FIG. 7 is a schematic diagram of initial values of a Bloom Filter bit array according to the present invention.
  • FIG. 8 is a schematic diagram of setting a bit array according to a hash function value of x according to the present invention.
  • Figure 9 is a schematic diagram of determining whether y belongs to a set in accordance with the present invention.
  • a Sales database or data warehouse can be designed in such a way that each sales record should include the products sold, the customers who sell, the suppliers of the products, the time of sale, the quantity sold, and the revenue earned.
  • the specific digital data such as the quantity and amount of sales is usually the object to be analyzed by the system. For time, products, customers, suppliers, we often want to obtain digital data from these different perspectives. A statistical result. Therefore, the digital data is generally stored in the fact table, and the time, product, customer, and supplier are stored in different dimension tables. Naturally, there is a primary-foreign key association between the dimension table and the fact table, and there is no relationship between the dimension tables.
  • the star model is the most important model for database systems or data warehouse modeling relationships and data.
  • snowflake models derived from the star model (Snowflake Schema).
  • the snowflake model is a model obtained by normalizing the dimension table on the basis of the star model. Since each dimension table normalization may result in a star topology or a multi-level star topology, making the entire model topology resemble a snowflake, it is called a snowflake model.
  • the snowflake model is more complex than the star model, and more tables need to be associated with the query.
  • FIG. 2 is a flow chart of a data partitioning method of a distributed parallel database system according to the present invention.
  • the data partitioning method of the distributed parallel database system of the present invention will be described in detail below with reference to FIG. 2:
  • a distributed parallel database system is constructed according to the nature of the data to be managed and the number of nodes.
  • the data table constructed includes data on the products sold, the customers who sell, the suppliers of the products, the time of sale, the quantity sold, and the income earned;
  • a fact table and a dimension table are created. Create a fact table for storing actual fact data, define the primary and foreign keys of the fact table, and insert a record of the fact data into the fact table, such as the quantity and the number of sales in the Sales database or data warehouse described above Revenue of such specific numeric data; creating a dimension table for storing data describing attributes from different angles, defining a primary key of the dimension table, and inserting a record describing the data of the attribute into the dimension table, describing The data of the attribute is the time, product, customer, supplier, and the like data in the Sales database or the data warehouse described above; and the fact table and the dimension table are associated by using the foreign key of the fact table and the primary key of the dimension table.
  • Table 3 is a fact table and a dimension table association diagram according to the present invention.
  • Table 1 and Table 2 are defined as fact tables
  • Table 3, Table 4, and Table 5 are defined as dimension tables.
  • Table1's foreign key Field11 is associated with Talbe3's primary key ID3
  • Table1's foreign key Field12 and Table2's foreign key Field21 are associated with Talbe4's primary key ID4
  • Table2's foreign key Field22 is associated with Talbe5's primary key ID5;
  • the dimension table Table 4 is a data table relationship diagram after being divided into a single star according to the present invention. As shown in FIG. 4, according to the fact table and the dimension table association diagram of FIG. 3, the dimension table Table 4 is divided into two logical tables to form 2 single star structures, the dimension table Table4 is still physically a table;
  • step 203 the fact table record and the dimension table record are inserted into the node.
  • the fact table record and the dimension table record are inserted into different nodes;
  • the dimension table record is copied. After the record of the fact table is inserted, in order to ensure the partial completeness of the data, the record of the dimension table associated with the record foreign key of the fact table is copied to the node. In this way, when the join between tables is generated, it is not necessary to carry data of other nodes, thereby reducing network overhead.
  • the method of determining to copy the record of the dimension table to the node of the fact table is: firstly, it is determined that the dimension table associated with the foreign key of the fact table needs to be copied; secondly, the foreign key in the newly inserted record is associated with The records in the dimension table need to be copied to the same node of the fact table record. For example, if the foreign key value of the record of the fact table is X, then the record with the primary key value of X in the dimension table needs to be copied to this node. If the record of the fact table has multiple foreign keys, you need to copy the records of the dimension table associated with each foreign key. Since the partition is generally based on the primary key of the table, it is easy to find out which node the required record in the dimension table is located based on the value of the foreign key of the fact table (that is, the primary key value of the dimension table).
  • FIG. 5 is a data distribution diagram of a dimension table record insertion according to the present invention. As shown in FIG. 5, taking the star types of Table 1, Table 3, and Table 4 in FIG. 4 as an example, after the dimension table (Table 3 and Table 4) records are inserted. The distribution of data on each node can be seen from Figure 5. Before the fact table record is inserted, the records of the dimension table are not overlapped on each node (Non-Overlap).
  • FIG. 6 is a schematic diagram showing the distribution of data after the fact table record insertion according to the present invention.
  • a record of Table1 is inserted in node 1, and Field1 (value 2) and Field12 (value 3) are associated with Table 3.
  • the dimension table record may overlap on different nodes, and the fact table records are non-overlap.
  • the node that divides a record according to the initial partitioning strategy is called the primary node of the record (Primary) Node), and the dimension table records the node that is used to maintain local completeness.
  • the past node is called the backup node of the record.
  • the system can quickly obtain the record of the foreign key association, because the related records have been stored in the same node, and the data need not be carried every time, thereby improving the query efficiency;
  • the front-end server For the query operation of the dimension table, the front-end server first sends the query request to each node, and each node obtains the record of the node, and then returns to the front-end server for aggregation. Since dimension table records may overlap on different nodes, the dimension table records received by the front-end server may be duplicated.
  • the solution to this problem can filter out duplicate records on the front-end server; on a single node, the master and backup nodes can be distinguished from the records, and the records of the backup nodes can be filtered out;
  • the data is deleted. Delete the record in the fact table. If the record in the fact table is deleted, if the record of the associated dimension table is no longer associated with other fact tables, you need to delete the associated dimension table record on the node (the record of the master node is not deleted). ); the deletion of records in the dimension table only needs to delete the records on the master node. Because the fact table record needs to be deleted before the dimension table record is deleted, and the dimension table record on that node has been deleted when the fact table record is deleted;
  • the data is updated.
  • the record in the fact table is updated, if the update of the foreign key is involved, it is necessary to delete the old dimension table record (the record of the master node and the record associated with other fact tables are not deleted), and then copy the new dimension table record; the dimension table
  • the record of the backup node needs to update the record of the backup node.
  • One way to update the dimension table record is to search the fact table of all nodes to see if there is a primary key in the fact table equal to the dimension table record to be updated, and if so, update the relevant record of the dimension table on the node.
  • This method requires traversing the fact table of all nodes and will take a long time; an optimized implementation of updating the dimension table records is to create a Bloom filter for each dimension table and each node (Bloom)
  • the Filter table records the distribution of the dimension table records on the nodes, so that it is easy to find the node that holds a specified record.
  • Bloom filter (Bloom) Filter) is a space-efficient random data structure that uses a bit array to represent a collection very succinctly and can determine whether an element belongs to the collection. Bloom The efficiency of Filter is costly: when judging whether an element belongs to a collection, it is possible to mistake elements that do not belong to this collection to belong to this collection (False Positive). Therefore, Bloom Filter is not suitable for those "zero error" applications. In applications that can tolerate low error rates, Bloom Filter exchanges a huge amount of savings in storage space with very few errors.
  • Figure 7 is a Bloom in accordance with the present invention.
  • the initial value of the Filter bit array is shown in Figure 7.
  • the Bloom Filter is an array of bits containing m bits, each of which is set to zero.
  • Bloom Filter uses k independent hash functions (Hash Function), which maps each element in the set to the range of ⁇ 1,...,m ⁇ .
  • Hash Function maps each element in the set to the range of ⁇ 1,...,m ⁇ .
  • the position hf(x) of the fth hash function map is set to 1 (1 ⁇ f ⁇ k). Note that if a location is set to 1 multiple times, then only the first time will work, and the next few will have no effect.
  • FIG. 9 is a schematic diagram of determining whether y belongs to a set according to the present invention. As shown in FIG. 9, in FIG. 9, y1 is not an element in the set, and y2 belongs to the set, or is just a False. Positive.
  • Bloom Filter introduces another factor in addition to the two factors of time space: error rate.
  • error rate Using Bloom When Filter determines whether an element belongs to a certain collection, there will be a certain error rate. In other words, it is possible to mistake elements that do not belong to this collection to belong to this collection (False Positive), but the elements belonging to this collection are not mistaken for belonging to this collection (False Negative). After increasing the error rate factor, Bloom Filter saves a lot of storage space by allowing a small number of errors.
  • the record distribution of each dimension table on each node is recorded in a Bloom.
  • the primary key of the dimension table is used as the query key of the Bloom Filter table.
  • Bloom Number of Filter Tables (number of dimension tables ⁇ number of nodes). If Bloom Filter has an error (False Positive), the result is an attempt to update the dimension table record on a node, but this record is not saved on this node. This kind of error does not affect the correctness and consistency of the data, it can be tolerated. And as long as the length of the hash algorithm and bit array is chosen properly, this error rate will be very low.
  • the Filter table can be stored on the front-end server as a global data set; it can also be distributed and stored on each node, and each node is responsible for recording the distribution of the dimension table records on the node. Due to Bloom The Filter table occupies a small amount of space. In the implementation, the memory can be preloaded to improve the query speed.
  • the data partitioning method of the present invention can be applied to a distributed database system involving a large number of query operations of the association table Join.
  • a distributed database system involving a large number of query operations of the association table Join.
  • users often need to classify according to the type of goods, sort according to prices, and the like.
  • the present invention we can define the product type and price in the fact table, and define some foreign key related dimension tables, such as sellers, manufacturers, and the like.
  • the fact table record is inserted, the associated dimension table record is copied to the same node.
  • the front-end server sends the query to each node, and each node can perform such a join operation without carrying data to other nodes. , greatly improve the efficiency of the query.
  • Each node returns its processing result to the global querier for summary.
  • sales data management we can define sales, profit value, etc. in the fact table, define the customer, sales time, etc. as the dimension table, and associate the fact table with the primary foreign key.
  • the fact table record When the fact table record is inserted, the associated dimension table record is copied to the same node.
  • the front-end server distributes the statistical work to each node.
  • Each node relies on the saved information to easily determine whether the fact table sales record belongs to the customer, because the customer information already exists on the node, so that the statistical work of the node can be easily completed, and finally sent to the front-end server for aggregation.

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种分布式并行数据库系统的数据分区方法，包括以下步骤：根据构建的分布式并行数据库系统，创建事实表和维度表；根据分区规则将维度表和事实表纪录插入到不同节点上；将维度表纪录复制到事实表的节点上；对数据进行删除和更新处理。本发明在对数据集或数据流分区导入或插入分布式数据库系统时，能在每一个节点，满足数据库方案所定义的表间关系，特别是主-外键约束条件，使每一个节点上的数据，具有数据的局部完备性。对于利用主-外键约束条件进行表间连接的查询处理，由于各节点的数据对这类查询具有局部完备性，不需要在节点间做数据动态再分区，避免了数据的网络传输耗时，降低查询响应时间，提高查询效率。

Description

一种分布式并行数据库系统的数据分区方法 技术领域
本发明涉及一种分布式并行数据库系统，尤其涉及一种分布式并行数据库系统的数据分区方法。
背景技术
将数据存储在数据库中是常用的数据管理方法，特别是存储在关系型数据库中。我们可以根据所要管理的数据需求，选择成熟的数据库管理系统(DBMS: Database Management System)，用标准的数据定义语言(如SQL DDL)，定义包含数据表（Table）或关系(Relation)、数据结构、索引、主键（Primary Key）和外键(Foreign Key)等信息的数据库数据管理方案(Database Schema)，部署数据库系统。而应用程序根据DBMS提供的数据操作语言(如SQL DML)，可以进行数据操作，如插入、查询、更新、导入和导出等。
当前许多行业应用，产生和累积的数据量非常巨大，例如物联网感知数据(Sensor Data) 、金融交易数据(Transaction Data)、电子商务商品数据（Goods Data）、公司销售数据（Sales Data）等数据集(Data Set)。这些数据集可能会达到几百TBs(TeraBytes)或PBs（PetaBytes）这样海量的规模，而且随着时间的增长和业务的发展，产生数据的速度也可能会不断提高。对这些海量数据的操作效率，如查询速度，提出了更高的要求。
对于海量数据的管理，单节点的数据库系统，受其计算或存储能力的局限，已经不能胜任。分布式并行结构或极大规模并行处理(MPP： Massively Parallel Processing）结构的数据库或数据仓库系统可以提供更好的容量和性能方面的伸缩性和扩展性。其中的多节点无共享集群 (Shared-nothing Cluster)架构已被证实具有管理大规模数据的优势。
无共享(Shared-nothing)多节点分布式并行数据库系统架构图如图1所示，前端服务器实现一个全局分区器(Partitioner)，它将各个数据表按照某种规则(如按各数据表特定属性域的HASH值或时间段等)进行分区(Partitioning)或分片(Sharding)，将数据分布存储在多个不同的存储和处理节点上（如图中的节点1~节点N），并由每个节点上运行的本地数据库实例（Local Database Instance）, 来管理根据分区器分配到该节点上的数据分区或分片；同时，一个运行在前端服务器上的全局优化查询器(Global Querier)，对应用发起的特定查询(Query)，进行分析，并发送(Dispatch)给各节点数据库系统实例，由各节点上的本地查询器(Local Querier)来处理，然后将结果返回给全局查询器，进行进一步的处理，如合并(Merge)和排序(Sort)等操作，最后将结果返回给相应的应用。
分区器在对各数据表进行划分时，采用诸如轮转划分(Round Robin Partitioning)、散列划分(Hash Partitioning)、范围划分(Range Partitioning)和链表划分（List Partitioning）等分区方法，将数据发送给相应的节点。由于采用的分区方法单独作用于各个数据表，因此，对于针对多个数据表的较复杂的关联查询时，特别是涉及多表间连接（Join）操作的查询，全局查询器无论根据Join查询判断式(Predicate)所涉及的任何一个表的分区信息，将查询发送给各分区所对应的节点上的局部查询器处理时，对于Join判断式所涉及的其他表，各节点都要从其他节点上的分区拷贝搬运数据。这种查询时的节点间数据搬运也称作动态再分区(Dynamic Repartitioning)，不仅会消耗网络带宽，也会产生传输耗时，极大地增加查询的响应时间，影响查询效率。
 技术解决方案
为了解决现有技术存在的不足，本发明的目的在于提供一种分布式并行数据库系统的数据分区方法，消除查询时节点间数据的拷贝和搬运，提高查询响应速度和效率。
为实现上述目的，本发明提供的一种分布式并行数据库系统的数据分区方法，该方法包括以下步骤：
根据构建的分布式并行数据库系统及分布规则，创建事实表和维度表，并将所述事实表纪录和维度表纪录插入到节点上；
将维度表纪录复制到事实表的节点上；
对数据进行删除和更新。
 有益效果
本发明在对数据集或数据流分区导入或插入分布式数据库系统时，能在每一个节点，满足数据库方案所定义的表间关系，特别是主-外键约束条件，使每一个节点上的数据，具有数据的局部完备性。对于利用主-外键约束条件进行表间连接的查询处理，由于各节点的数据对这类查询具有局部完备性，不需要在节点间做数据动态再分区，避免了数据的网络传输耗时，降低查询响应时间，提高查询效率。
本发明的其它特征和优点将在随后的说明书中阐述，并且，部分地从说明书中变得显而易见，或者通过实施本发明而了解。
 附图说明
附图用来提供对本发明的进一步理解，并且构成说明书的一部分，并与本发明的实施例一起，用于解释本发明，并不构成对本发明的限制。在附图中：
图1为现有技术中无共享多节点分布式并行数据库系统架构图；
图2为根据本发明的分布式并行数据库系统的数据分区方法流程图；
图3为根据本发明的事实表和维度表关联图；
图4为根据本发明的划分成单一星型后的数据表关系图；
图5为根据本发明的维度表纪录插入后数据分布图；
图6为根据本发明的事实表纪录插入后数据分布情况示意图；
图7为根据本发明的Bloom Filter位数组初始值示意图；
图8为根据本发明的根据x的哈希函数值设置位数组示意图；
图9为根据本发明的判断y是否属于集合示意图。
 本发明的实施方式
以下结合附图对本发明的优选实施例进行说明，应当理解，此处所描述的优选实施例仅用于说明和解释本发明，并不用于限定本发明。
在构建数据库系统或以分布式数据库为基础构建的数据仓库时，一般总是将实际的事实数据和用于描述属性的数据用不同的表分隔，实际的事实数据存放于一类被称为事实表(Fact table)的表中，而从不同角度来描述属性的数据则放到不同的维度表（Dimension table）中。比如，一个Sales数据库或数据仓库可以这样设计，每一笔销售记录，应该会包含销售的产品，销售的客户，产品的供货商，销售的时间，销售的数量和获得的收入等。对于销售的数量和金额这类具体的数字型的数据，通常是系统要分析的对象，而对于像时间，产品，客户，供货商，我们往往希望从这些不同的角度来得到数字型数据的一个统计结果。所以，一般将数字型的数据存放在事实表中，将时间、产品、客户、供货商存放在不同的维度表中。自然，在维度表和事实表之间存在一个主-外键的关联，各个维度表之间则没有关系。
以类似上述的方式来建模数据库系统关系和属性的方式，由于它将不同的数据表分为维度表和事实表，并以主-外键相关联，拓扑上，事实表处于中间的位置，维度表则绕事实表围成一圈，形似一颗星，所以被称数据库系统的星形模型(Star Schema)。事实表中除了区分每条纪录的外键（关联维度表的主键）外，就只有我们关心的数字型数据，所以事实表中的每条纪录，有个专门的术语称之为度量(Measurement)，因为我们利用数据库或数据仓库做统计分析的时候，这些数据就是统计分析的一个个基本单位，也就是度量值。我们知道，在数据库系统查询和分析中，一般的查询处理，总是基于对度量即事实表度量的分析和处理展开进行的，即在查询的判断式中，总是含有涉及事实表的判断式。
星型模型是数据库系统或数据仓库建模关系和数据的最主要的模型。另外，从星型模型中衍生出来主要有雪花模型(Snowflake Schema)。雪花模型就是在星形模型的基础上，对维度表做规范化后得到的模型。由于每个维度表规范化可能得到一个星形拓扑或多级的星形拓扑，使整个模型拓扑上形似雪花，所以称为雪花模型。雪花模型比起星模型就更加复杂，查询的时候也需要关联更多的表。
图2为根据本发明的分布式并行数据库系统的数据分区方法流程图，下面将参考图2，对本发明的分布式并行数据库系统的数据分区方法进行详细描述：
首先，在步骤201，根据所要管理的数据性质以及节点数，构建分布式并行数据库系统。例如，在销售数据库或数据仓库中，构建的数据表包含有销售的产品，销售的客户，产品的供货商，销售的时间，销售的数量和获得的收入等数据；
在步骤202，创建事实表和维度表。创建用于存放实际的事实数据的事实表，定义该事实表的主键和外键，并将事实数据的纪录插入到该事实表，该事实数据如上述Sales数据库或数据仓库中销售的数量和获得的收入这类具体的数字型的数据；创建用于存放从不同角度来描述属性的数据的维度表，定义该维度表的主键，并将描述属性的数据的纪录插入到该维度表中，描述属性的数据如上述Sales数据库或数据仓库中的时间、产品、客户、供货商等数据；利用事实表的外键与维度表的主键，对事实表和维度表进行关联。图3为根据本发明的事实表和维度表关联图，如图3所示，Table1和Table2定义为事实表，Table3、Table4和Table5定义为维度表。Table1的外键Field11关联Talbe3的主键ID3，Table1的外键Field12和Table2的外键Field21均关联Talbe4的主键ID4，Table2的外键Field22关联Talbe5的主键ID5；
图4为根据本发明的划分成单一星型后的数据表关系图，如图4所示，根据图3的事实表和维度表关联图，把维度表Table4划分成逻辑的2张表，形成2个单一的星型结构，维度表Table4在物理上仍然是一张表；
在步骤203，将事实表纪录和维度表纪录插入到节点上。在本步骤中是按照分区策略，将事实表纪录和维度表纪录插入到不同的节点上；
在步骤204，复制维度表纪录。事实表的纪录插入完成后，为了保证数据的局部完备性，将该事实表的纪录外键关联的维度表的纪录，复制到本节点。这样，表间连接（Join）生成连接表的时候，不需要搬运其他节点的数据，减少网络开销。
确定将维度表的纪录复制到事实表的节点上的方法是：首先要确定的是，事实表的外键所关联的维度表才需要复制；其次，该新插入纪录中的外键所关联的维度表中的纪录，需要复制到该事实表纪录的同一个节点上。例如，事实表的纪录的外键值为X，那么需要将维度表中主键值为X的纪录复制到本节点。如果事实表的纪录有多个外键，需要将每个外键关联的维度表的纪录复制过来。由于分区一般是以表的主键作为关键字，所以根据事实表外键的值（也就是维度表主键值），能够很容易找到维度表中需要的纪录位于哪个节点上。
图5为根据本发明的维度表纪录插入后数据分布图，如图5所示，以图4中的Table1、Table3和Table4这一星型为例，在维度表（Table3和Table4）纪录插入后，各节点上的数据分布情况，从图5可以看出，在事实表纪录插入之前，维度表的纪录在各节点上是不重叠的（Non-Overlap）。
图6为根据本发明的事实表纪录插入后数据分布情况示意图，如图6所示，在节点1插入一条Table1的纪录，其Field11（值为2）和Field12(值为3)所关联的Table3和Table4的纪录（分别为ID3＝2和ID4＝3）在节点1上不存在，所以需要分别从节点2和节点3复制过来；
在节点2插入一条Table1的纪录，其Field11（值为2）所关联的Table3的纪录（ID3＝2）在节点2上已经存在，不需要复制。而Field12(值为1)所关联的Table4的纪录（ID4＝1）在节点2上不存在，所以需要从节点1复制过来；
在节点3插入一条Table1的纪录，其Field11（值为3）和Field12(值为3)所关联的Table3和Table4的纪录（分别为ID3＝3和ID4＝3）在节点3上都已经存在，所以不需要复制。
我们可以看出，在事实表纪录插入后，维度表纪录可能在不同节点上产生重叠（Overlap），而事实表纪录是不重叠的（Non-Overlap）。我们把某个纪录按照初始分区策略划分的节点称为该纪录的主节点（Primary Node），而维度表纪录为保持局部完备性复制过去的节点称为该纪录的备份节点（Backup Node）。
上述方法，对于大量涉及到Join的查询操作，系统能够快速获取到外键关联的纪录，因为在同一节点已存储了这些关联的纪录，不需要每次都进行数据搬运，从而提高查询效率；
对于维度表的查询操作，先由前端服务器将查询请求发送到每个节点，每个节点获取本节点的纪录，然后返回给前端服务器进行汇总。由于维度表纪录可能在不同节点上产生重叠，所以前端服务器收到的维度表纪录可能会重复。解决这个问题的方法可以在前端服务器上过滤掉重复的纪录；也可以在单个节点上，对纪录区分主节点和备份节点，过滤掉备份节点的纪录；
在步骤205，数据的删除处理。删除事实表中的纪录，在事实表中的纪录被删除后，如果关联的维度表的纪录不再被其他事实表关联，则需要删除本节点上关联的维度表纪录（主节点的纪录不删除）；维度表中纪录的删除，只需要删除主节点上的纪录。因为删除维度表纪录之前，需要先删除事实表纪录，而在删除事实表纪录的时候，已经删除那个节点上维度表纪录；
在步骤206，数据的更新处理。事实表中纪录更新后，如果涉及到外键的更新，需要先删除旧的维度表纪录（主节点的纪录以及被其他事实表关联的纪录不删除），再复制新的维度表纪录；维度表中纪录的更新，除了要更新主节点的纪录外，还需要更新备份节点的纪录。更新维度表纪录的一种实现方法是搜索所有节点的事实表，查看事实表中是否存在外键等于要更新的维度表纪录的主键，如果存在，则更新该节点上维度表的相关纪录。这种方式需要遍历所有节点的事实表，将消耗较长的时间；更新维度表纪录的一种优化的实现方法是针对每个维度表和每个节点，建立一个布隆过滤器（Bloom Filter）表，记录维度表纪录在节点上的分布情况，从而轻易找到保存某条指定纪录的节点。
布隆过滤器（Bloom Filter）是一种空间效率很高的随机数据结构，它利用位数组很简洁地表示一个集合，并能判断一个元素是否属于这个集合。Bloom Filter的这种高效是有一定代价的：在判断一个元素是否属于某个集合时，有可能会把不属于这个集合的元素误认为属于这个集合（False Positive）。因此，Bloom Filter不适合那些“零错误”的应用场合。而在能容忍低错误率的应用场合下，Bloom Filter通过极少的错误换取了存储空间的极大节省。
下面我们具体来看Bloom Filter是如何用位数组表示集合的。图7为根据本发明的Bloom Filter位数组初始值示意图，如图7所示，初始状态时，Bloom Filter是一个包含m位的位数组，每一位都置为0。
为了表达S={x1, x2,…,xn}这样一个n个元素的集合，Bloom Filter使用k个相互独立的哈希函数（Hash Function），它们分别将集合中的每个元素映射到{1,…,m}的范围中。对任意一个元素x，第f个哈希函数映射的位置hf(x)就会被置为1（1≦f≦k）。注意，如果一个位置多次被置为1，那么只有第一次会起作用，后面几次将没有任何效果。图8为根据本发明的根据x的哈希函数值设置位数组示意图，如图8所示，在图8中，k=3，且有两个哈希函数选中同一个位置（从左边数第七位）。
在判断y是否属于这个集合时，我们对y应用k次哈希函数，如果所有hf(y)的位置都是1（1≦f≦k），那么我们就认为y是集合中的元素，否则就认为y不是集合中的元素。图9为根据本发明的判断y是否属于集合示意图，如图9所示，在图9中y1就不是集合中的元素，而y2要么属于这个集合，要么刚好是一个False Positive。
在计算机科学中，我们常常会碰到时间换空间或者空间换时间的情况，即为了达到某一个方面的最优而牺牲另一个方面。Bloom Filter在时间空间这两个因素之外又引入了另一个因素：错误率。在使用Bloom Filter判断一个元素是否属于某个集合时，会有一定的错误率。也就是说，有可能把不属于这个集合的元素误认为属于这个集合（False Positive），但不会把属于这个集合的元素误认为不属于这个集合（False Negative）。在增加了错误率这个因素之后，Bloom Filter通过允许少量的错误来节省大量的存储空间。
本发明中，将每张维度表在每个节点上的纪录分布情况记录在一个Bloom Filter表中，维度表的主键（Primary Key）作为Bloom Filter表的查询关键字，Bloom Filter表数量＝（维度表数量×节点数量）。如果Bloom Filter发生错误（False Positive），产生的后果是试图更新一个节点上维度表纪录，但是这个节点上却没有保存这条纪录。这种错误不会影响数据的正确性和一致性，它是可以被容忍的。而且只要哈希算法和位数组的长度选择得当，这种错误率将非常低。
这些Bloom Filter表可以存储在前端服务器上，作为一个全局数据集；也可以分布存储在每个节点上，各节点负责记录本节点上维度表纪录的分布情况。由于Bloom Filter表占用的空间很小，在实现中，可以预先载入内存，以提高查询速度。
本发明的数据分区方法可以应用于涉及到大量关联表Join的查询操作的分布式数据库系统，例如在商品数据管理中，用户往往需要根据商品种类进行分类，根据价格进行排序等。运用本发明，我们可以将商品种类和价格定义在事实表中，另外定义一些外键关联维度表，如卖家，生产厂商等。事实表纪录插入的时候，将关联的维度表纪录复制到同一节点。在进行种类/价格/卖家/生产厂商等关联表的连接查询（Join）的时候，前端服务器把查询发送给每个节点，每个节点就可以进行这种Join操作，不需要到其他节点搬运数据，大大提高查询效率。各节点把各自的处理结果返回给全局查询器做汇总就可以了。
而在销售数据管理中，我们可以将销售额、利润值等定义在事实表中，将客户、销售时间等定义为维度表，并以主外键关联事实表。事实表纪录插入的时候，将关联的维度表纪录复制到同一节点。在对某一客户的销售额进行统计的时候，由前端服务器将统计工作分发到各节点。每个节点依靠所保存的信息，可以轻易判断事实表销售纪录是否属于该客户，因为本节点上已经存在该客户信息，从而可以很轻松地完成本节点的统计工作，最后发送给前端服务器汇总。
本领域普通技术人员可以理解：以上所述仅为本发明的优选实施例而已，并不用于限制本发明，尽管参照前述实施例对本发明进行了详细的说明，对于本领域的技术人员来说，其依然可以对前述各实施例记载的技术方案进行修改，或者对其中部分技术特征进行等同替换。凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。

Claims (10)

  1. 一种分布式并行数据库系统的数据分区方法，该方法包括以下步骤：
    根据构建的分布式并行数据库系统及分布规则，创建事实表和维度表，并将所述事实表纪录和维度表纪录插入到节点上；
    将维度表纪录复制到事实表的节点上；
    对数据进行删除和更新处理。
  2. 根据权利要求1所述的分布式并行数据库系统的数据分区方法，其特征在于，所述事实表包含主键、外键和事实表纪录。
  3. 根据权利要求1所述的分布式并行数据库系统的数据分区方法，其特征在于，所述维度表包含主键和维度表纪录。
  4. 根据权利要求1所述的分布式并行数据库系统的数据分区方法，其特征在于，所述事实表和维度表是通过主-键外键进行关联，事实表的外键值与其关联的维度表的主键值相等。
  5. 根据权利要求1所述的分布式并行数据库系统的数据分区方法，其特征在于，所述将事实表纪录和维度表纪录插入到节点上是将所述事实表纪录和维度表纪录插入到不同节点上。
  6. 根据权利要求1所述的分布式并行数据库系统的数据分区方法，其特征在于，所述将维度表纪录复制到事实表的节点上的步骤进一步包括：
    根据事实表的外键确定关联的维度表；
    将事实表外键所关联的维度表中的纪录，复制到所述事实表的节点上。
  7. 根据权利要求1所述的分布式并行数据库系统的数据分区方法，其特征在于，所述对数据进行删除处理进一步包括以下步骤：
    删除事实表中的纪录；
    删除该节点中事实表关联的维度表纪录；
    不删除主节点维度表中的纪录。
  8. 根据权利要求1所述的分布式并行数据库系统的数据分区方法，其特征在于，所述对数据进行更新处理进一步包括以下步骤：
    更新某一节点的维度表；
    查找与该维度表关联的事实表；
    更新所述事实表节点上与其关联的维度表。
  9. 根据权利要求1所述的分布式并行数据库系统的数据分区方法，其特征在于，所述对数据的更新是针对每个维度表和每个节点，建立一个布隆过滤器表记录维度表纪录在节点上的分布情况，找到保存某条指定纪录的节点并对该节点上的维度表进行更新。
  10. 根据权利要求9所述的分布式并行数据库系统的数据分区方法，其特征在于，所述布隆过滤器表存储在前端服务器或每个节点上。
PCT/CN2010/077565 2010-07-28 2010-10-01 一种分布式并行数据库系统的数据分区方法 WO2012012968A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US13/325,810 US20120109888A1 (en) 2010-07-28 2011-12-14 Data partitioning method of distributed parallel database system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN2010102396560A CN101916261B (zh) 2010-07-28 2010-07-28 一种分布式并行数据库系统的数据分区方法
CN201010239656.0 2010-07-28

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US13/325,810 Continuation US20120109888A1 (en) 2010-07-28 2011-12-14 Data partitioning method of distributed parallel database system

Publications (1)

Publication Number Publication Date
WO2012012968A1 true WO2012012968A1 (zh) 2012-02-02

Family

ID=43323773

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2010/077565 WO2012012968A1 (zh) 2010-07-28 2010-10-01 一种分布式并行数据库系统的数据分区方法

Country Status (3)

Country Link
US (1) US20120109888A1 (zh)
CN (1) CN101916261B (zh)
WO (1) WO2012012968A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210334264A1 (en) * 2018-12-09 2021-10-28 Kohei KAIGAI System, method, and program for increasing efficiency of database queries

Families Citing this family (113)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102043726B (zh) * 2010-12-29 2012-08-15 北京播思软件技术有限公司 一种大规模时序数据的存储管理方法
JP5727258B2 (ja) * 2011-02-25 2015-06-03 ウイングアーク１ｓｔ株式会社 分散型データベースシステム
WO2012131927A1 (ja) * 2011-03-30 2012-10-04 株式会社日立製作所 計算機システム及びデータ管理方法
US20140280075A1 (en) * 2011-08-26 2014-09-18 Hewlett-Packard Development Company, L.P. Multidimension clusters for data partitioning
US8812564B2 (en) * 2011-12-20 2014-08-19 Sap Ag Parallel uniqueness checks for partitioned tables
CN102662968A (zh) * 2012-03-09 2012-09-12 浪潮通信信息系统有限公司 一种对于Oracle大数据量存储的优化方法
CN103309902A (zh) * 2012-03-16 2013-09-18 多玩娱乐信息技术（北京）有限公司 一种社交网络中用户信息存储和查找的方法和装置
US8996464B2 (en) * 2012-06-11 2015-03-31 Microsoft Technology Licensing, Llc Efficient partitioning techniques for massively distributed computation
CN103488645A (zh) * 2012-06-13 2014-01-01 镇江华扬信息科技有限公司 一种用于物联网数据更新的结构设计方法
WO2014015492A1 (zh) * 2012-07-26 2014-01-30 华为技术有限公司 数据分布的方法、装置及系统
US9430550B2 (en) 2012-09-28 2016-08-30 Oracle International Corporation Clustering a table in a relational database management system
US9514187B2 (en) 2012-09-28 2016-12-06 Oracle International Corporation Techniques for using zone map information for post index access pruning
US9239741B2 (en) * 2012-10-16 2016-01-19 Futurewei Technologies, Inc. System and method for flexible distributed massively parallel processing (MPP)
CN104871153B8 (zh) * 2012-10-29 2019-02-01 华为技术有限公司 用于分布式大规模并行处理数据库的方法和系统
CN109388638B (zh) * 2012-10-29 2022-04-05 华为技术有限公司 用于分布式大规模并行处理数据库的方法及系统
US9195701B2 (en) * 2012-10-29 2015-11-24 Futurewei Technologies, Inc. System and method for flexible distributed massively parallel processing (MPP) database
CN103838787B (zh) * 2012-11-27 2018-07-10 阿里巴巴集团控股有限公司 一种对分布式数据仓库进行更新的方法和设备
US8799284B2 (en) * 2012-11-30 2014-08-05 Futurewei Technologies, Inc. Method for automated scaling of a massive parallel processing (MPP) database
CN104077724A (zh) * 2013-03-28 2014-10-01 北京东方道迩信息技术股份有限公司 一种面向物联网综合应用的基础空间信息架构方法
WO2014154016A1 (zh) * 2013-03-29 2014-10-02 深圳市并行科技有限公司 并行数据库管理系统及设计方案
US9311380B2 (en) * 2013-03-29 2016-04-12 International Business Machines Corporation Processing spatial joins using a mapreduce framework
US9501526B2 (en) 2013-04-17 2016-11-22 Excalibur Ip, Llc Efficient database searching
US9390162B2 (en) * 2013-04-25 2016-07-12 International Business Machines Corporation Management of a database system
US10452632B1 (en) * 2013-06-29 2019-10-22 Teradata Us, Inc. Multi-input SQL-MR
CN103412897B (zh) * 2013-07-25 2017-03-01 中国科学院软件研究所 一种基于分布式结构的并行数据处理方法
CN103440362A (zh) * 2013-07-27 2013-12-11 国家电网公司 一种可扩展维度的输变电工程建设管理展示平台的建模方法
CN104376025B (zh) * 2013-08-16 2017-10-10 华为技术有限公司 分布式数据库的数据存储方法和装置
US9229996B2 (en) * 2013-12-30 2016-01-05 Microsoft Technology Licensing, Llc Providing consistent tenant experiences for multi-tenant databases
CN104809129B (zh) 2014-01-26 2018-07-20 华为技术有限公司 一种分布式数据存储方法、装置和系统
CN105264521B (zh) * 2014-02-18 2018-10-30 华为技术有限公司 一种数据表的导入方法、数据管理器以及服务器
US10545917B2 (en) 2014-02-19 2020-01-28 Snowflake Inc. Multi-range and runtime pruning
US9576039B2 (en) 2014-02-19 2017-02-21 Snowflake Computing Inc. Resource provisioning systems and methods
CN105517644B (zh) * 2014-03-05 2020-04-21 华为技术有限公司 一种数据分区方法和设备
US9454574B2 (en) 2014-03-28 2016-09-27 Sybase, Inc. Bloom filter costing estimation
US9491060B1 (en) * 2014-06-30 2016-11-08 EMC IP Holding Company LLC Integrated wireless sensor network (WSN) and massively parallel processing database management system (MPP DBMS)
US10289723B1 (en) * 2014-08-21 2019-05-14 Amazon Technologies, Inc. Distributed union all queries
US9875263B2 (en) 2014-10-21 2018-01-23 Microsoft Technology Licensing, Llc Composite partition functions
CN104391948B (zh) * 2014-12-01 2017-11-21 广东电网有限责任公司清远供电局 数据仓库的数据标准化构建方法及系统
US20160188643A1 (en) * 2014-12-31 2016-06-30 Futurewei Technologies, Inc. Method and apparatus for scalable sorting of a data set
WO2016112502A1 (zh) * 2015-01-14 2016-07-21 华为技术有限公司 存储查询结果的方法和装置、计算设备
CN106156168B (zh) * 2015-04-16 2019-10-22 华为技术有限公司 在跨分区数据库中查询数据的方法及跨分区查询装置
CN104794249B (zh) * 2015-05-15 2018-08-28 网易乐得科技有限公司 一种数据库的实现方法和设备
WO2016191995A1 (zh) * 2015-05-31 2016-12-08 华为技术有限公司 一种分布式数据库中关联表分区的方法和设备
US9922081B2 (en) 2015-06-11 2018-03-20 Microsoft Technology Licensing, Llc Bidirectional cross-filtering in analysis service systems
US10289707B2 (en) 2015-08-10 2019-05-14 International Business Machines Corporation Data skipping and compression through partitioning of data
CN106569731B (zh) * 2015-10-10 2020-08-04 阿里巴巴集团控股有限公司 极限存储方法、装置及设备
US11100073B2 (en) * 2015-11-12 2021-08-24 Verizon Media Inc. Method and system for data assignment in a distributed system
CN105740365B (zh) * 2016-01-27 2019-02-05 北京掌阔移动传媒科技有限公司 一种数据仓库快速查询方法和装置
CN107229635B (zh) * 2016-03-24 2020-06-02 华为技术有限公司 一种数据处理的方法、存储节点及协调节点
US10108632B2 (en) 2016-05-02 2018-10-23 Google Llc Splitting and moving ranges in a distributed system
CN106202441A (zh) * 2016-07-13 2016-12-07 腾讯科技（深圳）有限公司 基于关系型数据库的数据处理方法、装置和系统
US10437780B2 (en) 2016-07-14 2019-10-08 Snowflake Inc. Data pruning based on metadata
US10713276B2 (en) 2016-10-03 2020-07-14 Ocient, Inc. Data transition in highly parallel database management system
KR101961562B1 (ko) * 2016-10-20 2019-03-22 영남대학교 산학협력단 조인 방법, 이를 실행시키는 컴퓨터 프로그램 및 기록매체
US10868863B1 (en) 2016-12-14 2020-12-15 Ocient Inc. System and method for designating a leader using a consensus protocol within a database management system
US10706031B2 (en) 2016-12-14 2020-07-07 Ocient, Inc. Database management systems for managing data with data confidence
US20180173762A1 (en) * 2016-12-15 2018-06-21 Futurewei Technologies, Inc. System and Method of Adaptively Partitioning Data to Speed Up Join Queries on Distributed and Parallel Database Systems
CN108205571B (zh) * 2016-12-20 2022-04-29 航天信息股份有限公司 键值数据表的连接方法及装置
CN107066495B (zh) * 2016-12-29 2020-04-21 北京瑞卓喜投科技发展有限公司 沿纵向拓展的区块链的生成方法及系统
US10747765B2 (en) 2017-05-30 2020-08-18 Ocient Inc. System and method for optimizing large database management systems with multiple optimizers
CN107329983B (zh) * 2017-06-01 2020-12-01 昆仑智汇数据科技(北京)有限公司 一种机器数据分布式存储、读取方法及系统
US11182125B2 (en) 2017-09-07 2021-11-23 Ocient Inc. Computing device sort function
CN110019544B (zh) * 2017-09-30 2022-08-19 北京国双科技有限公司 数据查询方法及系统
US10585915B2 (en) 2017-10-25 2020-03-10 International Business Machines Corporation Database sharding
CN110109951B (zh) * 2017-12-29 2022-12-06 华为技术有限公司 一种关联查询的方法、数据库应用系统及服务器
CN108482429A (zh) * 2018-03-09 2018-09-04 南京南瑞继保电气有限公司 一种轨道交通综合监控系统架构
US11354310B2 (en) 2018-05-23 2022-06-07 Oracle International Corporation Dual purpose zone maps
US11157496B2 (en) 2018-06-01 2021-10-26 International Business Machines Corporation Predictive data distribution for parallel databases to optimize storage and query performance
US11163764B2 (en) 2018-06-01 2021-11-02 International Business Machines Corporation Predictive data distribution for parallel databases to optimize storage and query performance
CN109271408B (zh) * 2018-08-31 2020-07-28 阿里巴巴集团控股有限公司 一种分布式数据连接处理方法、装置、设备及存储介质
CN109299191A (zh) * 2018-09-18 2019-02-01 新华三大数据技术有限公司 一种数据分布方法、装置、服务器及计算机存储介质
US11256696B2 (en) 2018-10-15 2022-02-22 Ocient Holdings LLC Data set compression within a database system
US11709835B2 (en) 2018-10-15 2023-07-25 Ocient Holdings LLC Re-ordered processing of read requests
US11880368B2 (en) 2018-10-15 2024-01-23 Ocient Holdings LLC Compressing data sets for storage in a database system
US11886436B2 (en) 2018-10-15 2024-01-30 Ocient Inc. Segmenting a partition of a data set based on a data storage coding scheme
US11249916B2 (en) 2018-10-15 2022-02-15 Ocient Holdings LLC Single producer single consumer buffering in database systems
CN109871415B (zh) * 2019-01-21 2021-04-30 武汉光谷信息技术股份有限公司 一种基于图数据库的用户画像构建方法、系统及存储介质
CN109901948B (zh) * 2019-02-18 2022-04-12 国家计算机网络与信息安全管理中心 无共享数据库集群异地双活容灾系统
FR3096799B1 (fr) * 2019-05-29 2021-11-05 Amadeus Agrégation et mise à jour d’objets de donnée hétérogènes
US11093500B2 (en) 2019-10-28 2021-08-17 Ocient Holdings LLC Enforcement of minimum query cost rules required for access to a database system
US11106679B2 (en) 2019-10-30 2021-08-31 Ocient Holdings LLC Enforcement of sets of query rules for access to data supplied by a plurality of data providers
US11609911B2 (en) 2019-12-19 2023-03-21 Ocient Holdings LLC Selecting a normalized form for conversion of a query expression
US11567939B2 (en) 2019-12-26 2023-01-31 Snowflake Inc. Lazy reassembling of semi-structured data
US11308090B2 (en) 2019-12-26 2022-04-19 Snowflake Inc. Pruning index to support semi-structured data types
US10769150B1 (en) 2019-12-26 2020-09-08 Snowflake Inc. Pruning indexes to enhance database query processing
US11372860B2 (en) 2019-12-26 2022-06-28 Snowflake Inc. Processing techniques for queries where predicate values are unknown until runtime
US11061910B1 (en) 2020-01-31 2021-07-13 Ocient Holdings LLC Servicing concurrent queries via virtual segment recovery
US11853364B2 (en) 2020-01-31 2023-12-26 Ocient Holdings LLC Level-based queries in a database system and methods for use therewith
US11599463B2 (en) 2020-03-25 2023-03-07 Ocient Holdings LLC Servicing queries during data ingress
US11238041B2 (en) 2020-03-25 2022-02-01 Ocient Holdings LLC Facilitating query executions via dynamic data block routing
US11580102B2 (en) 2020-04-02 2023-02-14 Ocient Holdings LLC Implementing linear algebra functions via decentralized execution of query operator flows
CN111522641B (zh) * 2020-04-21 2023-11-14 北京嘀嘀无限科技发展有限公司 任务调度方法、装置、计算机设备和存储介质
US11294916B2 (en) 2020-05-20 2022-04-05 Ocient Holdings LLC Facilitating query executions via multiple modes of resultant correctness
US11775529B2 (en) 2020-07-06 2023-10-03 Ocient Holdings LLC Recursive functionality in relational database systems
US11321288B2 (en) 2020-08-05 2022-05-03 Ocient Holdings LLC Record deduplication in database systems
US11755589B2 (en) 2020-08-05 2023-09-12 Ocient Holdings LLC Delaying segment generation in database systems
US11880716B2 (en) 2020-08-05 2024-01-23 Ocient Holdings LLC Parallelized segment generation via key-based subdivision in database systems
US11468099B2 (en) 2020-10-12 2022-10-11 Oracle International Corporation Automatic creation and maintenance of zone maps
US11822532B2 (en) 2020-10-14 2023-11-21 Ocient Holdings LLC Per-segment secondary indexing in database systems
CN112256698B (zh) * 2020-10-16 2023-09-05 美林数据技术股份有限公司 一种基于多哈希函数的表关系自动关联方法
US11507578B2 (en) 2020-10-19 2022-11-22 Ocient Holdings LLC Delaying exceptions in query execution
US11675757B2 (en) 2020-10-29 2023-06-13 Ocient Holdings LLC Maintaining row durability data in database systems
US11297123B1 (en) 2020-12-11 2022-04-05 Ocient Holdings LLC Fault-tolerant data stream processing
US11314743B1 (en) 2020-12-29 2022-04-26 Ocient Holdings LLC Storing records via multiple field-based storage mechanisms
CN112650738B (zh) * 2020-12-31 2021-09-21 广西中科曙光云计算有限公司 一种开放数据库的构建方法
CN112800085B (zh) * 2021-04-13 2021-09-14 成都四方伟业软件股份有限公司 一种基于布隆过滤器识别表间主外键字段的方法及装置
US11645273B2 (en) 2021-05-28 2023-05-09 Ocient Holdings LLC Query execution utilizing probabilistic indexing
CN113468178B (zh) * 2021-07-07 2022-07-29 武汉达梦数据库股份有限公司 一种关联表的数据分区装载方法与装置
US11803544B2 (en) 2021-10-06 2023-10-31 Ocient Holdings LLC Missing data-based indexing in database systems
US11983172B2 (en) 2021-12-07 2024-05-14 Ocient Holdings LLC Generation of a predictive model for selection of batch sizes in performing data format conversion
CN114595294B (zh) * 2022-03-11 2022-09-20 北京梦诚科技有限公司 一种数据仓库建模和抽取方法及系统
US11880369B1 (en) 2022-11-21 2024-01-23 Snowflake Inc. Pruning data based on state of top K operator
CN115617817B (zh) * 2022-12-14 2023-02-17 深圳迅策科技有限公司 一种基于全链路的全域资产报表生成方法

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101120340A (zh) * 2004-02-21 2008-02-06 数据迅捷股份有限公司 超无共享并行数据库
US20080270363A1 (en) * 2007-01-26 2008-10-30 Herbert Dennis Hunt Cluster processing of a core information matrix
US20090006309A1 (en) * 2007-01-26 2009-01-01 Herbert Dennis Hunt Cluster processing of an aggregated dataset

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7739224B1 (en) * 1998-05-06 2010-06-15 Infor Global Solutions (Michigan), Inc. Method and system for creating a well-formed database using semantic definitions
US8671091B2 (en) * 2006-08-02 2014-03-11 Hewlett-Packard Development Company, L.P. Optimizing snowflake schema queries

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101120340A (zh) * 2004-02-21 2008-02-06 数据迅捷股份有限公司 超无共享并行数据库
US20080270363A1 (en) * 2007-01-26 2008-10-30 Herbert Dennis Hunt Cluster processing of a core information matrix
US20090006309A1 (en) * 2007-01-26 2009-01-01 Herbert Dennis Hunt Cluster processing of an aggregated dataset

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210334264A1 (en) * 2018-12-09 2021-10-28 Kohei KAIGAI System, method, and program for increasing efficiency of database queries

Also Published As

Publication number Publication date
CN101916261B (zh) 2013-07-17
US20120109888A1 (en) 2012-05-03
CN101916261A (zh) 2010-12-15

Similar Documents

Publication Publication Date Title
WO2012012968A1 (zh) 一种分布式并行数据库系统的数据分区方法
US11461356B2 (en) Large scale unstructured database systems
US9946780B2 (en) Interpreting relational database statements using a virtual multidimensional data model
Dehdouh et al. Using the column oriented NoSQL model for implementing big data warehouses
EP1222569B1 (en) Method and systems for making olap hierarchies summarisable
US7870174B2 (en) Reference partitioned tables
CN108600321A (zh) 一种基于分布式内存云的图数据存储方法和系统
US7917463B2 (en) System and method for data warehousing and analytics on a distributed file system
KR102177190B1 (ko) 유연한 스키마를 사용한 데이터 관리
US6449619B1 (en) Method and apparatus for pipelining the transformation of information between heterogeneous sets of data sources
US9836519B2 (en) Densely grouping dimensional data
CN108369587A (zh) 创建用于交换的表
CN102890678A (zh) 一种基于格雷编码的分布式数据布局方法及查询方法
US20110022581A1 (en) Derived statistics for query optimization
CN101908075A (zh) 基于sql的并行计算系统及方法
US20100235344A1 (en) Mechanism for utilizing partitioning pruning techniques for xml indexes
JP4483034B2 (ja) 異種データソース統合アクセス方法
He et al. Stylus: a strongly-typed store for serving massive RDF data
CN113934713A (zh) 一种订单数据索引方法、系统、计算机设备以及存储介质
WO2023066222A1 (zh) 数据处理方法、装置、电子设备、存储介质及程序产品
CN106339432A (zh) 一种按查询内容进行负载均衡的系统及其方法
US8554722B2 (en) Method for transferring data into database systems
Liu et al. EntityManager: Managing dirty data based on entity resolution
CN107924399A (zh) 处理对主机而言数据传输量不明的检索请求的检索处理系统和方法
CN110569310A (zh) 一种云计算环境下的关系大数据的管理方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 10855196

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 31/05/2013)

122 Ep: pct application non-entry in european phase

Ref document number: 10855196

Country of ref document: EP

Kind code of ref document: A1