CN111247518B - 用于数据库分片的方法和系统 - Google Patents

用于数据库分片的方法和系统 Download PDF

Info

Publication number
CN111247518B
CN111247518B CN201880068665.2A CN201880068665A CN111247518B CN 111247518 B CN111247518 B CN 111247518B CN 201880068665 A CN201880068665 A CN 201880068665A CN 111247518 B CN111247518 B CN 111247518B
Authority
CN
China
Prior art keywords
shard
database
bloom filter
record
records
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201880068665.2A
Other languages
English (en)
Chinese (zh)
Other versions
CN111247518A (zh
Inventor
C·N·小瓦伦
M·里安
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by International Business Machines Corp filed Critical International Business Machines Corp
Publication of CN111247518A publication Critical patent/CN111247518A/zh
Application granted granted Critical
Publication of CN111247518B publication Critical patent/CN111247518B/zh
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/211—Schema design and management
    • G06F16/212—Schema design and management with details for data modelling support
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2228—Indexing structures
    • G06F16/2237—Vectors, bitmaps or matrices
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24553—Query execution of query operations
    • G06F16/24554—Unary operations; Data partitioning operations

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
CN201880068665.2A 2017-10-25 2018-10-18 用于数据库分片的方法和系统 Active CN111247518B (zh)

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
US15/793,100 2017-10-25
US15/793,100 US10585915B2 (en) 2017-10-25 2017-10-25 Database sharding
US15/813,577 2017-11-15
US15/813,577 US10592532B2 (en) 2017-10-25 2017-11-15 Database sharding
PCT/EP2018/078495 WO2019081322A1 (en) 2017-10-25 2018-10-18 BASIC PARTITIONING OF DATA

Publications (2)

Publication Number Publication Date
CN111247518A CN111247518A (zh) 2020-06-05
CN111247518B true CN111247518B (zh) 2024-05-14

Family

ID=66169387

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201880068665.2A Active CN111247518B (zh) 2017-10-25 2018-10-18 用于数据库分片的方法和系统

Country Status (6)

Country Link
US (2) US10585915B2 (https=)
JP (1) JP7046172B2 (https=)
CN (1) CN111247518B (https=)
DE (1) DE112018004222T5 (https=)
GB (1) GB2581738A (https=)
WO (1) WO2019081322A1 (https=)

Families Citing this family (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113468232B (zh) * 2017-02-27 2024-10-18 分秒库公司 用于查询时间序列数据的可扩展数据库系统
US11126625B2 (en) 2019-05-31 2021-09-21 Salesforce.Com, Inc. Caching techniques for a database change stream
US20210034586A1 (en) 2019-08-02 2021-02-04 Timescale, Inc. Compressing data in database systems using hybrid row/column storage representations
US11194773B2 (en) * 2019-09-12 2021-12-07 Oracle International Corporation Integration of existing databases into a sharding environment
CN110968265B (zh) * 2019-11-05 2023-08-08 北京字节跳动网络技术有限公司 分片扩容方法、装置及电子设备
JP7458485B2 (ja) * 2019-12-20 2024-03-29 ナイアンティック， インコーポレイテッド 予測可能なクエリ応答時間を有するジオロケーションデータのシャードストレージ
JP7121195B2 (ja) * 2020-02-14 2022-08-17 グーグル エルエルシー セキュアマルチパーティリーチおよび頻度推定
US11531666B1 (en) 2020-08-20 2022-12-20 Amazon Technologies, Inc. Indexing partitions using distributed bloom filters
EP3961419A1 (en) * 2020-08-28 2022-03-02 Siemens Aktiengesellschaft Computer-implemented method for storing a dataset and computer network
CN112162981A (zh) * 2020-09-08 2021-01-01 杭州涂鸦信息技术有限公司 一种自适应的路由分库分表方法及系统
JP7479501B2 (ja) * 2020-10-05 2024-05-08 グーグル エルエルシー カウントのベクトルによるブルームフィルタのメタ推定
CN113760837B (zh) * 2020-10-27 2025-07-15 北京沃东天骏信息技术有限公司 数据写入、查询方法和装置
CN112417276A (zh) * 2020-11-18 2021-02-26 北京字节跳动网络技术有限公司 分页数据获取方法、装置、电子设备及计算机可读存储介质
CN114722360B (zh) * 2021-01-04 2024-10-18 中国移动通信有限公司研究院 水印插入方法、提取方法及装置
US11568065B2 (en) * 2021-01-15 2023-01-31 Bank Of America Corporation System for securing electronic data by aggregation of distributed electronic database entries
US11829394B2 (en) * 2021-03-11 2023-11-28 International Business Machines Corporation Soft deletion of data in sharded databases
CN114357024A (zh) * 2021-12-24 2022-04-15 南京苏宁软件技术有限公司 提升用户画像接口性能的方法、装置和计算机设备
JP7815000B2 (ja) * 2022-03-23 2026-02-17 株式会社エヌ・ティ・ティ・データ・セキスイシステムズ インデックス管理装置
CN114676139B (zh) * 2022-03-29 2025-04-18 浪潮云信息技术股份公司 一种索引数据存储方法、系统、设备及存储介质
CN115941787A (zh) * 2022-11-28 2023-04-07 北京青云科技股份有限公司 一种缓存数据访问处理方法、装置、电子设备及存储介质
CN115934761A (zh) * 2022-12-29 2023-04-07 天翼云科技有限公司 一种基于Counting布隆过滤器的数据库中间件查询优化方法
US11995084B1 (en) 2023-10-05 2024-05-28 Timescale, Inc. Database system for querying time-series data stored in a tiered storage using a cloud platform

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102473084A (zh) * 2009-07-14 2012-05-23 高通股份有限公司 用于在分布式网络中高效处理多关键字查询的方法和装置
CN104115146A (zh) * 2012-02-14 2014-10-22 阿尔卡特朗讯公司 在分布式系统中存储和搜索带标签的内容项的方法

Family Cites Families (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4722620B2 (ja) 2005-08-19 2011-07-13 Ｋｄｄｉ株式会社 暗号化文書検索方法および暗号化文書検索システム
US9275129B2 (en) 2006-01-23 2016-03-01 Symantec Corporation Methods and systems to efficiently find similar and near-duplicate emails and files
US8209178B1 (en) 2008-01-10 2012-06-26 Google Inc. Randomized language models
JP5353231B2 (ja) 2008-12-25 2013-11-27 日本電気株式会社 情報転送装置、情報転送方法およびプログラム
US20100312749A1 (en) 2009-06-04 2010-12-09 Microsoft Corporation Scalable lookup service for distributed database
CN102667761B (zh) 2009-06-19 2015-05-27 布雷克公司 可扩展的集群数据库
CN101916261B (zh) 2010-07-28 2013-07-17 北京播思软件技术有限公司 一种分布式并行数据库系统的数据分区方法
US8996463B2 (en) 2012-07-26 2015-03-31 Mongodb, Inc. Aggregation framework system architecture and method
US8924426B2 (en) 2011-04-29 2014-12-30 Google Inc. Joining tables in a mapreduce procedure
US9165074B2 (en) 2011-05-10 2015-10-20 Uber Technologies, Inc. Systems and methods for performing geo-search and retrieval of electronic point-of-interest records using a big index
US8856234B2 (en) 2013-02-28 2014-10-07 Workiva Llc System and method for performing distributed asynchronous calculations in a networked environment
US9507824B2 (en) * 2014-08-22 2016-11-29 Attivio Inc. Automated creation of join graphs for unrelated data sets among relational databases
US9875263B2 (en) 2014-10-21 2018-01-23 Microsoft Technology Licensing, Llc Composite partition functions
US9727275B2 (en) 2014-12-02 2017-08-08 International Business Machines Corporation Coordinating storage of data in dispersed storage networks
US20160328429A1 (en) 2015-03-17 2016-11-10 Cloudera, Inc. Mutations in a column store
US9886441B2 (en) 2015-04-06 2018-02-06 Sap Se Shard aware near real time indexing
US11210279B2 (en) 2016-04-15 2021-12-28 Apple Inc. Distributed offline indexing
US10430598B2 (en) * 2017-06-08 2019-10-01 The Government Of The United States, As Represented By The Secretary Of The Army Secure generalized bloom filter

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102473084A (zh) * 2009-07-14 2012-05-23 高通股份有限公司 用于在分布式网络中高效处理多关键字查询的方法和装置
CN104115146A (zh) * 2012-02-14 2014-10-22 阿尔卡特朗讯公司 在分布式系统中存储和搜索带标签的内容项的方法

Also Published As

Publication number Publication date
JP7046172B2 (ja) 2022-04-01
US10585915B2 (en) 2020-03-10
CN111247518A (zh) 2020-06-05
DE112018004222T5 (de) 2020-05-14
US10592532B2 (en) 2020-03-17
WO2019081322A1 (en) 2019-05-02
US20190121901A1 (en) 2019-04-25
JP2021500649A (ja) 2021-01-07
GB202007157D0 (en) 2020-07-01
GB2581738A (en) 2020-08-26
US20190121902A1 (en) 2019-04-25

Similar Documents

Publication Publication Date Title
CN111247518B (zh) 用于数据库分片的方法和系统
US10628449B2 (en) Method and apparatus for processing database data in distributed database system
US8793227B2 (en) Storage system for eliminating duplicated data
CN107704202B (zh) 一种数据快速读写的方法和装置
US10904316B2 (en) Data processing method and apparatus in service-oriented architecture system, and the service-oriented architecture system
US20230267116A1 (en) Translation of tenant identifiers
CN111857539B (zh) 用于管理存储系统的方法、设备和计算机可读介质
CN110362404B (zh) 一种基于sql的资源分配方法、装置和电子设备
CN105989015B (zh) 一种数据库扩容方法和装置以及访问数据库的方法和装置
JP2020123320A (ja) インデックスを管理するための方法、装置、設備及び記憶媒体
TWI579715B (zh) 搜尋伺服器、終端裝置及用於分散式網路之搜尋方法
CN104598652B (zh) 一种数据库查询方法及装置
CN110807028B (zh) 用于管理存储系统的方法、设备和计算机程序产品
CN112905587A (zh) 数据库的数据管理方法、装置及电子设备
CN107016115B (zh) 数据导出方法、装置、计算机可读存储介质及电子设备
US11151110B2 (en) Identification of records for post-cloning tenant identifier translation
US10437806B2 (en) Database management method and information processing apparatus
US11403273B1 (en) Optimizing hash table searching using bitmasks and linear probing
CN114817651A (zh) 数据存储方法、数据查询方法、装置和设备
EP3995972A1 (en) Metadata processing method and apparatus, and computer-readable storage medium
CN110968267B (zh) 数据管理方法、装置、服务器及系统
CN115729965A (zh) 信息流处理方法、装置、流服务器及存储介质
CN111309704B (zh) 数据库操作方法和数据库操作系统
CN120215967A (zh) 软件包安装方法和软件包安装装置
CN120256445A (zh) 数据查询方法、介质、设备和产品

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant