CN111213130B - 基于分散位置的去重的性能改进 - Google Patents

基于分散位置的去重的性能改进 Download PDF

Info

Publication number
CN111213130B
CN111213130B CN201880067324.3A CN201880067324A CN111213130B CN 111213130 B CN111213130 B CN 111213130B CN 201880067324 A CN201880067324 A CN 201880067324A CN 111213130 B CN111213130 B CN 111213130B
Authority
CN
China
Prior art keywords
memory region
owner
memory
predetermined number
regions
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201880067324.3A
Other languages
English (en)
Chinese (zh)
Other versions
CN111213130A (zh
Inventor
J·费舍尔-通博拉
Y·沙茨基
A·哈鲁米
A·波拉特-斯托勒
S·马伦科夫
T·西万
R·科恩
D·哈尼克
E·凯茨
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by International Business Machines Corp filed Critical International Business Machines Corp
Publication of CN111213130A publication Critical patent/CN111213130A/zh
Application granted granted Critical
Publication of CN111213130B publication Critical patent/CN111213130B/zh
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G06F3/0641—De-duplication techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operations
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1448—Management of the data involved in backup or backup restore
    • G06F11/1453—Management of the data involved in backup or backup restore using de-duplication of the data
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/174—Redundancy elimination performed by the file system
    • G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/174—Redundancy elimination performed by the file system
    • G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
    • G06F16/1752—De-duplication implemented within the file system, e.g. based on file segments based on file chunks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24553—Query execution of query operations
    • G06F16/24554—Unary operations; Data partitioning operations
    • G06F16/24556—Aggregation; Duplicate elimination

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Quality & Reliability (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Memory System (AREA)
CN201880067324.3A 2017-10-25 2018-10-12 基于分散位置的去重的性能改进 Active CN111213130B (zh)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US15/793,109 2017-10-25
US15/793,109 US11269531B2 (en) 2017-10-25 2017-10-25 Performance of dispersed location-based deduplication
PCT/IB2018/057924 WO2019082016A1 (en) 2017-10-25 2018-10-12 IMPROVED DEDUPLICATION PERFORMANCE BASED ON DISPERSED LOCATIONS

Publications (2)

Publication Number Publication Date
CN111213130A CN111213130A (zh) 2020-05-29
CN111213130B true CN111213130B (zh) 2024-03-01

Family

ID=66169951

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201880067324.3A Active CN111213130B (zh) 2017-10-25 2018-10-12 基于分散位置的去重的性能改进

Country Status (6)

Country Link
US (2) US11269531B2 (enExample)
JP (1) JP7087070B2 (enExample)
CN (1) CN111213130B (enExample)
DE (1) DE112018004402B4 (enExample)
GB (1) GB2580276B (enExample)
WO (1) WO2019082016A1 (enExample)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11269531B2 (en) 2017-10-25 2022-03-08 International Business Machines Corporation Performance of dispersed location-based deduplication
US12293227B2 (en) * 2018-06-21 2025-05-06 Telefonaktiebolaget Lm Ericsson (Publ) Memory allocation in a hierarchical memory system
US11455110B1 (en) * 2021-09-08 2022-09-27 International Business Machines Corporation Data deduplication
US12443536B1 (en) * 2024-04-10 2025-10-14 Dell Products L.P. Techniques for staging updated metadata pages based on owner and metadata

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101706825A (zh) * 2009-12-10 2010-05-12 华中科技大学 一种基于文件内容类型的重复数据删除方法
CN101710323A (zh) * 2008-09-11 2010-05-19 威睿公司 计算机存储去复制操作

Family Cites Families (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8782368B2 (en) * 2007-10-25 2014-07-15 Hewlett-Packard Development Company, L.P. Storing chunks in containers
US8825617B2 (en) 2008-03-14 2014-09-02 International Business Machines Corporation Limiting deduplication based on predetermined criteria
US7979658B2 (en) * 2008-03-25 2011-07-12 Spansion Llc Secure management of memory regions in a memory
US7567188B1 (en) * 2008-04-10 2009-07-28 International Business Machines Corporation Policy based tiered data deduplication strategy
US8156306B1 (en) 2009-12-18 2012-04-10 Emc Corporation Systems and methods for using thin provisioning to reclaim space identified by data reduction processes
JP5526824B2 (ja) 2010-02-02 2014-06-18 日本電気株式会社 ストレージシステム
US20110218967A1 (en) 2010-03-08 2011-09-08 Microsoft Corporation Partial Block Based Backups
US8577851B2 (en) * 2010-09-30 2013-11-05 Commvault Systems, Inc. Content aligned block-based deduplication
US9244967B2 (en) 2011-08-01 2016-01-26 Actifio, Inc. Incremental copy performance between data stores
US8806160B2 (en) 2011-08-16 2014-08-12 Pure Storage, Inc. Mapping in a storage system
JP5738471B2 (ja) * 2011-12-14 2015-06-24 株式会社日立製作所 ストレージ装置とそのメモリ制御方法
US9329987B1 (en) * 2012-06-14 2016-05-03 Marvell International Ltd. Systems and methods for dynamic tracking of memory regions
US9063864B2 (en) * 2012-07-16 2015-06-23 Hewlett-Packard Development Company, L.P. Storing data in presistent hybrid memory
JP6021680B2 (ja) 2013-02-19 2016-11-09 株式会社日立製作所 自律分散重複排除ファイルシステム、記憶装置ユニット及びデータアクセス方法
WO2014185974A2 (en) 2013-05-14 2014-11-20 Abercrombie Philip J Efficient data replication and garbage collection predictions
GB2518158A (en) * 2013-09-11 2015-03-18 Ibm Method and system for data access in a storage infrastructure
CN103559143A (zh) * 2013-11-08 2014-02-05 华为技术有限公司 数据拷贝管理装置及其数据拷贝方法
US9208167B1 (en) 2014-09-04 2015-12-08 Edifire LLC Distributed data synchronization and conflict resolution
US9965182B2 (en) 2015-10-21 2018-05-08 International Business Machines Corporation Optimization of data deduplication
US9817865B2 (en) * 2015-12-07 2017-11-14 International Business Machines Corporation Direct lookup for identifying duplicate data in a data deduplication system
US10013201B2 (en) 2016-03-29 2018-07-03 International Business Machines Corporation Region-integrated data deduplication
US10592348B2 (en) * 2016-06-17 2020-03-17 Acronis International Gmbh System and method for data deduplication using log-structured merge trees
US11269531B2 (en) 2017-10-25 2022-03-08 International Business Machines Corporation Performance of dispersed location-based deduplication

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101710323A (zh) * 2008-09-11 2010-05-19 威睿公司 计算机存储去复制操作
CN101706825A (zh) * 2009-12-10 2010-05-12 华中科技大学 一种基于文件内容类型的重复数据删除方法

Also Published As

Publication number Publication date
GB2580276B (en) 2020-12-09
JP7087070B2 (ja) 2022-06-20
GB2580276A (en) 2020-07-15
US12436700B2 (en) 2025-10-07
US20190121563A1 (en) 2019-04-25
US20220155987A1 (en) 2022-05-19
GB202007041D0 (en) 2020-06-24
JP2021500643A (ja) 2021-01-07
DE112018004402T5 (de) 2020-05-20
US11269531B2 (en) 2022-03-08
CN111213130A (zh) 2020-05-29
WO2019082016A1 (en) 2019-05-02
DE112018004402B4 (de) 2022-11-03

Similar Documents

Publication Publication Date Title
CN107870728B (zh) 用于移动数据的方法和设备
US12436700B2 (en) Performance of dispersed location-based deduplication
US10282124B2 (en) Opportunistic handling of freed data in data de-duplication
CN111949605A (zh) 用于实现文件系统的方法、设备和计算机程序产品
CN111124270B (zh) 缓存管理的方法、设备和计算机程序产品
CN110928846B (zh) 在混合安全环境中分割、编辑和传输安全文档
US10747452B1 (en) Hybrid log-structured array and allocated storage device
CN111684779B (zh) 分层存储管理系统中的数据迁移
US20180089210A1 (en) Tracking access pattern of inodes and pre-fetching inodes
CN115543965A (zh) 跨机房数据处理方法、设备、存储介质及程序产品
US9760577B2 (en) Write-behind caching in distributed file systems
US10901895B2 (en) Data file handling in a volatile memory
US9471246B2 (en) Data sharing using difference-on-write
US20180089086A1 (en) Tracking access pattern of inodes and pre-fetching inodes
US11662927B2 (en) Redirecting access requests between access engines of respective disk management devices
US8560544B2 (en) Clustering of analytic functions
US9626121B2 (en) De-duplication as part of other routinely performed processes
US10372516B2 (en) Message processing
US9857979B2 (en) Optimizing page boundary crossing in system memory using a reference bit and a change bit
US12197338B2 (en) Data feature detection and replacement in user-written data for caching
US10614092B2 (en) Optimizing data retrieval operation in big-data processing systems
US10621160B2 (en) Storage management inconsistency tracker
US10929793B2 (en) Utilizing analytic data to generate crowd-based custom logic units
HK1178278A1 (zh) 將rdma語義映射到高速存儲

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant