DE112018004402B4 - Verbesserte leistung von verteilter, ortsbezogener deduplikation - Google Patents

Verbesserte leistung von verteilter, ortsbezogener deduplikation Download PDF

Info

Publication number
DE112018004402B4
DE112018004402B4 DE112018004402.5T DE112018004402T DE112018004402B4 DE 112018004402 B4 DE112018004402 B4 DE 112018004402B4 DE 112018004402 T DE112018004402 T DE 112018004402T DE 112018004402 B4 DE112018004402 B4 DE 112018004402B4
Authority
DE
Germany
Prior art keywords
storage
regions
predetermined number
owner
region
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
DE112018004402.5T
Other languages
German (de)
English (en)
Other versions
DE112018004402T5 (de
Inventor
Jonathan FISCHER-TOUBOL
Yosef SHATSKY
Afief HALUMI
Asaf Porat-Stoler
Sergey MARENKOV
Tom Sivan
Reut Cohen
Danny Harnik
Ety KHAITZIN
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by International Business Machines Corp filed Critical International Business Machines Corp
Publication of DE112018004402T5 publication Critical patent/DE112018004402T5/de
Application granted granted Critical
Publication of DE112018004402B4 publication Critical patent/DE112018004402B4/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G06F3/0641—De-duplication techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1448—Management of the data involved in backup or backup restore
    • G06F11/1453—Management of the data involved in backup or backup restore using de-duplication of the data
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/174—Redundancy elimination performed by the file system
    • G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/174—Redundancy elimination performed by the file system
    • G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
    • G06F16/1752—De-duplication implemented within the file system, e.g. based on file segments based on file chunks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24553—Query execution of query operations
    • G06F16/24554—Unary operations; Data partitioning operations
    • G06F16/24556—Aggregation; Duplicate elimination

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Quality & Reliability (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Memory System (AREA)
DE112018004402.5T 2017-10-25 2018-10-12 Verbesserte leistung von verteilter, ortsbezogener deduplikation Active DE112018004402B4 (de)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US15/793,109 US11269531B2 (en) 2017-10-25 2017-10-25 Performance of dispersed location-based deduplication
US15/793,109 2017-10-25
PCT/IB2018/057924 WO2019082016A1 (en) 2017-10-25 2018-10-12 IMPROVED DEDUPLICATION PERFORMANCE BASED ON DISPERSED LOCATIONS

Publications (2)

Publication Number Publication Date
DE112018004402T5 DE112018004402T5 (de) 2020-05-20
DE112018004402B4 true DE112018004402B4 (de) 2022-11-03

Family

ID=66169951

Family Applications (1)

Application Number Title Priority Date Filing Date
DE112018004402.5T Active DE112018004402B4 (de) 2017-10-25 2018-10-12 Verbesserte leistung von verteilter, ortsbezogener deduplikation

Country Status (6)

Country Link
US (2) US11269531B2 (enExample)
JP (1) JP7087070B2 (enExample)
CN (1) CN111213130B (enExample)
DE (1) DE112018004402B4 (enExample)
GB (1) GB2580276B (enExample)
WO (1) WO2019082016A1 (enExample)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12436700B2 (en) 2017-10-25 2025-10-07 International Business Machines Corporation Performance of dispersed location-based deduplication

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12293227B2 (en) * 2018-06-21 2025-05-06 Telefonaktiebolaget Lm Ericsson (Publ) Memory allocation in a hierarchical memory system
US11455110B1 (en) * 2021-09-08 2022-09-27 International Business Machines Corporation Data deduplication
US12443536B1 (en) * 2024-04-10 2025-10-14 Dell Products L.P. Techniques for staging updated metadata pages based on owner and metadata

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170116229A1 (en) 2015-10-21 2017-04-27 International Business Machines Corporation Optimization of data deduplication

Family Cites Families (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8782368B2 (en) * 2007-10-25 2014-07-15 Hewlett-Packard Development Company, L.P. Storing chunks in containers
US8825617B2 (en) 2008-03-14 2014-09-02 International Business Machines Corporation Limiting deduplication based on predetermined criteria
US7979658B2 (en) * 2008-03-25 2011-07-12 Spansion Llc Secure management of memory regions in a memory
US7567188B1 (en) * 2008-04-10 2009-07-28 International Business Machines Corporation Policy based tiered data deduplication strategy
US10642794B2 (en) 2008-09-11 2020-05-05 Vmware, Inc. Computer storage deduplication
CN101706825B (zh) 2009-12-10 2011-04-20 华中科技大学 一种基于文件内容类型的重复数据删除方法
US8156306B1 (en) 2009-12-18 2012-04-10 Emc Corporation Systems and methods for using thin provisioning to reclaim space identified by data reduction processes
JP5526824B2 (ja) 2010-02-02 2014-06-18 日本電気株式会社 ストレージシステム
US20110218967A1 (en) 2010-03-08 2011-09-08 Microsoft Corporation Partial Block Based Backups
US8577851B2 (en) * 2010-09-30 2013-11-05 Commvault Systems, Inc. Content aligned block-based deduplication
US8688650B2 (en) 2011-08-01 2014-04-01 Actifio, Inc. Data fingerprinting for copy accuracy assurance
US8806160B2 (en) 2011-08-16 2014-08-12 Pure Storage, Inc. Mapping in a storage system
JP5738471B2 (ja) * 2011-12-14 2015-06-24 株式会社日立製作所 ストレージ装置とそのメモリ制御方法
US9329987B1 (en) * 2012-06-14 2016-05-03 Marvell International Ltd. Systems and methods for dynamic tracking of memory regions
US9063864B2 (en) * 2012-07-16 2015-06-23 Hewlett-Packard Development Company, L.P. Storing data in presistent hybrid memory
JP6021680B2 (ja) 2013-02-19 2016-11-09 株式会社日立製作所 自律分散重複排除ファイルシステム、記憶装置ユニット及びデータアクセス方法
CA2912394A1 (en) 2013-05-14 2014-11-20 Actifio, Inc. Efficient data replication and garbage collection predictions
GB2518158A (en) * 2013-09-11 2015-03-18 Ibm Method and system for data access in a storage infrastructure
CN103559143A (zh) * 2013-11-08 2014-02-05 华为技术有限公司 数据拷贝管理装置及其数据拷贝方法
US9208167B1 (en) 2014-09-04 2015-12-08 Edifire LLC Distributed data synchronization and conflict resolution
US9817865B2 (en) * 2015-12-07 2017-11-14 International Business Machines Corporation Direct lookup for identifying duplicate data in a data deduplication system
US10013201B2 (en) 2016-03-29 2018-07-03 International Business Machines Corporation Region-integrated data deduplication
US10592348B2 (en) * 2016-06-17 2020-03-17 Acronis International Gmbh System and method for data deduplication using log-structured merge trees
US11269531B2 (en) 2017-10-25 2022-03-08 International Business Machines Corporation Performance of dispersed location-based deduplication

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170116229A1 (en) 2015-10-21 2017-04-27 International Business Machines Corporation Optimization of data deduplication

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12436700B2 (en) 2017-10-25 2025-10-07 International Business Machines Corporation Performance of dispersed location-based deduplication

Also Published As

Publication number Publication date
GB2580276B (en) 2020-12-09
US12436700B2 (en) 2025-10-07
CN111213130A (zh) 2020-05-29
WO2019082016A1 (en) 2019-05-02
GB2580276A (en) 2020-07-15
US20220155987A1 (en) 2022-05-19
JP7087070B2 (ja) 2022-06-20
JP2021500643A (ja) 2021-01-07
US11269531B2 (en) 2022-03-08
US20190121563A1 (en) 2019-04-25
GB202007041D0 (en) 2020-06-24
CN111213130B (zh) 2024-03-01
DE112018004402T5 (de) 2020-05-20

Similar Documents

Publication Publication Date Title
DE112018004402B4 (de) Verbesserte leistung von verteilter, ortsbezogener deduplikation
DE102016221813B4 (de) Datenreplikation auf der Grundlage des Verlaufs des Komprimierungsverhältnisses
DE102013211071B4 (de) Mit geringem Mehraufwand verbundene Verbesserung der Zuverlässigkeit eines Journaling-Dateisystems unter Verwendung von Halbleiterspeicherung und Deduplizierung
DE112020006010T5 (de) Schulung eines neuronalen netzwerks durch verwenden eines datenflussgraphen und dynamische verwaltung von arbeitsspeicher
DE112013001905B4 (de) Erhöhte Inline-Deduplizierungseffizienz
DE112020004067B4 (de) Hybride daten-modell-parallelität für effizientes deep learning
DE102016013248A1 (de) Bezugsblockansammlung in einer Bezugsmenge zur Deduplizierung beim Speichermanagement
DE112020004651B4 (de) Multi-tenant-etl-ressourcenaufteilung
DE112012005222T5 (de) Halbleiter-Datenspeicherverwaltung
DE112018005205T5 (de) Komprimierung von vollständig verbundenen / wiederkehrenden Schichten von einem oder mehreren tiefen Netzen durch Durchsetzen von räumlicher Lokalität für Gewichtsmatrizen und erwirken von Frequenzkomprimierung
DE202012013432U1 (de) Speichern von Daten auf Speicherknoten
DE112013000650T5 (de) Datenzwischenspeicherungsbereich
DE112018004138B4 (de) Asynchrone aktualisierung von metadatenspuren in reaktion auf einen mittels einer e/a-operation über eine busschnittstelle erzeugten cachetreffer
DE112017007656T5 (de) Verschobene aktualisierung von datenbank-hashcode in einer blockchain
DE112018002266T5 (de) Kognitives Datenfiltern für Speicherumgebungen
DE112018001290T5 (de) Verfahren zum Schätzen der Löschbarkeit von Datenobjekten
DE112018005359T5 (de) Verhindern eines Beibehaltens von Datensatzsperren durch Transaktionen mit langer Laufzeit
DE112016003598B4 (de) Gleichzeitige Massenverarbeitung von baumbasierten Datenstrukturen
DE112018003585T5 (de) Deduplizierung eines bandlaufwerkspeichers
DE112021003506T5 (de) Hybrides ensemble-modell, das edge- und serverseitige inferenzen nutzt
DE102016100773A1 (de) Erfassen von Komprimierungsleistungsmesswerten für die Verarbeitung von Daten
DE112017004160T5 (de) Schützen eines Webservers vor einer nicht autorisierten Client-Anwendung
DE112017005014T5 (de) Qualifizieren des Durchsuchens eines Verzweigungsprädiktors unter Verwendung der Vorhersage einer Datenstromlänge
DE112021004115T5 (de) Sicherheitssystem für eine Segmentierung von Computerdatei-Metadaten
DE112018005620T5 (de) Auftragsverwaltung in einem datenverarbeitungssystem

Legal Events

Date Code Title Description
R012 Request for examination validly filed
R016 Response to examination communication
R018 Grant decision by examination section/examining division
R084 Declaration of willingness to licence
R020 Patent grant now final