WO2009091957A3 - Scalable de-duplication mechanism - Google Patents

Scalable de-duplication mechanism Download PDF

Info

Publication number
WO2009091957A3
WO2009091957A3 PCT/US2009/031222 US2009031222W WO2009091957A3 WO 2009091957 A3 WO2009091957 A3 WO 2009091957A3 US 2009031222 W US2009031222 W US 2009031222W WO 2009091957 A3 WO2009091957 A3 WO 2009091957A3
Authority
WO
WIPO (PCT)
Prior art keywords
data object
duplication
application layer
scalable
layer data
Prior art date
Application number
PCT/US2009/031222
Other languages
French (fr)
Other versions
WO2009091957A2 (en
Inventor
Miklos Sandorfi
Timmie G. Reiter
Original Assignee
Sepaton, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sepaton, Inc. filed Critical Sepaton, Inc.
Priority to CA2711273A priority Critical patent/CA2711273A1/en
Priority to EP09702041A priority patent/EP2235640A2/en
Priority to JP2010543270A priority patent/JP2011510405A/en
Priority to CN2009801016964A priority patent/CN101939737A/en
Priority to AU2009206038A priority patent/AU2009206038A1/en
Publication of WO2009091957A2 publication Critical patent/WO2009091957A2/en
Publication of WO2009091957A3 publication Critical patent/WO2009091957A3/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G06F3/0641—De-duplication techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0683—Plurality of storage devices
    • G06F3/0686—Libraries, e.g. tape libraries, jukebox

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Library & Information Science (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Signal Processing For Digital Recording And Reproducing (AREA)
  • Detection And Prevention Of Errors In Transmission (AREA)
  • Hardware Redundancy (AREA)

Abstract

A method for removing redundant data from a backup storage system is presented. In one example, the method may include receiving the application layer data object, selecting a de-duplication domain from a plurality of de-duplication domains based at least in part on a data object characteristic associated with the de-duplication domain, determining that the application layer data object has the characteristic and directing the application layer data object to the de-duplication domain.
PCT/US2009/031222 2008-01-16 2009-01-16 Scalable de-duplication mechanism WO2009091957A2 (en)

Priority Applications (5)

Application Number Priority Date Filing Date Title
CA2711273A CA2711273A1 (en) 2008-01-16 2009-01-16 Scalable de-duplication mechanism
EP09702041A EP2235640A2 (en) 2008-01-16 2009-01-16 Scalable de-duplication mechanism
JP2010543270A JP2011510405A (en) 2008-01-16 2009-01-16 Scalable deduplication mechanism
CN2009801016964A CN101939737A (en) 2008-01-16 2009-01-16 Scalable de-duplication mechanism
AU2009206038A AU2009206038A1 (en) 2008-01-16 2009-01-16 Scalable de-duplication mechanism

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US2150108P 2008-01-16 2008-01-16
US61/021,501 2008-01-16

Publications (2)

Publication Number Publication Date
WO2009091957A2 WO2009091957A2 (en) 2009-07-23
WO2009091957A3 true WO2009091957A3 (en) 2009-10-15

Family

ID=40885894

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2009/031222 WO2009091957A2 (en) 2008-01-16 2009-01-16 Scalable de-duplication mechanism

Country Status (6)

Country Link
EP (1) EP2235640A2 (en)
JP (1) JP2011510405A (en)
CN (1) CN101939737A (en)
AU (1) AU2009206038A1 (en)
CA (1) CA2711273A1 (en)
WO (1) WO2009091957A2 (en)

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8291183B2 (en) 2009-01-15 2012-10-16 Emc Corporation Assisted mainframe data de-duplication
JP5623239B2 (en) 2010-10-28 2014-11-12 インターナショナル・ビジネス・マシーンズ・コーポレーションＩｎｔｅｒｎａｔｉｏｎａｌ Ｂｕｓｉｎｅｓｓ Ｍａｃｈｉｎｅｓ Ｃｏｒｐｏｒａｔｉｏｎ Storage device for eliminating duplication of write record, and write method thereof
US8682873B2 (en) 2010-12-01 2014-03-25 International Business Machines Corporation Efficient construction of synthetic backups within deduplication storage system
US9933978B2 (en) 2010-12-16 2018-04-03 International Business Machines Corporation Method and system for processing data
US8332372B2 (en) * 2010-12-16 2012-12-11 International Business Machines Corporation Method and system for processing data
WO2013085519A1 (en) * 2011-12-08 2013-06-13 Empire Technology Development, Llc Storage discounts for allowing cross-user deduplication
US9128616B2 (en) * 2012-04-13 2015-09-08 Hitachi, Ltd. Storage device to backup content based on a deduplication system
WO2016115663A1 (en) 2015-01-19 2016-07-28 Nokia Technologies Oy Method and apparatus for heterogeneous data storage management in cloud computing
JP6720612B2 (en) * 2016-03-23 2020-07-08 日本電気株式会社 Information processing apparatus, storage system, storage control method, and computer program
US10235396B2 (en) * 2016-08-29 2019-03-19 International Business Machines Corporation Workload optimized data deduplication using ghost fingerprints
WO2018226228A1 (en) * 2017-06-08 2018-12-13 Hitachi Data Systems Corporation Deduplicating distributed erasure coded objects
US11182406B2 (en) 2020-03-27 2021-11-23 International Business Machines Corporation Increased data availability during replication
CN112632191B (en) * 2020-12-29 2024-06-11 中国农业银行股份有限公司 Data processing method and system
CN114020218B (en) * 2021-11-25 2023-06-02 建信金融科技有限责任公司 Hybrid de-duplication scheduling method and system
WO2024046554A1 (en) * 2022-08-31 2024-03-07 Huawei Technologies Co., Ltd. Parallel deduplication mechanism on sequential storage media

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20020066836A (en) * 2001-02-14 2002-08-21 한국통신정보기술 주식회사 Recording medium for recording hierarchical data structure and method for creating hierarchical data storage structure
US6795819B2 (en) * 2000-08-04 2004-09-21 Infoglide Corporation System and method for building and maintaining a database
US6889297B2 (en) * 2001-03-23 2005-05-03 Sun Microsystems, Inc. Methods and systems for eliminating data redundancies
KR20060073724A (en) * 2004-12-24 2006-06-29 주식회사 나우콤 Method and apparatus for storing and downloading duplicated file using different file information
US20080243914A1 (en) * 2006-12-22 2008-10-02 Anand Prahlad System and method for storing redundant information
US20080294696A1 (en) * 2007-05-22 2008-11-27 Yuval Frandzel System and method for on-the-fly elimination of redundant data

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6795819B2 (en) * 2000-08-04 2004-09-21 Infoglide Corporation System and method for building and maintaining a database
KR20020066836A (en) * 2001-02-14 2002-08-21 한국통신정보기술 주식회사 Recording medium for recording hierarchical data structure and method for creating hierarchical data storage structure
US6889297B2 (en) * 2001-03-23 2005-05-03 Sun Microsystems, Inc. Methods and systems for eliminating data redundancies
KR20060073724A (en) * 2004-12-24 2006-06-29 주식회사 나우콤 Method and apparatus for storing and downloading duplicated file using different file information
US20080243914A1 (en) * 2006-12-22 2008-10-02 Anand Prahlad System and method for storing redundant information
US20080294696A1 (en) * 2007-05-22 2008-11-27 Yuval Frandzel System and method for on-the-fly elimination of redundant data

Also Published As

Publication number Publication date
AU2009206038A1 (en) 2009-07-23
JP2011510405A (en) 2011-03-31
EP2235640A2 (en) 2010-10-06
WO2009091957A2 (en) 2009-07-23
CA2711273A1 (en) 2009-07-23
CN101939737A (en) 2011-01-05

Similar Documents

Publication Publication Date Title
WO2009091957A3 (en) Scalable de-duplication mechanism
EP2174225A4 (en) Emulated storage system
WO2012083308A3 (en) Apparatus, system, and method for persistent data management on a non-volatile storage media
WO2007078395A3 (en) System and method for automatically transferring dynamically changing content
WO2008080143A3 (en) Method and system for searching stored data
WO2008115670A3 (en) System and method for identifying content
ATE438894T1 (en) RETURNING A FILE TO ITS PROPER STORAGE LEVEL IN AN INFORMATION LIFECYCLE MANAGEMENT ENVIRONMENT
WO2008085708A3 (en) Data backup system and method associated therewith
WO2007078566A3 (en) System and method for creating and utilizing metadata regarding the structure of program content stored on a dvr
WO2008013634A3 (en) File system replication
WO2010077972A3 (en) Method and apparatus to implement a hierarchical cache system with pnfs
WO2009134462A3 (en) Method and system to predict the likelihood of topics
WO2007030304A3 (en) Snapshot restore method and apparatus
WO2011041606A3 (en) Storage replication systems and methods
WO2007103932A3 (en) Coupon code systems and methods
DE602007005468D1 (en) METHOD AND SYSTEM FOR SCALABLE, DISTRIBUTED AND DIFFERENTIAL STORAGE AND ARCHIVING OF ELECTRONIC DATA
WO2008080140A3 (en) System and method for storing redundant information
WO2011031660A3 (en) Identifying at-risk data in non-volatile storage
WO2010080591A3 (en) Methods and apparatus for content-aware data partitioning and data de-duplication
WO2011071990A3 (en) Resource search operations
WO2008155188A3 (en) Firewall control using remote system information
WO2010037031A3 (en) System and method for aggregating web feeds relevant to a geographical locale from multiple sources
WO2008019259A3 (en) Architecture for back up and/or recovery of electronic data
WO2008013894A3 (en) Signal continuity assessment using embedded watermarks
WO2011113042A3 (en) Distributed catalog, data store, and indexing

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 200980101696.4

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 09702041

Country of ref document: EP

Kind code of ref document: A2

WWE Wipo information: entry into national phase

Ref document number: 2009206038

Country of ref document: AU

ENP Entry into the national phase

Ref document number: 2009206038

Country of ref document: AU

Date of ref document: 20090116

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 2009702041

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 2711273

Country of ref document: CA

WWE Wipo information: entry into national phase

Ref document number: 2010543270

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE