WO2009033074A3 - De-duplication in virtualized server and virtualized storage environments - Google Patents

De-duplication in virtualized server and virtualized storage environments Download PDF

Info

Publication number
WO2009033074A3
WO2009033074A3 PCT/US2008/075467 US2008075467W WO2009033074A3 WO 2009033074 A3 WO2009033074 A3 WO 2009033074A3 US 2008075467 W US2008075467 W US 2008075467W WO 2009033074 A3 WO2009033074 A3 WO 2009033074A3
Authority
WO
WIPO (PCT)
Prior art keywords
storage
data
virtualized
duplication
storage capacity
Prior art date
Application number
PCT/US2008/075467
Other languages
French (fr)
Other versions
WO2009033074A2 (en
Inventor
Jedidiah Yueh
Original Assignee
Emc Corp
Jedidiah Yueh
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US11/864,756 external-priority patent/US8880797B2/en
Priority claimed from US11/864,583 external-priority patent/US8209506B2/en
Application filed by Emc Corp, Jedidiah Yueh filed Critical Emc Corp
Priority to EP20080829858 priority Critical patent/EP2186015A4/en
Priority to CN2008801058233A priority patent/CN101809559B/en
Publication of WO2009033074A2 publication Critical patent/WO2009033074A2/en
Publication of WO2009033074A3 publication Critical patent/WO2009033074A3/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/174—Redundancy elimination performed by the file system
    • G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
    • G06F16/1752—De-duplication implemented within the file system, e.g. based on file segments based on file chunks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G06F3/0641—De-duplication techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A data de-duplication application de-duplicates data on the primary storage read/write pathway of a virtualized server environment and/or in pooled storage capacity of a virtualized storage environment. A virtualized server environment includes multiple server applications operating on a virtualization layer provided on a computer architecture that includes memory for temporarily storing data and storage for persistently storing data. A virtualized storage environment includes multiple storage devices and a virtualization layer that aggregates all or a portion of the storage capacity of each storage device into a single pool of storage capacity. In the virtualized environments, the de- duplication application identifies redundant data in memory, storage, and/or pooled storage capacity and replaces the redundant data with one or more pointers pointing to a single copy of the data. The de-duplication application operates on fixed or variable size blocks of data and de-duplicates data either post-process or in-line.
PCT/US2008/075467 2007-09-05 2008-09-05 De-duplication in virtualized server and virtualized storage environments WO2009033074A2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP20080829858 EP2186015A4 (en) 2007-09-05 2008-09-05 De-duplication in virtualized server and virtualized storage environments
CN2008801058233A CN101809559B (en) 2007-09-05 2008-09-05 De-duplication in virtualized server and virtualized storage environments

Applications Claiming Priority (8)

Application Number Priority Date Filing Date Title
US97018107P 2007-09-05 2007-09-05
US97018707P 2007-09-05 2007-09-05
US60/970,181 2007-09-05
US60/970,187 2007-09-05
US11/864,583 2007-09-28
US11/864,756 US8880797B2 (en) 2007-09-05 2007-09-28 De-duplication in a virtualized server environment
US11/864,583 US8209506B2 (en) 2007-09-05 2007-09-28 De-duplication in a virtualized storage environment
US11/864,756 2007-09-28

Publications (2)

Publication Number Publication Date
WO2009033074A2 WO2009033074A2 (en) 2009-03-12
WO2009033074A3 true WO2009033074A3 (en) 2009-05-14

Family

ID=40429720

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2008/075467 WO2009033074A2 (en) 2007-09-05 2008-09-05 De-duplication in virtualized server and virtualized storage environments

Country Status (3)

Country Link
EP (1) EP2186015A4 (en)
CN (2) CN101809559B (en)
WO (1) WO2009033074A2 (en)

Families Citing this family (31)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8209506B2 (en) 2007-09-05 2012-06-26 Emc Corporation De-duplication in a virtualized storage environment
US8380681B2 (en) * 2010-12-16 2013-02-19 Microsoft Corporation Extensible pipeline for data deduplication
WO2013081637A2 (en) * 2010-12-29 2013-06-06 Amazon Technologies, Inc. Receiver-side data deduplication in data systems
CN102221982B (en) * 2011-06-13 2013-09-11 北京卓微天成科技咨询有限公司 Method and system for implementing deletion of repeated data on block-level virtual storage equipment
CN102223409B (en) * 2011-06-13 2013-08-21 浪潮(北京)电子信息产业有限公司 Network storage resource application system and method
US8468138B1 (en) * 2011-12-02 2013-06-18 International Business Machines Corporation Managing redundant immutable files using deduplication in storage clouds
US9235589B2 (en) * 2011-12-13 2016-01-12 International Business Machines Corporation Optimizing storage allocation in a virtual desktop environment
US9417811B2 (en) 2012-03-07 2016-08-16 International Business Machines Corporation Efficient inline data de-duplication on a storage system
US8923195B2 (en) * 2012-03-20 2014-12-30 Futurewei Technologies, Inc. Method and apparatus for efficient content delivery in radio access networks
CN104364774B (en) * 2012-04-27 2017-10-20 不列颠哥伦比亚大学 Deduplication virtual machine image translator
US9104328B2 (en) 2012-10-31 2015-08-11 Hitachi, Ltd. Storage apparatus and method for controlling storage apparatus
GB2510185A (en) 2013-01-29 2014-07-30 Ibm Data de-duplication between emulated disk sub-systems
US9729659B2 (en) 2013-03-14 2017-08-08 Microsoft Technology Licensing, Llc Caching content addressable data chunks for storage virtualization
WO2014185916A1 (en) 2013-05-16 2014-11-20 Hewlett-Packard Development Company, L.P. Selecting a store for deduplicated data
WO2014185918A1 (en) * 2013-05-16 2014-11-20 Hewlett-Packard Development Company, L.P. Selecting a store for deduplicated data
CN103559282B (en) * 2013-11-07 2018-02-23 北京国双科技有限公司 The De-weight method and device of real-time system data
EP3126984A4 (en) 2014-04-03 2017-10-11 Strato Scale Ltd. Cluster-wide memory management using similarity-preserving signatures
US20150286414A1 (en) * 2014-04-03 2015-10-08 Strato Scale Ltd. Scanning memory for de-duplication using rdma
CN103942292A (en) * 2014-04-11 2014-07-23 华为技术有限公司 Virtual machine mirror image document processing method, device and system
WO2016003454A1 (en) 2014-07-02 2016-01-07 Hewlett-Packard Development Company, L.P. Managing port connections
CN104133888B (en) * 2014-07-30 2019-08-02 宇龙计算机通信科技(深圳)有限公司 A kind of multisystem data processing method, device and terminal
EP3195135A4 (en) * 2014-09-05 2018-05-02 Hewlett-Packard Enterprise Development LP Data storage over fibre channel
AU2014403332B2 (en) * 2014-09-15 2017-04-20 Huawei Technologies Co., Ltd. Data deduplication method and storage array
US9390028B2 (en) 2014-10-19 2016-07-12 Strato Scale Ltd. Coordination between memory-saving mechanisms in computers that run virtual machines
US9912748B2 (en) 2015-01-12 2018-03-06 Strato Scale Ltd. Synchronization of snapshots in a distributed storage system
EP3126987A4 (en) 2015-02-26 2017-11-22 Strato Scale Ltd. Using access-frequency hierarchy for selection of eviction destination
CN107515723B (en) * 2016-06-16 2020-04-24 伊姆西Ip控股有限责任公司 Method and system for managing memory in a storage system
CN107870922B (en) * 2016-09-23 2022-02-22 伊姆西Ip控股有限责任公司 Method, equipment and system for data deduplication
TWI663515B (en) * 2017-07-18 2019-06-21 先智雲端數據股份有限公司 Storage system of distributed deduplication for internet of things backup in data center and method for achieving the same
US11467775B2 (en) * 2019-10-15 2022-10-11 Hewlett Packard Enterprise Development Lp Virtual persistent volumes for containerized applications
CN111209229B (en) * 2019-12-30 2021-12-21 苏州艾利特机器人有限公司 Fieldbus method based on virtual equipment

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20050088067A (en) * 2002-07-11 2005-09-01 베리타스 오퍼레이팅 코포레이션 Storage services and systems
KR20060042989A (en) * 2004-10-28 2006-05-15 후지쯔 가부시끼가이샤 Program, method and apparatus for virtual storage management
KR20060044567A (en) * 2004-11-09 2006-05-16 후지쯔 가부시끼가이샤 Storage virtualization apparatus
KR20070086325A (en) * 2004-12-10 2007-08-27 인텔 코오퍼레이션 Method and apparatus for providing virtual server blades

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6075938A (en) * 1997-06-10 2000-06-13 The Board Of Trustees Of The Leland Stanford Junior University Virtual machine monitors for scalable multiprocessors
US6374266B1 (en) * 1998-07-28 2002-04-16 Ralph Shnelvar Method and apparatus for storing information in a data processing system
US6389433B1 (en) * 1999-07-16 2002-05-14 Microsoft Corporation Method and system for automatically merging files into a single instance store
US6789156B1 (en) * 2001-05-22 2004-09-07 Vmware, Inc. Content-based, transparent sharing of memory units
JP2006528862A (en) * 2003-07-24 2006-12-21 コーニンクレッカ　フィリップス　エレクトロニクス　エヌ　ヴィ Optimizing stored video data
US20050081099A1 (en) * 2003-10-09 2005-04-14 International Business Machines Corporation Method and apparatus for ensuring valid journaled file system metadata during a backup operation
US20070050423A1 (en) * 2005-08-30 2007-03-01 Scentric, Inc. Intelligent general duplicate management system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20050088067A (en) * 2002-07-11 2005-09-01 베리타스 오퍼레이팅 코포레이션 Storage services and systems
KR20060042989A (en) * 2004-10-28 2006-05-15 후지쯔 가부시끼가이샤 Program, method and apparatus for virtual storage management
KR20060044567A (en) * 2004-11-09 2006-05-16 후지쯔 가부시끼가이샤 Storage virtualization apparatus
KR20070086325A (en) * 2004-12-10 2007-08-27 인텔 코오퍼레이션 Method and apparatus for providing virtual server blades

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
BAK YOUNG JIM.: "Data Writing of Deduplication System.", IT BUSINESS LEADER'S MAGAZINE. SEOUL:KYUNGCOM, vol. 369, July 2007 (2007-07-01), pages 134 - 137, XP008168432 *
See also references of EP2186015A4 *

Also Published As

Publication number Publication date
WO2009033074A2 (en) 2009-03-12
EP2186015A4 (en) 2015-04-29
CN102880626B (en) 2016-02-10
EP2186015A2 (en) 2010-05-19
CN102880626A (en) 2013-01-16
CN101809559B (en) 2013-10-16
CN101809559A (en) 2010-08-18

Similar Documents

Publication Publication Date Title
WO2009033074A3 (en) De-duplication in virtualized server and virtualized storage environments
US10031675B1 (en) Method and system for tiering data
US9952769B2 (en) Data storage system with data storage devices operative to manage storage device functions specific to a particular data storage device
GB2449521B (en) Foresight data transfer type hierarchical storage system
US8495350B2 (en) Running operating system on dynamic virtual memory
CN104021090B (en) Integrated circuit and its operating method and system including integrated circuit
US20170344430A1 (en) Method and apparatus for data checkpointing and restoration in a storage device
KR20170113013A (en) Multi-ware smart ssd
CN102609360A (en) Data processing method, data processing device and data processing system
US20100070544A1 (en) Virtual block-level storage over a file system
JP2008152807A5 (en)
WO2005111804A3 (en) Extension of write anywhere file system layout
WO2010080591A3 (en) Methods and apparatus for content-aware data partitioning and data de-duplication
TWI696188B (en) Hybrid memory system
WO2007021435A3 (en) Archiving data in a virtual application environment
US20170206170A1 (en) Reducing a size of a logical to physical data address translation table
US20100262755A1 (en) Memory systems for computing devices and systems
US9678871B2 (en) Data flush of group table
DE602008004088D1 (en) COMPUTER STORAGE SYSTEM
CN107577492A (en) The NVM block device drives method and system of accelerating file system read-write
CN114556313A (en) Memory mapping device and method
US11561720B2 (en) Enabling access to a partially migrated dataset
US9323617B2 (en) Remap raid to maintain raid level
US20190324868A1 (en) Backup portion of persistent memory
Song et al. Enhanced flash swap: Using NAND flash as a swap device with lifetime control

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 200880105823.3

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 08829858

Country of ref document: EP

Kind code of ref document: A2

WWE Wipo information: entry into national phase

Ref document number: 2008829858

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 1442/DELNP/2010

Country of ref document: IN

NENP Non-entry into the national phase

Ref country code: DE