KR20180052739A - 고체 상태 드라이브 컨트롤러를 사용한 데이터 중복 제거 - Google Patents

고체 상태 드라이브 컨트롤러를 사용한 데이터 중복 제거 Download PDF

Info

Publication number
KR20180052739A
KR20180052739A KR1020187010549A KR20187010549A KR20180052739A KR 20180052739 A KR20180052739 A KR 20180052739A KR 1020187010549 A KR1020187010549 A KR 1020187010549A KR 20187010549 A KR20187010549 A KR 20187010549A KR 20180052739 A KR20180052739 A KR 20180052739A
Authority
KR
South Korea
Prior art keywords
signature
data
information
block
controller
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
KR1020187010549A
Other languages
English (en)
Korean (ko)
Inventor
슈 리
용 리
궁뱌오 뉴
Original Assignee
알리바바 그룹 홀딩 리미티드
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 알리바바 그룹 홀딩 리미티드 filed Critical 알리바바 그룹 홀딩 리미티드
Publication of KR20180052739A publication Critical patent/KR20180052739A/ko
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G06F3/0641—De-duplication techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/38—Information transfer, e.g. on bus
    • G06F13/42—Bus transfer protocol, e.g. handshake; Synchronisation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G06F3/0611—Improving I/O performance in relation to response time
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2213/00—Indexing scheme relating to interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F2213/0032—Serial ATA [SATA]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
KR1020187010549A 2015-09-18 2016-09-16 고체 상태 드라이브 컨트롤러를 사용한 데이터 중복 제거 Withdrawn KR20180052739A (ko)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US14/858,257 2015-09-18
US14/858,257 US9665287B2 (en) 2015-09-18 2015-09-18 Data deduplication using a solid state drive controller
PCT/US2016/052222 WO2017049142A1 (en) 2015-09-18 2016-09-16 Data deduplication using a solid state drive controller

Publications (1)

Publication Number Publication Date
KR20180052739A true KR20180052739A (ko) 2018-05-18

Family

ID=58282696

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020187010549A Withdrawn KR20180052739A (ko) 2015-09-18 2016-09-16 고체 상태 드라이브 컨트롤러를 사용한 데이터 중복 제거

Country Status (6)

Country Link
US (2) US9665287B2 (enExample)
EP (1) EP3350683B1 (enExample)
JP (1) JP2018527681A (enExample)
KR (1) KR20180052739A (enExample)
CN (1) CN108027713B (enExample)
WO (1) WO2017049142A1 (enExample)

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9665287B2 (en) 2015-09-18 2017-05-30 Alibaba Group Holding Limited Data deduplication using a solid state drive controller
CN107402725B (zh) * 2017-03-20 2020-08-25 威盛电子股份有限公司 非易失性存储装置及其数据去重复方法
US10318202B2 (en) * 2017-03-20 2019-06-11 Via Technologies, Inc. Non-volatile memory apparatus and data deduplication method thereof
CN109669623B (zh) * 2017-10-13 2021-09-03 杭州海康威视系统技术有限公司 一种文件管理方法、文件管理装置、电子设备及存储介质
CN108920964B (zh) * 2018-06-21 2020-09-29 深圳忆联信息系统有限公司 可重构硬件加解密方法、系统、计算机设备及存储介质
CN109062514B (zh) * 2018-08-16 2021-08-31 郑州云海信息技术有限公司 一种基于命名空间的带宽控制方法、装置和存储介质
CN110968537B (zh) * 2018-09-28 2021-02-02 方一信息科技（上海）有限公司 一种基于pcie ssd的fpga搜索匹配方法
US11029874B2 (en) 2019-07-30 2021-06-08 Western Digital Technologies, Inc. Rolling XOR protection in efficient pipeline
KR102810527B1 (ko) 2019-09-23 2025-05-22 삼성전자주식회사 스토리지 장치 및 그것의 동작 방법
CN113365282B (zh) * 2021-06-22 2023-04-07 成都信息工程大学 一种wsn障碍性区域覆盖部署方法
US12007968B2 (en) 2022-05-26 2024-06-11 International Business Machines Corporation Full allocation volume to deduplication volume migration in a storage system
WO2025240883A1 (en) * 2024-05-17 2025-11-20 Micron Technology, Inc. Data de-duplication using content-addressable memory

Family Cites Families (73)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7636767B2 (en) 2005-11-29 2009-12-22 Cisco Technology, Inc. Method and apparatus for reducing network traffic over low bandwidth links
US8412682B2 (en) 2006-06-29 2013-04-02 Netapp, Inc. System and method for retrieving and using block fingerprints for data deduplication
US7840537B2 (en) 2006-12-22 2010-11-23 Commvault Systems, Inc. System and method for storing redundant information
US7761425B1 (en) 2007-03-29 2010-07-20 Symantec Corporation Low-overhead means of performing data backup
US20090132616A1 (en) 2007-10-02 2009-05-21 Richard Winter Archival backup integration
US7962452B2 (en) 2007-12-28 2011-06-14 International Business Machines Corporation Data deduplication by separating data from meta data
US8825617B2 (en) 2008-03-14 2014-09-02 International Business Machines Corporation Limiting deduplication based on predetermined criteria
US7567188B1 (en) 2008-04-10 2009-07-28 International Business Machines Corporation Policy based tiered data deduplication strategy
WO2009149059A1 (en) * 2008-06-04 2009-12-10 Initio Corporation Ssd with a controller accelerator
US7814149B1 (en) 2008-09-29 2010-10-12 Symantec Operating Corporation Client side data deduplication
WO2010045262A1 (en) 2008-10-14 2010-04-22 Wanova Technologies, Ltd. Storage-network de-duplication
US8082228B2 (en) * 2008-10-31 2011-12-20 Netapp, Inc. Remote office duplication
JP5444728B2 (ja) * 2009-01-26 2014-03-19 日本電気株式会社 ストレージシステム、ストレージシステムにおけるデータ書込方法及びデータ書込プログラム
US8812874B1 (en) 2009-03-31 2014-08-19 Symantec Corporation Content deduplication in enterprise rights management
US8407186B1 (en) 2009-03-31 2013-03-26 Symantec Corporation Systems and methods for data-selection-specific data deduplication
US8281066B1 (en) * 2009-04-30 2012-10-02 Netapp, Inc. System and method for real-time deduplication utilizing an electronic storage medium
CN101882141A (zh) * 2009-05-08 2010-11-10 北京众志和达信息技术有限公司 一种实现重复数据数据删除的方法和系统
US8442954B2 (en) 2009-07-21 2013-05-14 Stephen Philip SPACKMAN Creating and managing links to deduplication information
US8204867B2 (en) 2009-07-29 2012-06-19 International Business Machines Corporation Apparatus, system, and method for enhanced block-level deduplication
US20110093439A1 (en) 2009-10-16 2011-04-21 Fanglu Guo De-duplication Storage System with Multiple Indices for Efficient File Storage
US8321648B2 (en) 2009-10-26 2012-11-27 Netapp, Inc Use of similarity hash to route data for improved deduplication in a storage server cluster
US8478933B2 (en) 2009-11-24 2013-07-02 International Business Machines Corporation Systems and methods for performing deduplicated data processing on tape
US8407193B2 (en) 2010-01-27 2013-03-26 International Business Machines Corporation Data deduplication for streaming sequential data storage applications
JP5434705B2 (ja) 2010-03-12 2014-03-05 富士通株式会社 ストレージ装置、ストレージ装置制御プログラムおよびストレージ装置制御方法
US8370593B2 (en) 2010-04-14 2013-02-05 Hitachi, Ltd. Method and apparatus to manage groups for deduplication
US9047301B2 (en) 2010-04-19 2015-06-02 Greenbytes, Inc. Method for optimizing the memory usage and performance of data deduplication storage systems
US8639658B1 (en) 2010-04-21 2014-01-28 Symantec Corporation Cache management for file systems supporting shared blocks
US9053032B2 (en) 2010-05-05 2015-06-09 Microsoft Technology Licensing, Llc Fast and low-RAM-footprint indexing for data deduplication
US8935487B2 (en) 2010-05-05 2015-01-13 Microsoft Corporation Fast and low-RAM-footprint indexing for data deduplication
US8489855B2 (en) 2010-05-07 2013-07-16 Ocz Technology Group Inc. NAND flash-based solid state drive and method of operation
US20120173795A1 (en) * 2010-05-25 2012-07-05 Ocz Technology Group, Inc. Solid state drive with low write amplification
US8370315B1 (en) 2010-05-28 2013-02-05 Symantec Corporation System and method for high performance deduplication indexing
US9092151B1 (en) 2010-09-17 2015-07-28 Permabit Technology Corporation Managing deduplication of stored data
US10162553B2 (en) * 2010-11-24 2018-12-25 Western Digital Technologies, Inc. Methods and systems for object level de-duplication for solid state devices
US8898119B2 (en) 2010-12-15 2014-11-25 Netapp, Inc. Fingerprints datastore and stale fingerprint removal in de-duplication environments
US8332372B2 (en) 2010-12-16 2012-12-11 International Business Machines Corporation Method and system for processing data
US8495304B1 (en) 2010-12-23 2013-07-23 Emc Corporation Multi source wire deduplication
US9116909B2 (en) * 2010-12-29 2015-08-25 Amazon Technologies, Inc. Reduced bandwidth data uploading in data systems
CA2823134C (en) * 2010-12-29 2020-07-07 Amazon Technologies, Inc. Receiver-side data deduplication in data systems
US9223511B2 (en) 2011-04-08 2015-12-29 Micron Technology, Inc. Data deduplication
US8600949B2 (en) 2011-06-21 2013-12-03 Netapp, Inc. Deduplication in an extent-based architecture
US8589640B2 (en) * 2011-10-14 2013-11-19 Pure Storage, Inc. Method for maintaining multiple fingerprint tables in a deduplicating storage system
US8533231B2 (en) 2011-08-12 2013-09-10 Nexenta Systems, Inc. Cloud storage system with distributed metadata
US8484170B2 (en) 2011-09-19 2013-07-09 International Business Machines Corporation Scalable deduplication system with small blocks
US8620886B1 (en) * 2011-09-20 2013-12-31 Netapp Inc. Host side deduplication
CN103034659B (zh) 2011-09-29 2015-08-19 国际商业机器公司 一种重复数据删除的方法和系统
US8898120B1 (en) 2011-10-09 2014-11-25 Symantec Corporation Systems and methods for distributed data deduplication
US8572312B2 (en) 2011-12-07 2013-10-29 Jeffrey Tofano Data de-duplication and solid state memory device
KR20130064518A (ko) * 2011-12-08 2013-06-18 삼성전자주식회사 저장 장치 및 그것의 동작 방법
CN102646069B (zh) * 2012-02-23 2014-12-10 华中科技大学 一种延长固态盘使用寿命的方法
US20130282672A1 (en) * 2012-04-18 2013-10-24 Hitachi Computer Peripherals Co., Ltd. Storage apparatus and storage control method
US9177028B2 (en) 2012-04-30 2015-11-03 International Business Machines Corporation Deduplicating storage with enhanced frequent-block detection
US8930648B1 (en) 2012-05-23 2015-01-06 Netapp, Inc. Distributed deduplication using global chunk data structure and epochs
US8788468B2 (en) 2012-05-24 2014-07-22 International Business Machines Corporation Data depulication using short term history
US8930612B2 (en) 2012-05-31 2015-01-06 Seagate Technology Llc Background deduplication of data sets in a memory
CN102981969A (zh) * 2012-11-21 2013-03-20 记忆科技(深圳)有限公司 重复数据删除的方法及其固态硬盘
JP5774794B2 (ja) * 2012-12-05 2015-09-09 株式会社日立製作所 ストレージシステム及びストレージシステムの制御方法
US8935222B2 (en) 2013-01-02 2015-01-13 International Business Machines Corporation Optimizing a partition in data deduplication
US9219784B2 (en) * 2013-03-07 2015-12-22 International Business Machines Corporation Synchronization of a server side deduplication cache with a client side deduplication cache
US9116941B2 (en) 2013-03-15 2015-08-25 International Business Machines Corporation Reducing digest storage consumption by tracking similarity elements in a data deduplication system
US9244937B2 (en) 2013-03-15 2016-01-26 International Business Machines Corporation Efficient calculation of similarity search values and digest block boundaries for data deduplication
CN104246722B (zh) * 2013-03-29 2017-02-22 株式会社东芝 用于基于哈希表排除数据重复的存储系统，存储控制器及方法
US20140304464A1 (en) 2013-04-03 2014-10-09 Lsi Corporation Methods and systems for performing deduplication in a data storage system
CN103473266A (zh) * 2013-08-09 2013-12-25 记忆科技(深圳)有限公司 固态硬盘及其删除重复数据的方法
GB2518158A (en) * 2013-09-11 2015-03-18 Ibm Method and system for data access in a storage infrastructure
KR20150067583A (ko) * 2013-12-10 2015-06-18 삼성전자주식회사 불휘발성 메모리 장치 및 그것의 중복 데이터 제거 방법
US10380072B2 (en) * 2014-03-17 2019-08-13 Commvault Systems, Inc. Managing deletions from a deduplication database
JP6307624B2 (ja) * 2014-05-30 2018-04-04 株式会社日立製作所 データ重複排除ストレージシステムの方法及び装置
CN104407982B (zh) 2014-11-19 2018-09-21 湖南国科微电子股份有限公司 一种ssd盘片垃圾回收方法
US10416915B2 (en) * 2015-05-15 2019-09-17 ScaleFlux Assisting data deduplication through in-memory computation
US20170017571A1 (en) * 2015-07-17 2017-01-19 Samsung Electronics Co., Ltd. Method and apparatus fori n-line deduplication in storage devices
US9665287B2 (en) 2015-09-18 2017-05-30 Alibaba Group Holding Limited Data deduplication using a solid state drive controller
JP6067819B1 (ja) * 2015-10-21 2017-01-25 株式会社東芝 階層化ストレージシステム、ストレージコントローラ、並びに重複排除及びストレージ階層化のための方法

Also Published As

Publication number Publication date
WO2017049142A1 (en) 2017-03-23
CN108027713B (zh) 2021-10-12
US20170242616A1 (en) 2017-08-24
US9864542B2 (en) 2018-01-09
US9665287B2 (en) 2017-05-30
US20170083245A1 (en) 2017-03-23
JP2018527681A (ja) 2018-09-20
CN108027713A (zh) 2018-05-11
EP3350683B1 (en) 2023-05-10
EP3350683A4 (en) 2019-04-24
EP3350683A1 (en) 2018-07-25

Similar Documents

Publication Publication Date Title
US9864542B2 (en) Data deduplication using a solid state drive controller
US9405684B1 (en) System and method for cache management
US9740403B2 (en) Methods for managing storage in a data storage cluster with distributed zones based on parity values and devices thereof
US10545833B1 (en) Block-level deduplication
US8683156B2 (en) Format-preserving deduplication of data
US10339112B1 (en) Restoring data in deduplicated storage
US10019323B1 (en) Method and system for container data recovery in a storage system
US9619478B1 (en) Method and system for compressing logs
US20130110784A1 (en) Managing backups of data objects in containers
CN115114059A (zh) 使用区来管理归因于存储装置故障的容量减小
US9959049B1 (en) Aggregated background processing in a data storage system to improve system resource utilization
US10846301B1 (en) Container reclamation using probabilistic data structures
CN115114055B (zh) 管理归因于存储装置故障的容量减小和恢复
US10606499B2 (en) Computer system, storage apparatus, and method of managing data
US10255288B2 (en) Distributed data deduplication in a grid of processors
CN115114054B (zh) 管理发生故障的多层级存储器单元的存储空间减小和再用
CN115114061A (zh) 管理归因于存储装置故障的容量减小
US9619336B2 (en) Managing production data
US11132137B2 (en) Methods and systems for providing read-optimized scalable offline de-duplication for blocks of data
US11068208B2 (en) Capacity reduction in a storage system
US11366795B2 (en) System and method for generating bitmaps of metadata changes
US11163748B1 (en) Fingerprint backward compatibility in deduplication backup systems
US9626121B2 (en) De-duplication as part of other routinely performed processes
US10922003B1 (en) Realizing host-assisted device-level data deduplication on solid-state data storage devices with embedded non-volatile memory
US20210318986A1 (en) Using prefix-delete operations for data containers

Legal Events

Date Code Title Description
PA0105 International application

Patent event date: 20180413

Patent event code: PA01051R01D

Comment text: International Patent Application

PG1501 Laying open of application
PC1203 Withdrawal of no request for examination