CN108027713B - 用于固态驱动器控制器的重复数据删除 - Google Patents

用于固态驱动器控制器的重复数据删除 Download PDF

Info

Publication number
CN108027713B
CN108027713B CN201680054387.6A CN201680054387A CN108027713B CN 108027713 B CN108027713 B CN 108027713B CN 201680054387 A CN201680054387 A CN 201680054387A CN 108027713 B CN108027713 B CN 108027713B
Authority
CN
China
Prior art keywords
signature
information
ssd
controller
library
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201680054387.6A
Other languages
English (en)
Chinese (zh)
Other versions
CN108027713A (zh
Inventor
李舒
李勇
牛功彪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Publication of CN108027713A publication Critical patent/CN108027713A/zh
Application granted granted Critical
Publication of CN108027713B publication Critical patent/CN108027713B/zh
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G06F3/0641—De-duplication techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/38—Information transfer, e.g. on bus
    • G06F13/42—Bus transfer protocol, e.g. handshake; Synchronisation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G06F3/0611—Improving I/O performance in relation to response time
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2213/00—Indexing scheme relating to interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F2213/0032—Serial ATA [SATA]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
CN201680054387.6A 2015-09-18 2016-09-16 用于固态驱动器控制器的重复数据删除 Active CN108027713B (zh)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US14/858257 2015-09-18
US14/858,257 US9665287B2 (en) 2015-09-18 2015-09-18 Data deduplication using a solid state drive controller
PCT/US2016/052222 WO2017049142A1 (en) 2015-09-18 2016-09-16 Data deduplication using a solid state drive controller

Publications (2)

Publication Number Publication Date
CN108027713A CN108027713A (zh) 2018-05-11
CN108027713B true CN108027713B (zh) 2021-10-12

Family

ID=58282696

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201680054387.6A Active CN108027713B (zh) 2015-09-18 2016-09-16 用于固态驱动器控制器的重复数据删除

Country Status (6)

Country Link
US (2) US9665287B2 (enExample)
EP (1) EP3350683B1 (enExample)
JP (1) JP2018527681A (enExample)
KR (1) KR20180052739A (enExample)
CN (1) CN108027713B (enExample)
WO (1) WO2017049142A1 (enExample)

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9665287B2 (en) 2015-09-18 2017-05-30 Alibaba Group Holding Limited Data deduplication using a solid state drive controller
CN107402725B (zh) * 2017-03-20 2020-08-25 威盛电子股份有限公司 非易失性存储装置及其数据去重复方法
US10318202B2 (en) * 2017-03-20 2019-06-11 Via Technologies, Inc. Non-volatile memory apparatus and data deduplication method thereof
CN109669623B (zh) * 2017-10-13 2021-09-03 杭州海康威视系统技术有限公司 一种文件管理方法、文件管理装置、电子设备及存储介质
CN108920964B (zh) * 2018-06-21 2020-09-29 深圳忆联信息系统有限公司 可重构硬件加解密方法、系统、计算机设备及存储介质
CN109062514B (zh) * 2018-08-16 2021-08-31 郑州云海信息技术有限公司 一种基于命名空间的带宽控制方法、装置和存储介质
CN110968537B (zh) * 2018-09-28 2021-02-02 方一信息科技（上海）有限公司 一种基于pcie ssd的fpga搜索匹配方法
US11029874B2 (en) 2019-07-30 2021-06-08 Western Digital Technologies, Inc. Rolling XOR protection in efficient pipeline
KR102810527B1 (ko) 2019-09-23 2025-05-22 삼성전자주식회사 스토리지 장치 및 그것의 동작 방법
CN113365282B (zh) * 2021-06-22 2023-04-07 成都信息工程大学 一种wsn障碍性区域覆盖部署方法
US12007968B2 (en) 2022-05-26 2024-06-11 International Business Machines Corporation Full allocation volume to deduplication volume migration in a storage system
WO2025240883A1 (en) * 2024-05-17 2025-11-20 Micron Technology, Inc. Data de-duplication using content-addressable memory

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TW201007452A (en) * 2008-06-04 2010-02-16 Initio Corp SSD with a controller accelerator
US7761425B1 (en) * 2007-03-29 2010-07-20 Symantec Corporation Low-overhead means of performing data backup
CN101882141A (zh) * 2009-05-08 2010-11-10 北京众志和达信息技术有限公司 一种实现重复数据数据删除的方法和系统
CN102591947A (zh) * 2010-12-28 2012-07-18 微软公司 用于数据去重复的快速且低ram占用的索引
CN103473266A (zh) * 2013-08-09 2013-12-25 记忆科技(深圳)有限公司 固态硬盘及其删除重复数据的方法
CN103547991A (zh) * 2010-12-29 2014-01-29 亚马逊科技公司 数据系统中的接收器侧数据重复删除

Family Cites Families (67)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7636767B2 (en) 2005-11-29 2009-12-22 Cisco Technology, Inc. Method and apparatus for reducing network traffic over low bandwidth links
US8412682B2 (en) 2006-06-29 2013-04-02 Netapp, Inc. System and method for retrieving and using block fingerprints for data deduplication
US7840537B2 (en) 2006-12-22 2010-11-23 Commvault Systems, Inc. System and method for storing redundant information
US20090132616A1 (en) 2007-10-02 2009-05-21 Richard Winter Archival backup integration
US7962452B2 (en) 2007-12-28 2011-06-14 International Business Machines Corporation Data deduplication by separating data from meta data
US8825617B2 (en) 2008-03-14 2014-09-02 International Business Machines Corporation Limiting deduplication based on predetermined criteria
US7567188B1 (en) 2008-04-10 2009-07-28 International Business Machines Corporation Policy based tiered data deduplication strategy
US7814149B1 (en) 2008-09-29 2010-10-12 Symantec Operating Corporation Client side data deduplication
WO2010045262A1 (en) 2008-10-14 2010-04-22 Wanova Technologies, Ltd. Storage-network de-duplication
US8082228B2 (en) * 2008-10-31 2011-12-20 Netapp, Inc. Remote office duplication
JP5444728B2 (ja) * 2009-01-26 2014-03-19 日本電気株式会社 ストレージシステム、ストレージシステムにおけるデータ書込方法及びデータ書込プログラム
US8812874B1 (en) 2009-03-31 2014-08-19 Symantec Corporation Content deduplication in enterprise rights management
US8407186B1 (en) 2009-03-31 2013-03-26 Symantec Corporation Systems and methods for data-selection-specific data deduplication
US8281066B1 (en) * 2009-04-30 2012-10-02 Netapp, Inc. System and method for real-time deduplication utilizing an electronic storage medium
US8442954B2 (en) 2009-07-21 2013-05-14 Stephen Philip SPACKMAN Creating and managing links to deduplication information
US8204867B2 (en) 2009-07-29 2012-06-19 International Business Machines Corporation Apparatus, system, and method for enhanced block-level deduplication
US20110093439A1 (en) 2009-10-16 2011-04-21 Fanglu Guo De-duplication Storage System with Multiple Indices for Efficient File Storage
US8321648B2 (en) 2009-10-26 2012-11-27 Netapp, Inc Use of similarity hash to route data for improved deduplication in a storage server cluster
US8478933B2 (en) 2009-11-24 2013-07-02 International Business Machines Corporation Systems and methods for performing deduplicated data processing on tape
US8407193B2 (en) 2010-01-27 2013-03-26 International Business Machines Corporation Data deduplication for streaming sequential data storage applications
JP5434705B2 (ja) 2010-03-12 2014-03-05 富士通株式会社 ストレージ装置、ストレージ装置制御プログラムおよびストレージ装置制御方法
US8370593B2 (en) 2010-04-14 2013-02-05 Hitachi, Ltd. Method and apparatus to manage groups for deduplication
US9047301B2 (en) 2010-04-19 2015-06-02 Greenbytes, Inc. Method for optimizing the memory usage and performance of data deduplication storage systems
US8639658B1 (en) 2010-04-21 2014-01-28 Symantec Corporation Cache management for file systems supporting shared blocks
US9053032B2 (en) 2010-05-05 2015-06-09 Microsoft Technology Licensing, Llc Fast and low-RAM-footprint indexing for data deduplication
US8489855B2 (en) 2010-05-07 2013-07-16 Ocz Technology Group Inc. NAND flash-based solid state drive and method of operation
US20120173795A1 (en) * 2010-05-25 2012-07-05 Ocz Technology Group, Inc. Solid state drive with low write amplification
US8370315B1 (en) 2010-05-28 2013-02-05 Symantec Corporation System and method for high performance deduplication indexing
US9092151B1 (en) 2010-09-17 2015-07-28 Permabit Technology Corporation Managing deduplication of stored data
US10162553B2 (en) * 2010-11-24 2018-12-25 Western Digital Technologies, Inc. Methods and systems for object level de-duplication for solid state devices
US8898119B2 (en) 2010-12-15 2014-11-25 Netapp, Inc. Fingerprints datastore and stale fingerprint removal in de-duplication environments
US8332372B2 (en) 2010-12-16 2012-12-11 International Business Machines Corporation Method and system for processing data
US8495304B1 (en) 2010-12-23 2013-07-23 Emc Corporation Multi source wire deduplication
US9116909B2 (en) * 2010-12-29 2015-08-25 Amazon Technologies, Inc. Reduced bandwidth data uploading in data systems
US9223511B2 (en) 2011-04-08 2015-12-29 Micron Technology, Inc. Data deduplication
US8600949B2 (en) 2011-06-21 2013-12-03 Netapp, Inc. Deduplication in an extent-based architecture
US8589640B2 (en) * 2011-10-14 2013-11-19 Pure Storage, Inc. Method for maintaining multiple fingerprint tables in a deduplicating storage system
US8533231B2 (en) 2011-08-12 2013-09-10 Nexenta Systems, Inc. Cloud storage system with distributed metadata
US8484170B2 (en) 2011-09-19 2013-07-09 International Business Machines Corporation Scalable deduplication system with small blocks
US8620886B1 (en) * 2011-09-20 2013-12-31 Netapp Inc. Host side deduplication
CN103034659B (zh) 2011-09-29 2015-08-19 国际商业机器公司 一种重复数据删除的方法和系统
US8898120B1 (en) 2011-10-09 2014-11-25 Symantec Corporation Systems and methods for distributed data deduplication
US8572312B2 (en) 2011-12-07 2013-10-29 Jeffrey Tofano Data de-duplication and solid state memory device
KR20130064518A (ko) * 2011-12-08 2013-06-18 삼성전자주식회사 저장 장치 및 그것의 동작 방법
CN102646069B (zh) * 2012-02-23 2014-12-10 华中科技大学 一种延长固态盘使用寿命的方法
US20130282672A1 (en) * 2012-04-18 2013-10-24 Hitachi Computer Peripherals Co., Ltd. Storage apparatus and storage control method
US9177028B2 (en) 2012-04-30 2015-11-03 International Business Machines Corporation Deduplicating storage with enhanced frequent-block detection
US8930648B1 (en) 2012-05-23 2015-01-06 Netapp, Inc. Distributed deduplication using global chunk data structure and epochs
US8788468B2 (en) 2012-05-24 2014-07-22 International Business Machines Corporation Data depulication using short term history
US8930612B2 (en) 2012-05-31 2015-01-06 Seagate Technology Llc Background deduplication of data sets in a memory
CN102981969A (zh) * 2012-11-21 2013-03-20 记忆科技(深圳)有限公司 重复数据删除的方法及其固态硬盘
JP5774794B2 (ja) * 2012-12-05 2015-09-09 株式会社日立製作所 ストレージシステム及びストレージシステムの制御方法
US8935222B2 (en) 2013-01-02 2015-01-13 International Business Machines Corporation Optimizing a partition in data deduplication
US9219784B2 (en) * 2013-03-07 2015-12-22 International Business Machines Corporation Synchronization of a server side deduplication cache with a client side deduplication cache
US9116941B2 (en) 2013-03-15 2015-08-25 International Business Machines Corporation Reducing digest storage consumption by tracking similarity elements in a data deduplication system
US9244937B2 (en) 2013-03-15 2016-01-26 International Business Machines Corporation Efficient calculation of similarity search values and digest block boundaries for data deduplication
CN104246722B (zh) * 2013-03-29 2017-02-22 株式会社东芝 用于基于哈希表排除数据重复的存储系统，存储控制器及方法
US20140304464A1 (en) 2013-04-03 2014-10-09 Lsi Corporation Methods and systems for performing deduplication in a data storage system
GB2518158A (en) * 2013-09-11 2015-03-18 Ibm Method and system for data access in a storage infrastructure
KR20150067583A (ko) * 2013-12-10 2015-06-18 삼성전자주식회사 불휘발성 메모리 장치 및 그것의 중복 데이터 제거 방법
US10380072B2 (en) * 2014-03-17 2019-08-13 Commvault Systems, Inc. Managing deletions from a deduplication database
JP6307624B2 (ja) * 2014-05-30 2018-04-04 株式会社日立製作所 データ重複排除ストレージシステムの方法及び装置
CN104407982B (zh) 2014-11-19 2018-09-21 湖南国科微电子股份有限公司 一种ssd盘片垃圾回收方法
US10416915B2 (en) * 2015-05-15 2019-09-17 ScaleFlux Assisting data deduplication through in-memory computation
US20170017571A1 (en) * 2015-07-17 2017-01-19 Samsung Electronics Co., Ltd. Method and apparatus fori n-line deduplication in storage devices
US9665287B2 (en) 2015-09-18 2017-05-30 Alibaba Group Holding Limited Data deduplication using a solid state drive controller
JP6067819B1 (ja) * 2015-10-21 2017-01-25 株式会社東芝 階層化ストレージシステム、ストレージコントローラ、並びに重複排除及びストレージ階層化のための方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7761425B1 (en) * 2007-03-29 2010-07-20 Symantec Corporation Low-overhead means of performing data backup
TW201007452A (en) * 2008-06-04 2010-02-16 Initio Corp SSD with a controller accelerator
CN101882141A (zh) * 2009-05-08 2010-11-10 北京众志和达信息技术有限公司 一种实现重复数据数据删除的方法和系统
CN102591947A (zh) * 2010-12-28 2012-07-18 微软公司 用于数据去重复的快速且低ram占用的索引
CN103547991A (zh) * 2010-12-29 2014-01-29 亚马逊科技公司 数据系统中的接收器侧数据重复删除
CN103473266A (zh) * 2013-08-09 2013-12-25 记忆科技(深圳)有限公司 固态硬盘及其删除重复数据的方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
"一种基于历史信息的一致性Hash集群重复数据删除路由策略";邢玉轩 等;《计算机研究与发展》;20141215(第2014年第S2期);第182-188页 *

Also Published As

Publication number Publication date
WO2017049142A1 (en) 2017-03-23
US20170242616A1 (en) 2017-08-24
US9864542B2 (en) 2018-01-09
KR20180052739A (ko) 2018-05-18
US9665287B2 (en) 2017-05-30
US20170083245A1 (en) 2017-03-23
JP2018527681A (ja) 2018-09-20
CN108027713A (zh) 2018-05-11
EP3350683B1 (en) 2023-05-10
EP3350683A4 (en) 2019-04-24
EP3350683A1 (en) 2018-07-25

Similar Documents

Publication Publication Date Title
CN108027713B (zh) 用于固态驱动器控制器的重复数据删除
US11874815B2 (en) Key-value storage device and method of operating the same
US10642522B2 (en) Method and system for in-line deduplication in a storage drive based on a non-collision hash
US9740403B2 (en) Methods for managing storage in a data storage cluster with distributed zones based on parity values and devices thereof
US9846642B2 (en) Efficient key collision handling
US10545833B1 (en) Block-level deduplication
US9405684B1 (en) System and method for cache management
US8683156B2 (en) Format-preserving deduplication of data
US20200133545A1 (en) Efficient compression of data in storage systems through offloading computation to storage devices
CN115114055B (zh) 管理归因于存储装置故障的容量减小和恢复
US12008254B2 (en) Deduplication of storage device encoded data
CN115114054B (zh) 管理发生故障的多层级存储器单元的存储空间减小和再用
US11099756B2 (en) Managing data block compression in a storage system
US20150193311A1 (en) Managing production data
US20190310788A1 (en) Similarity-based data deduplication on solid-state storage devices with embedded nonvolatile memory
US11132137B2 (en) Methods and systems for providing read-optimized scalable offline de-duplication for blocks of data
EP4052374B1 (en) Storage efficiency increase in a storage system
US11681436B2 (en) Systems and methods for asynchronous input/output scanning and aggregation for solid state drive
US11392547B2 (en) Using prefix-delete operations for data containers
US10922003B1 (en) Realizing host-assisted device-level data deduplication on solid-state data storage devices with embedded non-volatile memory
US10089032B1 (en) Controlling write sizes to reduce flash wear
US20240028234A1 (en) Multi-fingerprint deduplication processing

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant