JP2018527681A - ソリッドステートドライブコントローラを使用するデータ重複排除 - Google Patents

ソリッドステートドライブコントローラを使用するデータ重複排除 Download PDF

Info

Publication number
JP2018527681A
JP2018527681A JP2018514841A JP2018514841A JP2018527681A JP 2018527681 A JP2018527681 A JP 2018527681A JP 2018514841 A JP2018514841 A JP 2018514841A JP 2018514841 A JP2018514841 A JP 2018514841A JP 2018527681 A JP2018527681 A JP 2018527681A
Authority
JP
Japan
Prior art keywords
signature
information
data
ssd
controller
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
JP2018514841A
Other languages
English (en)
Japanese (ja)
Other versions
JP2018527681A5 (enExample
Inventor
リー シュー
リー シュー
リー ヨン
リー ヨン
ニウ ゴンビアオ
ニウ ゴンビアオ
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Publication of JP2018527681A publication Critical patent/JP2018527681A/ja
Publication of JP2018527681A5 publication Critical patent/JP2018527681A5/ja
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G06F3/0641—De-duplication techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/38—Information transfer, e.g. on bus
    • G06F13/42—Bus transfer protocol, e.g. handshake; Synchronisation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G06F3/0611—Improving I/O performance in relation to response time
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2213/00—Indexing scheme relating to interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F2213/0032—Serial ATA [SATA]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
JP2018514841A 2015-09-18 2016-09-16 ソリッドステートドライブコントローラを使用するデータ重複排除 Pending JP2018527681A (ja)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US14/858,257 2015-09-18
US14/858,257 US9665287B2 (en) 2015-09-18 2015-09-18 Data deduplication using a solid state drive controller
PCT/US2016/052222 WO2017049142A1 (en) 2015-09-18 2016-09-16 Data deduplication using a solid state drive controller

Publications (2)

Publication Number Publication Date
JP2018527681A true JP2018527681A (ja) 2018-09-20
JP2018527681A5 JP2018527681A5 (enExample) 2019-09-26

Family

ID=58282696

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2018514841A Pending JP2018527681A (ja) 2015-09-18 2016-09-16 ソリッドステートドライブコントローラを使用するデータ重複排除

Country Status (6)

Country Link
US (2) US9665287B2 (enExample)
EP (1) EP3350683B1 (enExample)
JP (1) JP2018527681A (enExample)
KR (1) KR20180052739A (enExample)
CN (1) CN108027713B (enExample)
WO (1) WO2017049142A1 (enExample)

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9665287B2 (en) 2015-09-18 2017-05-30 Alibaba Group Holding Limited Data deduplication using a solid state drive controller
CN107402725B (zh) * 2017-03-20 2020-08-25 威盛电子股份有限公司 非易失性存储装置及其数据去重复方法
US10318202B2 (en) * 2017-03-20 2019-06-11 Via Technologies, Inc. Non-volatile memory apparatus and data deduplication method thereof
CN109669623B (zh) * 2017-10-13 2021-09-03 杭州海康威视系统技术有限公司 一种文件管理方法、文件管理装置、电子设备及存储介质
CN108920964B (zh) * 2018-06-21 2020-09-29 深圳忆联信息系统有限公司 可重构硬件加解密方法、系统、计算机设备及存储介质
CN109062514B (zh) * 2018-08-16 2021-08-31 郑州云海信息技术有限公司 一种基于命名空间的带宽控制方法、装置和存储介质
CN110968537B (zh) * 2018-09-28 2021-02-02 方一信息科技（上海）有限公司 一种基于pcie ssd的fpga搜索匹配方法
US11029874B2 (en) 2019-07-30 2021-06-08 Western Digital Technologies, Inc. Rolling XOR protection in efficient pipeline
KR102810527B1 (ko) 2019-09-23 2025-05-22 삼성전자주식회사 스토리지 장치 및 그것의 동작 방법
CN113365282B (zh) * 2021-06-22 2023-04-07 成都信息工程大学 一种wsn障碍性区域覆盖部署方法
US12007968B2 (en) 2022-05-26 2024-06-11 International Business Machines Corporation Full allocation volume to deduplication volume migration in a storage system
WO2025240883A1 (en) * 2024-05-17 2025-11-20 Micron Technology, Inc. Data de-duplication using content-addressable memory

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090307416A1 (en) * 2008-06-04 2009-12-10 Intitio Corporation Ssd with a controller accelerator
US7761425B1 (en) * 2007-03-29 2010-07-20 Symantec Corporation Low-overhead means of performing data backup
JP2010170475A (ja) * 2009-01-26 2010-08-05 Nec Corp ストレージシステム、ストレージシステムにおけるデータ書込方法及びデータ書込プログラム
JP2014511129A (ja) * 2010-12-29 2014-05-08 アマゾン・テクノロジーズ・インコーポレーテッド データシステムにおける受信器側データの重複排除
WO2014087508A1 (ja) * 2012-12-05 2014-06-12 株式会社日立製作所 ストレージシステム及びストレージシステムの制御方法
WO2014155668A1 (ja) * 2013-03-29 2014-10-02 株式会社 東芝 データの重複をハッシュテーブルに基づいて排除するためのストレージシステム、ストレージコントローラ及び方法
US9092151B1 (en) * 2010-09-17 2015-07-28 Permabit Technology Corporation Managing deduplication of stored data

Family Cites Families (66)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7636767B2 (en) 2005-11-29 2009-12-22 Cisco Technology, Inc. Method and apparatus for reducing network traffic over low bandwidth links
US8412682B2 (en) 2006-06-29 2013-04-02 Netapp, Inc. System and method for retrieving and using block fingerprints for data deduplication
US7840537B2 (en) 2006-12-22 2010-11-23 Commvault Systems, Inc. System and method for storing redundant information
US20090132616A1 (en) 2007-10-02 2009-05-21 Richard Winter Archival backup integration
US7962452B2 (en) 2007-12-28 2011-06-14 International Business Machines Corporation Data deduplication by separating data from meta data
US8825617B2 (en) 2008-03-14 2014-09-02 International Business Machines Corporation Limiting deduplication based on predetermined criteria
US7567188B1 (en) 2008-04-10 2009-07-28 International Business Machines Corporation Policy based tiered data deduplication strategy
US7814149B1 (en) 2008-09-29 2010-10-12 Symantec Operating Corporation Client side data deduplication
WO2010045262A1 (en) 2008-10-14 2010-04-22 Wanova Technologies, Ltd. Storage-network de-duplication
US8082228B2 (en) * 2008-10-31 2011-12-20 Netapp, Inc. Remote office duplication
US8812874B1 (en) 2009-03-31 2014-08-19 Symantec Corporation Content deduplication in enterprise rights management
US8407186B1 (en) 2009-03-31 2013-03-26 Symantec Corporation Systems and methods for data-selection-specific data deduplication
US8281066B1 (en) * 2009-04-30 2012-10-02 Netapp, Inc. System and method for real-time deduplication utilizing an electronic storage medium
CN101882141A (zh) * 2009-05-08 2010-11-10 北京众志和达信息技术有限公司 一种实现重复数据数据删除的方法和系统
US8442954B2 (en) 2009-07-21 2013-05-14 Stephen Philip SPACKMAN Creating and managing links to deduplication information
US8204867B2 (en) 2009-07-29 2012-06-19 International Business Machines Corporation Apparatus, system, and method for enhanced block-level deduplication
US20110093439A1 (en) 2009-10-16 2011-04-21 Fanglu Guo De-duplication Storage System with Multiple Indices for Efficient File Storage
US8321648B2 (en) 2009-10-26 2012-11-27 Netapp, Inc Use of similarity hash to route data for improved deduplication in a storage server cluster
US8478933B2 (en) 2009-11-24 2013-07-02 International Business Machines Corporation Systems and methods for performing deduplicated data processing on tape
US8407193B2 (en) 2010-01-27 2013-03-26 International Business Machines Corporation Data deduplication for streaming sequential data storage applications
JP5434705B2 (ja) 2010-03-12 2014-03-05 富士通株式会社 ストレージ装置、ストレージ装置制御プログラムおよびストレージ装置制御方法
US8370593B2 (en) 2010-04-14 2013-02-05 Hitachi, Ltd. Method and apparatus to manage groups for deduplication
US9047301B2 (en) 2010-04-19 2015-06-02 Greenbytes, Inc. Method for optimizing the memory usage and performance of data deduplication storage systems
US8639658B1 (en) 2010-04-21 2014-01-28 Symantec Corporation Cache management for file systems supporting shared blocks
US9053032B2 (en) 2010-05-05 2015-06-09 Microsoft Technology Licensing, Llc Fast and low-RAM-footprint indexing for data deduplication
US8935487B2 (en) 2010-05-05 2015-01-13 Microsoft Corporation Fast and low-RAM-footprint indexing for data deduplication
US8489855B2 (en) 2010-05-07 2013-07-16 Ocz Technology Group Inc. NAND flash-based solid state drive and method of operation
US20120173795A1 (en) * 2010-05-25 2012-07-05 Ocz Technology Group, Inc. Solid state drive with low write amplification
US8370315B1 (en) 2010-05-28 2013-02-05 Symantec Corporation System and method for high performance deduplication indexing
US10162553B2 (en) * 2010-11-24 2018-12-25 Western Digital Technologies, Inc. Methods and systems for object level de-duplication for solid state devices
US8898119B2 (en) 2010-12-15 2014-11-25 Netapp, Inc. Fingerprints datastore and stale fingerprint removal in de-duplication environments
US8332372B2 (en) 2010-12-16 2012-12-11 International Business Machines Corporation Method and system for processing data
US8495304B1 (en) 2010-12-23 2013-07-23 Emc Corporation Multi source wire deduplication
US9116909B2 (en) * 2010-12-29 2015-08-25 Amazon Technologies, Inc. Reduced bandwidth data uploading in data systems
US9223511B2 (en) 2011-04-08 2015-12-29 Micron Technology, Inc. Data deduplication
US8600949B2 (en) 2011-06-21 2013-12-03 Netapp, Inc. Deduplication in an extent-based architecture
US8589640B2 (en) * 2011-10-14 2013-11-19 Pure Storage, Inc. Method for maintaining multiple fingerprint tables in a deduplicating storage system
US8533231B2 (en) 2011-08-12 2013-09-10 Nexenta Systems, Inc. Cloud storage system with distributed metadata
US8484170B2 (en) 2011-09-19 2013-07-09 International Business Machines Corporation Scalable deduplication system with small blocks
US8620886B1 (en) * 2011-09-20 2013-12-31 Netapp Inc. Host side deduplication
CN103034659B (zh) 2011-09-29 2015-08-19 国际商业机器公司 一种重复数据删除的方法和系统
US8898120B1 (en) 2011-10-09 2014-11-25 Symantec Corporation Systems and methods for distributed data deduplication
US8572312B2 (en) 2011-12-07 2013-10-29 Jeffrey Tofano Data de-duplication and solid state memory device
KR20130064518A (ko) * 2011-12-08 2013-06-18 삼성전자주식회사 저장 장치 및 그것의 동작 방법
CN102646069B (zh) * 2012-02-23 2014-12-10 华中科技大学 一种延长固态盘使用寿命的方法
US20130282672A1 (en) * 2012-04-18 2013-10-24 Hitachi Computer Peripherals Co., Ltd. Storage apparatus and storage control method
US9177028B2 (en) 2012-04-30 2015-11-03 International Business Machines Corporation Deduplicating storage with enhanced frequent-block detection
US8930648B1 (en) 2012-05-23 2015-01-06 Netapp, Inc. Distributed deduplication using global chunk data structure and epochs
US8788468B2 (en) 2012-05-24 2014-07-22 International Business Machines Corporation Data depulication using short term history
US8930612B2 (en) 2012-05-31 2015-01-06 Seagate Technology Llc Background deduplication of data sets in a memory
CN102981969A (zh) * 2012-11-21 2013-03-20 记忆科技(深圳)有限公司 重复数据删除的方法及其固态硬盘
US8935222B2 (en) 2013-01-02 2015-01-13 International Business Machines Corporation Optimizing a partition in data deduplication
US9219784B2 (en) * 2013-03-07 2015-12-22 International Business Machines Corporation Synchronization of a server side deduplication cache with a client side deduplication cache
US9116941B2 (en) 2013-03-15 2015-08-25 International Business Machines Corporation Reducing digest storage consumption by tracking similarity elements in a data deduplication system
US9244937B2 (en) 2013-03-15 2016-01-26 International Business Machines Corporation Efficient calculation of similarity search values and digest block boundaries for data deduplication
US20140304464A1 (en) 2013-04-03 2014-10-09 Lsi Corporation Methods and systems for performing deduplication in a data storage system
CN103473266A (zh) * 2013-08-09 2013-12-25 记忆科技(深圳)有限公司 固态硬盘及其删除重复数据的方法
GB2518158A (en) * 2013-09-11 2015-03-18 Ibm Method and system for data access in a storage infrastructure
KR20150067583A (ko) * 2013-12-10 2015-06-18 삼성전자주식회사 불휘발성 메모리 장치 및 그것의 중복 데이터 제거 방법
US10380072B2 (en) * 2014-03-17 2019-08-13 Commvault Systems, Inc. Managing deletions from a deduplication database
JP6307624B2 (ja) * 2014-05-30 2018-04-04 株式会社日立製作所 データ重複排除ストレージシステムの方法及び装置
CN104407982B (zh) 2014-11-19 2018-09-21 湖南国科微电子股份有限公司 一种ssd盘片垃圾回收方法
US10416915B2 (en) * 2015-05-15 2019-09-17 ScaleFlux Assisting data deduplication through in-memory computation
US20170017571A1 (en) * 2015-07-17 2017-01-19 Samsung Electronics Co., Ltd. Method and apparatus fori n-line deduplication in storage devices
US9665287B2 (en) 2015-09-18 2017-05-30 Alibaba Group Holding Limited Data deduplication using a solid state drive controller
JP6067819B1 (ja) * 2015-10-21 2017-01-25 株式会社東芝 階層化ストレージシステム、ストレージコントローラ、並びに重複排除及びストレージ階層化のための方法

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7761425B1 (en) * 2007-03-29 2010-07-20 Symantec Corporation Low-overhead means of performing data backup
US20090307416A1 (en) * 2008-06-04 2009-12-10 Intitio Corporation Ssd with a controller accelerator
JP2010170475A (ja) * 2009-01-26 2010-08-05 Nec Corp ストレージシステム、ストレージシステムにおけるデータ書込方法及びデータ書込プログラム
US9092151B1 (en) * 2010-09-17 2015-07-28 Permabit Technology Corporation Managing deduplication of stored data
JP2014511129A (ja) * 2010-12-29 2014-05-08 アマゾン・テクノロジーズ・インコーポレーテッド データシステムにおける受信器側データの重複排除
WO2014087508A1 (ja) * 2012-12-05 2014-06-12 株式会社日立製作所 ストレージシステム及びストレージシステムの制御方法
WO2014155668A1 (ja) * 2013-03-29 2014-10-02 株式会社 東芝 データの重複をハッシュテーブルに基づいて排除するためのストレージシステム、ストレージコントローラ及び方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
平澤寿康: "平澤寿康の周辺機器レビュー Ｓａｍｓｕｎｇ 「Ｓａｍｓｕｎｇ ＳＳＤ ８４０ ＥＶＯ」 〜疑似ＳＬＣ", ＰＣ ＷＡＴＣＨ, JPN7020003948, 26 July 2013 (2013-07-26), ISSN: 0004404350 *

Also Published As

Publication number Publication date
WO2017049142A1 (en) 2017-03-23
CN108027713B (zh) 2021-10-12
US20170242616A1 (en) 2017-08-24
US9864542B2 (en) 2018-01-09
KR20180052739A (ko) 2018-05-18
US9665287B2 (en) 2017-05-30
US20170083245A1 (en) 2017-03-23
CN108027713A (zh) 2018-05-11
EP3350683B1 (en) 2023-05-10
EP3350683A4 (en) 2019-04-24
EP3350683A1 (en) 2018-07-25

Similar Documents

Publication Publication Date Title
US9864542B2 (en) Data deduplication using a solid state drive controller
US9405684B1 (en) System and method for cache management
US9846642B2 (en) Efficient key collision handling
US8751763B1 (en) Low-overhead deduplication within a block-based data storage
US8683156B2 (en) Format-preserving deduplication of data
US10019323B1 (en) Method and system for container data recovery in a storage system
US20200133545A1 (en) Efficient compression of data in storage systems through offloading computation to storage devices
US9619478B1 (en) Method and system for compressing logs
US20110066666A1 (en) File management method and storage system
US20180285014A1 (en) Data storage method and apparatus
CN107850983B (zh) 计算机系统、存储装置和数据的管理方法
CN112286714B (zh) 在基于nand的读取源存储中提高大数据分析吞吐量的方法和系统
US20200089784A1 (en) Method and system for reduced data movement compression using in-storage computing and a customized file system
CN105493080B (zh) 基于上下文感知的重复数据删除的方法和装置
US9575680B1 (en) Deduplication rehydration
US9619336B2 (en) Managing production data
US11068208B2 (en) Capacity reduction in a storage system
US11132137B2 (en) Methods and systems for providing read-optimized scalable offline de-duplication for blocks of data
US9563363B2 (en) Flexible storage block for a solid state drive (SSD)-based file system
US10990535B2 (en) Storage control apparatus and storage control method for deduplication
US11163748B1 (en) Fingerprint backward compatibility in deduplication backup systems
US11175842B2 (en) Method and system for performing data deduplication in a data pipeline
CN116738510A (zh) 有效地获得存储在地址空间中的信息的系统和方法
US20210278977A1 (en) Method and system for performing data deduplication and compression in a data cluster
US20190188102A1 (en) Method and system for data recovery in a cloud based computing environment utilizing object storage

Legal Events

Date Code Title Description
A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20190816

A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20190816

A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20200812

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20200901

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20201027

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20201208

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20210226

A02 Decision of refusal

Free format text: JAPANESE INTERMEDIATE CODE: A02

Effective date: 20210907