CN114072759A - 存储系统中数据处理方法、装置及计算机存储可读存储介质 - Google Patents

存储系统中数据处理方法、装置及计算机存储可读存储介质 Download PDF

Info

Publication number
CN114072759A
CN114072759A CN201980028810.9A CN201980028810A CN114072759A CN 114072759 A CN114072759 A CN 114072759A CN 201980028810 A CN201980028810 A CN 201980028810A CN 114072759 A CN114072759 A CN 114072759A
Authority
CN
China
Prior art keywords
fingerprint
mapping
data block
storage
storage address
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201980028810.9A
Other languages
English (en)
Inventor
任仁
王晨
代海军
朱芳芳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to CN202210400497.0A priority Critical patent/CN114968090A/zh
Priority to CN202210400441.5A priority patent/CN114816251A/zh
Publication of CN114072759A publication Critical patent/CN114072759A/zh
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G06F3/0641—De-duplication techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0629—Configuration or reconfiguration of storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0683—Plurality of storage devices
    • G06F3/0685—Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

提供了存储系统中的数据处理方法，包括：当存储系统处于第一负载时，进行在线重删操作；当存储系统处于第二负载时，直接存储收到的据块而不进行在线重复数据删除操作，其中，所述第一负载小于所述第二负载。

Description

PCT国内申请，说明书已公开。

Claims (23)

  1. PCT国内申请，权利要求书已公开。
CN201980028810.9A 2019-07-26 2019-07-26 存储系统中数据处理方法、装置及计算机存储可读存储介质 Pending CN114072759A (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202210400497.0A CN114968090A (zh) 2019-07-26 2019-07-26 数据处理方法、装置及计算机存储可读存储介质
CN202210400441.5A CN114816251A (zh) 2019-07-26 2019-07-26 数据处理方法、装置及计算机存储可读存储介质

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2019/097804 WO2021016728A1 (zh) 2019-07-26 2019-07-26 存储系统中数据处理方法、装置及计算机存储可读存储介质

Related Child Applications (2)

Application Number Title Priority Date Filing Date
CN202210400441.5A Division CN114816251A (zh) 2019-07-26 2019-07-26 数据处理方法、装置及计算机存储可读存储介质
CN202210400497.0A Division CN114968090A (zh) 2019-07-26 2019-07-26 数据处理方法、装置及计算机存储可读存储介质

Publications (1)

Publication Number Publication Date
CN114072759A true CN114072759A (zh) 2022-02-18

Family

ID=74228197

Family Applications (3)

Application Number Title Priority Date Filing Date
CN202210400497.0A Pending CN114968090A (zh) 2019-07-26 2019-07-26 数据处理方法、装置及计算机存储可读存储介质
CN201980028810.9A Pending CN114072759A (zh) 2019-07-26 2019-07-26 存储系统中数据处理方法、装置及计算机存储可读存储介质
CN202210400441.5A Pending CN114816251A (zh) 2019-07-26 2019-07-26 数据处理方法、装置及计算机存储可读存储介质

Family Applications Before (1)

Application Number Title Priority Date Filing Date
CN202210400497.0A Pending CN114968090A (zh) 2019-07-26 2019-07-26 数据处理方法、装置及计算机存储可读存储介质

Family Applications After (1)

Application Number Title Priority Date Filing Date
CN202210400441.5A Pending CN114816251A (zh) 2019-07-26 2019-07-26 数据处理方法、装置及计算机存储可读存储介质

Country Status (4)

Country Link
US (2) US12019890B2 (zh)
EP (2) EP4130970A1 (zh)
CN (3) CN114968090A (zh)
WO (1) WO2021016728A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113986891B (zh) * 2021-09-09 2024-03-12 新华三大数据技术有限公司 一种重复数据删除方法及装置
CN117631957A (zh) * 2022-08-15 2024-03-01 华为技术有限公司 一种数据的缩减方法、装置、设备、存储介质及处理器

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103049508A (zh) * 2012-12-13 2013-04-17 华为技术有限公司 一种数据处理方法及装置
US20140114932A1 (en) * 2012-10-18 2014-04-24 Netapp, Inc. Selective deduplication
US20140214794A1 (en) * 2013-01-30 2014-07-31 International Business Machines Corporation Join operation partitioning
CN105607867A (zh) * 2014-11-14 2016-05-25 爱思开海力士有限公司 使用主装置和从装置的重复删除
CN106610790A (zh) * 2015-10-26 2017-05-03 华为技术有限公司 一种重复数据删除方法及装置
US20180067680A1 (en) * 2016-09-07 2018-03-08 Fujitsu Limited Storage control apparatus, system, and storage medium
US20180239553A1 (en) * 2016-09-28 2018-08-23 Huawei Technologies Co., Ltd. Method for deduplication in storage system, storage system, and controller
CN108762679A (zh) * 2018-05-30 2018-11-06 郑州云海信息技术有限公司 一种在线ddp与离线ddp相结合的方法及其相关装置
CN109542360A (zh) * 2018-12-03 2019-03-29 郑州云海信息技术有限公司 数据重删方法、装置、设备、系统及计算机可读存储介质

Family Cites Families (56)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8635194B2 (en) * 2006-10-19 2014-01-21 Oracle International Corporation System and method for data compression
US10642794B2 (en) * 2008-09-11 2020-05-05 Vmware, Inc. Computer storage deduplication
US20100088296A1 (en) * 2008-10-03 2010-04-08 Netapp, Inc. System and method for organizing data to facilitate data deduplication
US8751462B2 (en) * 2008-11-14 2014-06-10 Emc Corporation Delta compression after identity deduplication
US8161255B2 (en) * 2009-01-06 2012-04-17 International Business Machines Corporation Optimized simultaneous storing of data into deduplicated and non-deduplicated storage pools
US8195636B2 (en) * 2009-04-29 2012-06-05 Netapp, Inc. Predicting space reclamation in deduplicated datasets
US20100333116A1 (en) * 2009-06-30 2010-12-30 Anand Prahlad Cloud gateway system for managing data storage to cloud storage sites
US8572163B1 (en) * 2009-08-31 2013-10-29 Symantec Corporation Systems and methods for deduplicating data based on performance of a deduplication system
US8539148B1 (en) * 2010-12-22 2013-09-17 Emc Corporation Deduplication efficiency
US8589640B2 (en) * 2011-10-14 2013-11-19 Pure Storage, Inc. Method for maintaining multiple fingerprint tables in a deduplicating storage system
US8930307B2 (en) * 2011-09-30 2015-01-06 Pure Storage, Inc. Method for removing duplicate data from a storage array
US9715434B1 (en) * 2011-09-30 2017-07-25 EMC IP Holding Company LLC System and method for estimating storage space needed to store data migrated from a source storage to a target storage
US8732403B1 (en) * 2012-03-14 2014-05-20 Netapp, Inc. Deduplication of data blocks on storage devices
KR20140114515A (ko) * 2013-03-15 2014-09-29 삼성전자주식회사 불휘발성 메모리 장치 및 그것의 중복 데이터 제거 방법
CN104246722B (zh) * 2013-03-29 2017-02-22 株式会社东芝 用于基于哈希表排除数据重复的存储系统，存储控制器及方法
US9384145B2 (en) * 2013-08-26 2016-07-05 Oracle International Corporation Systems and methods for implementing dynamically configurable perfect hash tables
US9317363B2 (en) * 2013-11-06 2016-04-19 International Business Machines Corporation Management of a secure delete operation in a parity-based system
WO2015067382A1 (en) * 2013-11-08 2015-05-14 Fujitsu Technology Solutions Intellectual Property Gmbh Storage appliance and method thereof for inline deduplication with segmentation
US11455282B1 (en) * 2013-12-10 2022-09-27 EMC IP Holding Company LLC System for queuing backup operations in a deduplicating storage system
US9384205B1 (en) * 2013-12-18 2016-07-05 Veritas Technologies Llc Auto adaptive deduplication to cloud based storage
US10380072B2 (en) * 2014-03-17 2019-08-13 Commvault Systems, Inc. Managing deletions from a deduplication database
CN105612488B (zh) * 2014-09-15 2017-08-18 华为技术有限公司 数据写请求处理方法和存储阵列
BR112016003763B1 (pt) * 2014-09-15 2019-04-02 Huawei Technologies Co., Ltd. Método de desduplicação de dados e arranjo de armazenamento.
US10747440B2 (en) * 2014-09-24 2020-08-18 Hitachi, Ltd. Storage system and storage system management method
US9792069B2 (en) * 2014-09-29 2017-10-17 Western Digital Technologies, Inc. Offline deduplication for solid-state storage devices
US10228858B1 (en) * 2015-02-11 2019-03-12 Violin Systems Llc System and method for granular deduplication
US9733836B1 (en) * 2015-02-11 2017-08-15 Violin Memory Inc. System and method for granular deduplication
US10089319B2 (en) * 2015-02-20 2018-10-02 International Business Machines Corporation Policy-based, multi-scheme data reduction for computer memory
WO2016147281A1 (ja) * 2015-03-16 2016-09-22 株式会社日立製作所 分散型ストレージシステム及び分散型ストレージシステムの制御方法
US9940337B2 (en) * 2015-05-31 2018-04-10 Vmware, Inc. Predictive probabilistic deduplication of storage
US20170038978A1 (en) * 2015-08-05 2017-02-09 HGST Netherlands B.V. Delta Compression Engine for Similarity Based Data Deduplication
US10031937B2 (en) * 2015-11-25 2018-07-24 International Business Machines Corporation Similarity based data deduplication of initial snapshots of data sets
SG10201610516RA (en) * 2015-12-17 2017-07-28 Agency Science Tech & Res Encrypted data deduplication in cloud storage
US9575681B1 (en) * 2016-04-29 2017-02-21 International Business Machines Corporation Data deduplication with reduced hash computations
US10572475B2 (en) * 2016-09-23 2020-02-25 Oracle International Corporation Leveraging columnar encoding for query operations
US10108543B1 (en) * 2016-09-26 2018-10-23 EMC IP Holding Company LLC Efficient physical garbage collection using a perfect hash vector
US10565205B2 (en) * 2016-11-14 2020-02-18 Sap Se Incrementally building hash collision tables
US10565204B2 (en) * 2016-11-14 2020-02-18 Sap Se Hash collision tables for relational join operations
JP6781377B2 (ja) * 2016-11-21 2020-11-04 富士通株式会社 情報処理装置、情報処理方法およびプログラム
US10001942B1 (en) * 2016-12-21 2018-06-19 Netapp Inc. Asynchronous semi-inline deduplication
US10771369B2 (en) * 2017-03-20 2020-09-08 International Business Machines Corporation Analyzing performance and capacity of a complex storage environment for predicting expected incident of resource exhaustion on a data path of interest by analyzing maximum values of resource usage over time
US10282125B2 (en) * 2017-04-17 2019-05-07 International Business Machines Corporation Distributed content deduplication using hash-trees with adaptive resource utilization in distributed file systems
US10558646B2 (en) * 2017-04-30 2020-02-11 International Business Machines Corporation Cognitive deduplication-aware data placement in large scale storage systems
CN107329692B (zh) * 2017-06-07 2020-02-28 杭州宏杉科技股份有限公司 一种数据重删的方法及存储设备
US10715177B2 (en) * 2017-06-20 2020-07-14 Samsung Electronics Co., Ltd. Lossy compression drive
US10795812B1 (en) * 2017-06-30 2020-10-06 EMC IP Holding Company LLC Virtual copy forward method and system for garbage collection in cloud computing networks
US10346076B1 (en) * 2017-07-03 2019-07-09 EMC IP Holding Company LLC Method and system for data deduplication based on load information associated with different phases in a data deduplication pipeline
US10754696B1 (en) * 2017-07-20 2020-08-25 EMC IP Holding Company LLC Scale out capacity load-balancing for backup appliances
CN107391761B (zh) * 2017-08-28 2020-03-06 苏州浪潮智能科技有限公司 一种基于重复数据删除技术的数据管理方法及装置
US10754557B2 (en) * 2017-09-26 2020-08-25 Seagate Technology Llc Data storage system with asynchronous data replication
JP7075077B2 (ja) * 2018-03-13 2022-05-25 Ｎｅｃソリューションイノベータ株式会社 バックアップサーバ、バックアップ方法、プログラム、ストレージシステム
US10922188B2 (en) * 2019-01-28 2021-02-16 EMC IP Holding Company LLC Method and system to tag and route the striped backups to a single deduplication instance on a deduplication appliance
US11507305B2 (en) * 2019-03-29 2022-11-22 EMC IP Holding Company LLC Concurrently performing normal system operations and garbage collection
US10664165B1 (en) * 2019-05-10 2020-05-26 EMC IP Holding Company LLC Managing inline data compression and deduplication in storage systems
CN112544038B (zh) * 2019-07-22 2024-07-05 华为技术有限公司 存储系统数据压缩的方法、装置、设备及可读存储介质
US11687424B2 (en) * 2020-05-28 2023-06-27 Commvault Systems, Inc. Automated media agent state management

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140114932A1 (en) * 2012-10-18 2014-04-24 Netapp, Inc. Selective deduplication
CN103049508A (zh) * 2012-12-13 2013-04-17 华为技术有限公司 一种数据处理方法及装置
US20140214794A1 (en) * 2013-01-30 2014-07-31 International Business Machines Corporation Join operation partitioning
CN105607867A (zh) * 2014-11-14 2016-05-25 爱思开海力士有限公司 使用主装置和从装置的重复删除
CN106610790A (zh) * 2015-10-26 2017-05-03 华为技术有限公司 一种重复数据删除方法及装置
US20180067680A1 (en) * 2016-09-07 2018-03-08 Fujitsu Limited Storage control apparatus, system, and storage medium
US20180239553A1 (en) * 2016-09-28 2018-08-23 Huawei Technologies Co., Ltd. Method for deduplication in storage system, storage system, and controller
CN108762679A (zh) * 2018-05-30 2018-11-06 郑州云海信息技术有限公司 一种在线ddp与离线ddp相结合的方法及其相关装置
CN109542360A (zh) * 2018-12-03 2019-03-29 郑州云海信息技术有限公司 数据重删方法、装置、设备、系统及计算机可读存储介质

Also Published As

Publication number Publication date
US20220147256A1 (en) 2022-05-12
WO2021016728A1 (zh) 2021-02-04
CN114968090A (zh) 2022-08-30
EP3971700A4 (en) 2022-05-25
CN114816251A (zh) 2022-07-29
US12019890B2 (en) 2024-06-25
EP4130970A1 (en) 2023-02-08
EP3971700A1 (en) 2022-03-23
EP3971700B1 (en) 2024-09-11
US20220300180A1 (en) 2022-09-22

Similar Documents

Publication Publication Date Title
US10719253B2 (en) Efficient compression of data in storage systems through offloading computation to storage devices
CN108427538B (zh) 全闪存阵列的存储数据压缩方法、装置、及可读存储介质
US10613976B2 (en) Method and storage device for reducing data duplication
CN108427539B (zh) 缓存设备数据的离线去重压缩方法、装置及可读存储介质
US10031675B1 (en) Method and system for tiering data
US11886704B2 (en) System and method for granular deduplication
US9569357B1 (en) Managing compressed data in a storage system
US11531641B2 (en) Storage system deduplication with service level agreements
WO2021073635A1 (zh) 一种数据存储方法及装置
CN109074226B (zh) 一种存储系统中重复数据删除方法、存储系统及控制器
CN110908589B (zh) 数据文件的处理方法、装置、系统和存储介质
US20220300180A1 (en) Data Deduplication Method and Apparatus, and Computer Program Product
US10606499B2 (en) Computer system, storage apparatus, and method of managing data
US11593312B2 (en) File layer to block layer communication for selective data reduction
US11513739B2 (en) File layer to block layer communication for block organization in storage
US10776052B2 (en) Information processing apparatus, data compressing method, and computer-readable recording medium
CN106383670B (zh) 一种数据处理方法及存储设备
WO2023050856A1 (zh) 数据处理方法及存储系统
CN103885859A (zh) 一种基于全局统计的去碎片方法及系统
CN114185850A (zh) 一种基于滑动窗口分块优化算法的云存储去重方法及设备
CN115145467A (zh) 数据压缩方法、控制器、设备、介质及程序产品
JP6733214B2 (ja) 制御装置、ストレージシステム、制御方法及びプログラム
US20230367477A1 (en) Storage system, data management program, and data management method
CN112988034B (zh) 一种分布式系统数据写入方法及装置
CN111611179B (zh) 元数据命中率提升方法、装置、存储介质及电子设备

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination