CN111149093A - 分布式存储系统的数据编码、解码及修复方法 - Google Patents

分布式存储系统的数据编码、解码及修复方法 Download PDF

Info

Publication number
CN111149093A
CN111149093A CN201880032017.1A CN201880032017A CN111149093A CN 111149093 A CN111149093 A CN 111149093A CN 201880032017 A CN201880032017 A CN 201880032017A CN 111149093 A CN111149093 A CN 111149093A
Authority
CN
China
Prior art keywords
block
coding
data
decoding
fault
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201880032017.1A
Other languages
English (en)
Other versions
CN111149093B (zh
Inventor
郝斌
朱健
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Huaer Data Technology Co Ltd
Original Assignee
Shenzhen Huaer Data Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Huaer Data Technology Co Ltd filed Critical Shenzhen Huaer Data Technology Co Ltd
Publication of CN111149093A publication Critical patent/CN111149093A/zh
Application granted granted Critical
Publication of CN111149093B publication Critical patent/CN111149093B/zh
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1448—Management of the data involved in backup or backup restore
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1458—Management of the backup or restore process
    • G06F11/1464—Management of the backup or restore process for networked environments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1458—Management of the backup or restore process
    • G06F11/1469—Backup restoration techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2023—Failover techniques
    • G06F11/2028—Failover techniques eliminating a faulty processor or activating a spare
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/80—Database-specific techniques

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)

Abstract

一种分布式存储系统的数据编码、解码及修复方法，可用于实现分布式存储系统的数据保护策略。其采用局部可修复编码方法，根据编码参数，对文件段切分后的数据块调用Reed‑Solomon编码算法生成全局编码块，并分别对数据块和全局编码块进行局部编码生成局部编码块；且能根据当前节点状态，计算解码块索引和修复块索引，读取辅助节点块数据，完成文件段解码和故障块修复。本发明编码方法通过增加局部编码块，降低修复故障节点时需要传输的数据量，加快了节点修复速度。本方案可解决分布式存储系统中采用Reed‑Solomon编码时，故障修复时修复带宽过大问题，可减少故障节点修复时间，继而提高存储系统数据访问速度和吞吐量。

Description

PCT国内申请，说明书已公开。

Claims (12)

  1. PCT国内申请，权利要求书已公开。
CN201880032017.1A 2018-09-03 2018-09-03 分布式存储系统的数据编码、解码及修复方法 Active CN111149093B (zh)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2018/103805 WO2020047707A1 (zh) 2018-09-03 2018-09-03 分布式存储系统的数据编码、解码及修复方法

Publications (2)

Publication Number Publication Date
CN111149093A true CN111149093A (zh) 2020-05-12
CN111149093B CN111149093B (zh) 2023-07-11

Family

ID=69721733

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201880032017.1A Active CN111149093B (zh) 2018-09-03 2018-09-03 分布式存储系统的数据编码、解码及修复方法

Country Status (3)

Country Link
US (1) US11531593B2 (zh)
CN (1) CN111149093B (zh)
WO (1) WO2020047707A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112463435A (zh) * 2020-12-07 2021-03-09 广东工业大学 一种基于数据块访问频度的局部修复方法
CN114915377A (zh) * 2022-05-12 2022-08-16 中国人民解放军国防科技大学 一种基于喷泉码的联盟链存储系统
CN115793984A (zh) * 2023-01-03 2023-03-14 苏州浪潮智能科技有限公司 一种数据存储方法、装置、计算机设备及存储介质
WO2023056904A1 (zh) * 2021-10-09 2023-04-13 阿里云计算有限公司 校验块的生成方法及装置
CN115964445A (zh) * 2023-02-23 2023-04-14 合肥申威睿思信息科技有限公司 一种分布式数据库的多副本实现方法和装置

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210326320A1 (en) * 2018-10-15 2021-10-21 Ocient Inc. Data segment storing in a database system
GB202102240D0 (en) 2021-02-17 2021-03-31 Univ Oxford Innovation Ltd Catalytic upcycling process
CN113505021B (zh) * 2021-05-26 2023-07-18 南京大学 基于多主节点主从分布式架构的容错方法及系统
CN116266777A (zh) * 2021-12-16 2023-06-20 华为技术有限公司 数据传输的方法、装置和通信系统
CN114866561B (zh) * 2022-05-03 2023-09-01 中国人民解放军国防科技大学 一种组合本地纠删码联盟链存储方法及系统
CN115454712B (zh) * 2022-11-11 2023-02-28 苏州浪潮智能科技有限公司 一种校验码恢复方法、系统、电子设备及存储介质
CN116467037B (zh) * 2023-06-09 2023-09-22 成都融见软件科技有限公司 一种图形用户界面工作状态的恢复方法
CN117370067B (zh) * 2023-12-07 2024-04-12 融科联创(天津)信息技术有限公司 一种大规模对象存储系统的数据布局和编码方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100112957A1 (en) * 2007-04-19 2010-05-06 Lg Electronics Inc. Method of communication in mobile communication system
US20140317222A1 (en) * 2012-01-13 2014-10-23 Hui Li Data Storage Method, Device and Distributed Network Storage System
CN105956128A (zh) * 2016-05-09 2016-09-21 南京大学 一种基于简单再生码的自适应编码存储容错方法
CN106776112A (zh) * 2017-02-09 2017-05-31 长安大学 一种基于Pyramid码的局部性修复编码方法
CN108347306A (zh) * 2018-03-16 2018-07-31 长安大学 分布式存储系统中类局部重构码编码及节点故障修复方法

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101967884B1 (ko) * 2012-07-12 2019-04-12 삼성전자주식회사 방송 및 통신 시스템에서 패킷 송/수신 장치 및 방법
CN103688515B (zh) * 2013-03-26 2016-10-05 北京大学深圳研究生院 一种最小带宽再生码的编码和存储节点修复方法
US10503611B1 (en) * 2016-12-23 2019-12-10 EMC IP Holding Company LLC Data protection management for distributed storage
CN106844098B (zh) * 2016-12-29 2020-04-03 中国科学院计算技术研究所 一种基于十字交叉纠删编码的快速数据恢复方法及系统
US10733061B2 (en) * 2017-06-27 2020-08-04 Western Digital Technologies, Inc. Hybrid data storage system with private storage cloud and public storage cloud

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100112957A1 (en) * 2007-04-19 2010-05-06 Lg Electronics Inc. Method of communication in mobile communication system
US20140317222A1 (en) * 2012-01-13 2014-10-23 Hui Li Data Storage Method, Device and Distributed Network Storage System
CN105956128A (zh) * 2016-05-09 2016-09-21 南京大学 一种基于简单再生码的自适应编码存储容错方法
CN106776112A (zh) * 2017-02-09 2017-05-31 长安大学 一种基于Pyramid码的局部性修复编码方法
CN108347306A (zh) * 2018-03-16 2018-07-31 长安大学 分布式存储系统中类局部重构码编码及节点故障修复方法

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
李挥;张宇蒙;陈俊;: "大数据环境下的可靠存储技术思考" *
王静;张崇;梁伟;刘向阳;: "分布式存储系统中基于Pyramid码的局部性修复编码" *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112463435A (zh) * 2020-12-07 2021-03-09 广东工业大学 一种基于数据块访问频度的局部修复方法
WO2023056904A1 (zh) * 2021-10-09 2023-04-13 阿里云计算有限公司 校验块的生成方法及装置
CN114915377A (zh) * 2022-05-12 2022-08-16 中国人民解放军国防科技大学 一种基于喷泉码的联盟链存储系统
CN114915377B (zh) * 2022-05-12 2024-04-02 中国人民解放军国防科技大学 一种基于喷泉码的联盟链存储系统
CN115793984A (zh) * 2023-01-03 2023-03-14 苏州浪潮智能科技有限公司 一种数据存储方法、装置、计算机设备及存储介质
CN115964445A (zh) * 2023-02-23 2023-04-14 合肥申威睿思信息科技有限公司 一种分布式数据库的多副本实现方法和装置
CN115964445B (zh) * 2023-02-23 2024-03-05 合肥申威睿思信息科技有限公司 一种分布式数据库的多副本实现方法和装置

Also Published As

Publication number Publication date
WO2020047707A1 (zh) 2020-03-12
US11531593B2 (en) 2022-12-20
US20210271557A1 (en) 2021-09-02
CN111149093B (zh) 2023-07-11

Similar Documents

Publication Publication Date Title
CN111149093B (zh) 分布式存储系统的数据编码、解码及修复方法
CN109643258B (zh) 使用高速率最小存储再生擦除代码的多节点修复
US10951236B2 (en) Hierarchical data integrity verification of erasure coded data in a distributed computing system
EP2394220B1 (en) Distributed storage of recoverable data
US9280416B1 (en) Selection of erasure code parameters for no data repair
US20170192848A1 (en) Distributed data storage with reduced storage overhead using reduced-dependency erasure codes
CN111124738B (zh) 用于独立冗余磁盘阵列的数据管理方法、设备和计算机程序产品
CN110389858B (zh) 存储设备的故障恢复方法和设备
CN110750382A (zh) 用于提高数据修复性能的最小存储再生码编码方法及系统
US10644726B2 (en) Method and apparatus for reconstructing a data block
US11886705B2 (en) System and method for using free space to improve erasure code locality
CN113687975B (zh) 数据处理方法、装置、设备及存储介质
US9489254B1 (en) Verification of erasure encoded fragments
CN108762978B (zh) 一种局部部分重复循环码的分组构造方法
CN107003933A (zh) 部分复制码的构建方法、装置及其数据修复的方法
CN114816837A (zh) 一种纠删码融合方法、系统、电子设备及存储介质
US9552254B1 (en) Verification of erasure encoded fragments
CN111224747A (zh) 可降低修复带宽和磁盘读取开销的编码方法及其修复方法
US9489252B1 (en) File recovery using diverse erasure encoded fragments
CN111506450B (zh) 用于数据处理的方法、设备和计算机程序产品
CN108647108B (zh) 一种基于循环vfrc的最小带宽再生码的构造方法
CN107463462B (zh) 数据修复方法及数据修复装置
CN109144767B (zh) 数据存储系统及其操作方法
CN114564337A (zh) 一种基于x码的分布式存储系统容错方法及系统
US9450617B2 (en) Distribution and replication of erasure codes

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant