CN103761162B - 分布式文件系统的数据备份方法 - Google Patents
分布式文件系统的数据备份方法 Download PDFInfo
- Publication number
- CN103761162B CN103761162B CN201410013486.2A CN201410013486A CN103761162B CN 103761162 B CN103761162 B CN 103761162B CN 201410013486 A CN201410013486 A CN 201410013486A CN 103761162 B CN103761162 B CN 103761162B
- Authority
- CN
- China
- Prior art keywords
- source
- file
- block
- chunk
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 230000005540 biological transmission Effects 0.000 claims abstract description 26
- 238000012795 verification Methods 0.000 claims description 63
- 125000002015 acyclic group Chemical group 0.000 claims description 16
- 239000003550 marker Substances 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 2
- 238000012216 screening Methods 0.000 claims description 2
- 238000009826 distribution Methods 0.000 abstract description 7
- 230000007423 decrease Effects 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 11
- 238000010586 diagram Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 238000013507 mapping Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 230000020509 sex determination Effects 0.000 description 1
- 235000015170 shellfish Nutrition 0.000 description 1
- 238000013509 system migration Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
- G06F11/1451—Management of the data involved in backup or backup restore by selection of backup contents
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1456—Hardware arrangements for backup
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1464—Management of the backup or restore process for networked environments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/82—Solving problems relating to consistency
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Claims (9)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201410013486.2A CN103761162B (zh) | 2014-01-11 | 2014-01-11 | 分布式文件系统的数据备份方法 |
US14/593,358 US20150199243A1 (en) | 2014-01-11 | 2015-01-09 | Data backup method of distributed file system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201410013486.2A CN103761162B (zh) | 2014-01-11 | 2014-01-11 | 分布式文件系统的数据备份方法 |
Publications (2)
Publication Number | Publication Date |
---|---|
CN103761162A CN103761162A (zh) | 2014-04-30 |
CN103761162B true CN103761162B (zh) | 2016-12-07 |
Family
ID=50528404
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201410013486.2A Active CN103761162B (zh) | 2014-01-11 | 2014-01-11 | 分布式文件系统的数据备份方法 |
Country Status (2)
Country | Link |
---|---|
US (1) | US20150199243A1 (zh) |
CN (1) | CN103761162B (zh) |
Families Citing this family (63)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9495478B2 (en) * | 2014-03-31 | 2016-11-15 | Amazon Technologies, Inc. | Namespace management in distributed storage systems |
CN104079623B (zh) * | 2014-05-08 | 2018-03-20 | 深圳市中博科创信息技术有限公司 | 多级云存储同步控制方法及系统 |
CN104133674B (zh) * | 2014-07-11 | 2017-07-11 | 国家电网公司 | 一种异构系统及异构系统的模型同步方法 |
CN104202387B (zh) * | 2014-08-27 | 2017-11-24 | 华为技术有限公司 | 一种元数据恢复方法及相关装置 |
KR101960339B1 (ko) * | 2014-10-21 | 2019-03-20 | 삼성에스디에스 주식회사 | 파일 동기화 방법 |
CN105657337B (zh) * | 2014-11-20 | 2019-09-20 | 湘潭中星电子有限公司 | 视频数据处理方法和装置 |
CN105740248B (zh) * | 2014-12-09 | 2019-11-12 | 华为软件技术有限公司 | 一种数据同步方法、装置及系统 |
US10303666B2 (en) | 2015-03-09 | 2019-05-28 | International Business Machines Corporation | File transfer system using file backup times |
US10229124B2 (en) | 2015-05-01 | 2019-03-12 | Microsoft Technology Licensing, Llc | Re-directing tenants during a data move |
US10261943B2 (en) * | 2015-05-01 | 2019-04-16 | Microsoft Technology Licensing, Llc | Securely moving data across boundaries |
US10678762B2 (en) | 2015-05-01 | 2020-06-09 | Microsoft Technology Licensing, Llc | Isolating data to be moved across boundaries |
CN104866394B (zh) * | 2015-06-08 | 2018-03-09 | 肖选文 | 一种分布式文件备份方法和系统 |
TW201719402A (zh) * | 2015-11-27 | 2017-06-01 | Chunghwa Telecom Co Ltd | 資料倉儲異地備援方法與系統 |
US9396251B1 (en) | 2016-01-07 | 2016-07-19 | International Business Machines Corporation | Detecting and tracking virtual containers |
CN105956123A (zh) * | 2016-05-03 | 2016-09-21 | 无锡雅座在线科技发展有限公司 | 基于局部更新软件的数据处理方法及装置 |
US10216379B2 (en) | 2016-10-25 | 2019-02-26 | Microsoft Technology Licensing, Llc | User interaction processing in an electronic mail system |
CN108241556A (zh) * | 2016-12-26 | 2018-07-03 | 航天信息股份有限公司 | Hdfs中数据异地备份的方法及装置 |
CN106874403A (zh) * | 2017-01-18 | 2017-06-20 | 武汉天喻教育科技有限公司 | 对压缩文件进行差异同步的系统及方法 |
CN108804253B (zh) * | 2017-05-02 | 2021-08-06 | 中国科学院高能物理研究所 | 一种用于海量数据备份的并行作业备份方法 |
CN108874825B (zh) * | 2017-05-12 | 2021-11-02 | 北京京东尚科信息技术有限公司 | 一种异常数据的校验方法和装置 |
US10884977B1 (en) | 2017-06-22 | 2021-01-05 | Jpmorgan Chase Bank, N.A. | Systems and methods for distributed file processing |
CN109471901B (zh) * | 2017-08-18 | 2021-12-07 | 北京国双科技有限公司 | 一种数据同步方法及装置 |
CN107632781B (zh) * | 2017-08-28 | 2020-05-05 | 深圳市云舒网络技术有限公司 | 一种分布式存储多副本快速校验一致性的方法及存储结构 |
CN107491565B (zh) * | 2017-10-10 | 2020-01-14 | 语联网(武汉)信息技术有限公司 | 一种数据同步方法 |
US10331363B2 (en) * | 2017-11-22 | 2019-06-25 | Seagate Technology Llc | Monitoring modifications to data blocks |
CN108197155A (zh) * | 2017-12-08 | 2018-06-22 | 深圳前海微众银行股份有限公司 | 信息数据同步方法、装置及计算机可读存储介质 |
CN110636090B (zh) * | 2018-06-22 | 2022-09-20 | 北京东土科技股份有限公司 | 窄带宽条件下的数据同步方法和装置 |
CN110633168A (zh) * | 2018-06-22 | 2019-12-31 | 北京东土科技股份有限公司 | 一种分布式存储系统的数据备份方法和系统 |
US11099743B2 (en) | 2018-06-29 | 2021-08-24 | International Business Machines Corporation | Determining when to replace a storage device using a machine learning module |
US11119662B2 (en) * | 2018-06-29 | 2021-09-14 | International Business Machines Corporation | Determining when to perform a data integrity check of copies of a data set using a machine learning module |
US11119850B2 (en) | 2018-06-29 | 2021-09-14 | International Business Machines Corporation | Determining when to perform error checking of a storage unit by using a machine learning module |
CN109299056B (zh) * | 2018-09-19 | 2019-10-01 | 潍坊工程职业学院 | 一种基于分布式文件系统的数据同步方法和装置 |
CN109614383B (zh) * | 2018-11-21 | 2021-01-15 | 金色熊猫有限公司 | 数据复制方法、装置、电子设备及存储介质 |
CN111274311A (zh) * | 2018-12-05 | 2020-06-12 | 聚好看科技股份有限公司 | 一种跨机房数据库的数据同步方法和装置 |
CN111314403B (zh) * | 2018-12-12 | 2022-09-02 | 阿里巴巴集团控股有限公司 | 资源一致性的校验方法和装置 |
CN111522688B (zh) * | 2019-02-01 | 2023-09-15 | 阿里巴巴集团控股有限公司 | 分布式系统的数据备份方法及装置 |
CN110083615A (zh) * | 2019-04-12 | 2019-08-02 | 平安普惠企业管理有限公司 | 一种数据验证方法、装置、电子设备及存储介质 |
CN110163009B (zh) * | 2019-05-23 | 2021-06-15 | 北京交通大学 | Hdfs存储平台的安全校验及修复的方法和装置 |
CN110209653B (zh) * | 2019-06-04 | 2021-11-23 | 中国农业银行股份有限公司 | HBase数据迁移方法及迁移装置 |
CN110504002B (zh) * | 2019-08-01 | 2021-08-17 | 苏州浪潮智能科技有限公司 | 一种硬盘数据一致性测试方法与装置 |
US11010367B2 (en) | 2019-08-07 | 2021-05-18 | Micro Focus Llc | Parallel batch metadata transfer update process within sharded columnar database system |
CN110633164B (zh) * | 2019-08-09 | 2023-05-16 | 锐捷网络股份有限公司 | 一种面向消息的中间件故障恢复方法及装置 |
TWI719609B (zh) * | 2019-08-28 | 2021-02-21 | 威進國際資訊股份有限公司 | 異地備援系統 |
CN110597778B (zh) * | 2019-09-11 | 2022-04-22 | 北京宝兰德软件股份有限公司 | 一种分布式文件备份和监控的方法及装置 |
CN110851417B (zh) * | 2019-10-11 | 2022-11-29 | 苏宁云计算有限公司 | 一种分布式文件系统文件的拷贝方法及装置 |
CN111124755B (zh) * | 2019-12-06 | 2023-08-15 | 中国联合网络通信集团有限公司 | 集群节点的故障恢复方法、装置、电子设备及存储介质 |
CN111382011B (zh) | 2020-02-28 | 2022-11-29 | 苏州浪潮智能科技有限公司 | 一种文件数据存取方法、装置和计算机可读存储介质 |
CN113495877A (zh) * | 2020-04-03 | 2021-10-12 | 北京罗克维尔斯科技有限公司 | 数据同步方法及系统 |
CN111581031A (zh) * | 2020-05-13 | 2020-08-25 | 上海英方软件股份有限公司 | 一种基于rdc不定长分块策略的数据同步方法及装置 |
US20210357364A1 (en) * | 2020-05-13 | 2021-11-18 | Magnet Forensics Inc. | System and method for identifying files based on hash values |
CN111880970A (zh) * | 2020-08-04 | 2020-11-03 | 杭州东方通信软件技术有限公司 | 一种快捷远程文件备份方法 |
CN112015560B (zh) * | 2020-09-08 | 2023-12-26 | 财拓云计算(上海)有限公司 | 一种用于构建it基础设施的装置 |
CN112527521B (zh) * | 2020-12-03 | 2023-07-04 | 中国联合网络通信集团有限公司 | 消息处理方法及设备 |
CN112463457A (zh) * | 2020-12-10 | 2021-03-09 | 上海爱数信息技术股份有限公司 | 一种保障应用一致性的数据保护方法、装置、介质及系统 |
CN113760897A (zh) * | 2021-01-19 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | 数据重分片方法、装置、计算机系统和计算机可读存储介质 |
CN113157645B (zh) * | 2021-04-21 | 2023-12-19 | 平安科技(深圳)有限公司 | 集群数据迁移方法、装置、设备及存储介质 |
CN113064672A (zh) * | 2021-04-30 | 2021-07-02 | 中国工商银行股份有限公司 | 一种负载均衡设备配置信息的校验方法及装置 |
CN113641628B (zh) * | 2021-08-13 | 2023-06-16 | 中国联合网络通信集团有限公司 | 数据质量检测方法、装置、设备及存储介质 |
CN113821485A (zh) * | 2021-09-27 | 2021-12-21 | 深信服科技股份有限公司 | 一种数据变更方法、装置、设备及计算机可读存储介质 |
US12032522B2 (en) * | 2021-11-02 | 2024-07-09 | Paul Tsyganko | System, method, and computer program product for cataloging data integrity |
CN114253761A (zh) * | 2021-12-20 | 2022-03-29 | 中国科学院微小卫星创新研究院 | 一种基于校验的程序块三模存储方法 |
CN114328030B (zh) * | 2022-03-03 | 2022-05-20 | 成都云祺科技有限公司 | 一种文件数据备份方法、系统及存储介质 |
CN118368294B (zh) * | 2024-06-19 | 2024-09-10 | 鹏城实验室 | 数据传输方法、装置、设备和存储介质 |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102394923A (zh) * | 2011-10-27 | 2012-03-28 | 周诗琦 | 一种基于n×n陈列结构的云系统平台 |
CN102646127A (zh) * | 2012-02-29 | 2012-08-22 | 浪潮(北京)电子信息产业有限公司 | 分布式文件系统副本选择方法和装置 |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101539873B (zh) * | 2009-04-15 | 2011-02-09 | 成都市华为赛门铁克科技有限公司 | 数据恢复的方法、数据节点及分布式文件系统 |
US20100274772A1 (en) * | 2009-04-23 | 2010-10-28 | Allen Samuels | Compressed data objects referenced via address references and compression references |
US8504517B2 (en) * | 2010-03-29 | 2013-08-06 | Commvault Systems, Inc. | Systems and methods for selective data replication |
US8712960B2 (en) * | 2011-05-19 | 2014-04-29 | Vmware, Inc. | Method and system for parallelizing data copy in a distributed file system |
-
2014
- 2014-01-11 CN CN201410013486.2A patent/CN103761162B/zh active Active
-
2015
- 2015-01-09 US US14/593,358 patent/US20150199243A1/en not_active Abandoned
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102394923A (zh) * | 2011-10-27 | 2012-03-28 | 周诗琦 | 一种基于n×n陈列结构的云系统平台 |
CN102646127A (zh) * | 2012-02-29 | 2012-08-22 | 浪潮(北京)电子信息产业有限公司 | 分布式文件系统副本选择方法和装置 |
Also Published As
Publication number | Publication date |
---|---|
US20150199243A1 (en) | 2015-07-16 |
CN103761162A (zh) | 2014-04-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN103761162B (zh) | 分布式文件系统的数据备份方法 | |
US10628378B2 (en) | Replication of snapshots and clones | |
EP3532935B1 (en) | Snapshot metadata arrangement for cloud integration | |
CN104641365B (zh) | 在文件存储系统中使用检查点管理去复制的系统和方法 | |
US10956364B2 (en) | Efficient data synchronization for storage containers | |
US9904599B2 (en) | Method, device, and system for data reconstruction | |
US11714785B2 (en) | Deduplicating extents across systems | |
US7213116B2 (en) | Method and apparatus for mirroring objects between storage systems | |
CN110096891B (zh) | 对象库中的对象签名 | |
US10289692B2 (en) | Preserving file metadata during atomic save operations | |
US11782886B2 (en) | Incremental virtual machine metadata extraction | |
US20110153570A1 (en) | Data replication and recovery method in asymmetric clustered distributed file system | |
US20180213035A1 (en) | File replication using file content location identifiers | |
CN106547859B (zh) | 一种多租户数据存储系统下的数据文件的存储方法及装置 | |
US20170300550A1 (en) | Data Cloning System and Process | |
EP3532934A1 (en) | Reducing stable data eviction with synthetic baseline snapshot and eviction state refresh | |
US11886412B2 (en) | Large content file optimization | |
US20070033370A1 (en) | Continuous data protection | |
CN110515543B (zh) | 基于对象存储桶的快照方法、装置和系统 | |
US11989095B2 (en) | Techniques for preserving clone relationships between files | |
CA2825891A1 (en) | Storage system for storing data in a plurality of storage devices | |
CN104951475A (zh) | 分布式文件系统和实现方法 | |
KR20130026738A (ko) | 파일 데이터 분산 저장 장치 및 방법 | |
CN114328375A (zh) | 用于存储管理的方法、设备和计算机程序产品 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
C14 | Grant of patent or utility model | ||
GR01 | Patent grant | ||
TR01 | Transfer of patent right |
Effective date of registration: 20170821 Address after: Nanshan District Guangdong streets of Shenzhen science and technology of Guangdong Province in 518054 southern Shenzhen Research Institute of Tsinghua University, A304-1 Patentee after: Cleanergy Aike (Shenzhen) Energy Technology Co. Ltd Address before: 518057 Shenzhen Institute of technology, Nanshan District high tech Industrial Park, Guangdong,, Tsinghua University, A302 Patentee before: Shenzhen Institute of Stinghua University |
|
TR01 | Transfer of patent right | ||
TR01 | Transfer of patent right |
Effective date of registration: 20171019 Address after: 518057 Shenzhen Institute of technology, Nanshan District high tech Industrial Park, Guangdong,, Tsinghua University, A302 Patentee after: Shenzhen Institute of Stinghua University Address before: Nanshan District Guangdong streets of Shenzhen science and technology of Guangdong Province in 518054 southern Shenzhen Research Institute of Tsinghua University, A304-1 Patentee before: Cleanergy Aike (Shenzhen) Energy Technology Co. Ltd |
|
TR01 | Transfer of patent right | ||
TR01 | Transfer of patent right |
Effective date of registration: 20180205 Address after: 518000 Guangdong city of Shenzhen province Nanshan District Guangdong streets Technology Park A304-1 District Research Institute of Tsinghua University in Shenzhen room Patentee after: Cleanergy Aike (Shenzhen) Energy Technology Co. Ltd Address before: 518057 Shenzhen Institute of technology, Nanshan District high tech Industrial Park, Guangdong,, Tsinghua University, A302 Patentee before: Shenzhen Institute of Stinghua University |
|
TR01 | Transfer of patent right |