CN105052110A - 一种Hadoop集群中数据备份的方法、设备及系统 - Google Patents

一种Hadoop集群中数据备份的方法、设备及系统 Download PDF

Info

Publication number
CN105052110A
CN105052110A CN201380002586.9A CN201380002586A CN105052110A CN 105052110 A CN105052110 A CN 105052110A CN 201380002586 A CN201380002586 A CN 201380002586A CN 105052110 A CN105052110 A CN 105052110A
Authority
CN
China
Prior art keywords
calculate node
application power
data block
adjusted
power value
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201380002586.9A
Other languages
English (en)
Other versions
CN105052110B (zh
Inventor
王朱珍
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing Mingyuan Development Software Co.,Ltd.
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of CN105052110A publication Critical patent/CN105052110A/zh
Application granted granted Critical
Publication of CN105052110B publication Critical patent/CN105052110B/zh
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
    • G06F11/2094—Redundant storage or storage space

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本发明公开了一种Hadoop集群中数据备份的方法，包括：获取每个计算节点所具有的各种应用能力值，确定所述各种应用能力值中对待调整数据块的处理效率影响最大的应用能力值，以及所述待调整数据块的备份数量N，所述N为大于1的整数，确定所述每个计算节点中所述影响最大的应用能力值最强的前N个可用计算节点，将所述待调整数据块备份到确定的所述前N个可用计算节点上。本发明实施例提供的方法，可以将数据块备份到处理该数据块能力最强的计算节点上，从而提高了数据处理的效率。

Description

PCT国内申请，说明书已公开。

Claims (1)

  1. PCT国内申请，权利要求书已公开。
CN201380002586.9A 2013-12-27 2013-12-27 一种Hadoop集群中数据备份的方法、设备及系统 Active CN105052110B (zh)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2013/090723 WO2015096142A1 (zh) 2013-12-27 2013-12-27 一种Hadoop集群中数据备份的方法、设备及系统

Publications (2)

Publication Number Publication Date
CN105052110A true CN105052110A (zh) 2015-11-11
CN105052110B CN105052110B (zh) 2018-10-30

Family

ID=53477404

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201380002586.9A Active CN105052110B (zh) 2013-12-27 2013-12-27 一种Hadoop集群中数据备份的方法、设备及系统

Country Status (2)

Country Link
CN (1) CN105052110B (zh)
WO (1) WO2015096142A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108153759A (zh) * 2016-12-05 2018-06-12 中国移动通信集团公司 一种分布式数据库的数据传输方法、中间层服务器及系统

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103108029A (zh) * 2012-12-13 2013-05-15 中国科学院深圳先进技术研究院 vod系统的数据访问方法
CN103220183A (zh) * 2013-05-02 2013-07-24 杭州电子科技大学 一种基于双主机热备份的Hadoop高可用性系统实现方法

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103108029A (zh) * 2012-12-13 2013-05-15 中国科学院深圳先进技术研究院 vod系统的数据访问方法
CN103220183A (zh) * 2013-05-02 2013-07-24 杭州电子科技大学 一种基于双主机热备份的Hadoop高可用性系统实现方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108153759A (zh) * 2016-12-05 2018-06-12 中国移动通信集团公司 一种分布式数据库的数据传输方法、中间层服务器及系统
CN108153759B (zh) * 2016-12-05 2021-07-09 中国移动通信集团公司 一种分布式数据库的数据传输方法、中间层服务器及系统

Also Published As

Publication number Publication date
CN105052110B (zh) 2018-10-30
WO2015096142A1 (zh) 2015-07-02

Similar Documents

Publication Publication Date Title
US8893146B2 (en) Method and system of an I/O stack for controlling flows of workload specific I/O requests
US20230208781A1 (en) Technologies for flexible and automatic mapping of disaggregated network communication resources
CN108089814B (zh) 一种数据存储方法及装置
CN102111337A (zh) 任务调度方法和系统
US9641431B1 (en) System and methods for utilization-based balancing of traffic to an information retrieval system
CN103986766A (zh) 自适应负载均衡作业任务调度方法及装置
EP3349130B1 (en) Data compression method and storage system
CN106648456A (zh) 基于用户访问量以及预测机制的动态副本文件访问方法
CN106775949B (zh) 感知复合应用特征与网络带宽的虚拟机在线迁移优化方法
CN111522636A (zh) 应用容器的调整方法、调整系统、计算机可读介质及终端设备
CN104301243A (zh) 一种负载控制方法和装置
WO2016123972A1 (zh) 一种负载均衡的方法及负载均衡装置
WO2012106906A1 (zh) 多核处理器的流分发方法及装置
CN112825023A (zh) 集群资源管理方法、装置、电子设备和存储介质
CN105095495A (zh) 一种分布式文件系统缓存管理方法和系统
CN107294865B (zh) 一种软件交换机的负载均衡方法及软件交换机
CN110958297B (zh) 一种数据迁移方法及系统
CN103412800B (zh) 一种虚拟机热备份方法和设备
CN105162622A (zh) 一种存储方法和系统
CN105052110A (zh) 一种Hadoop集群中数据备份的方法、设备及系统
CN105740077B (zh) 一种适用于云计算的任务分配方法
CN105740076B (zh) 一种负载均衡方法及装置
DE102019113048A1 (de) Rechen-abladung in blockspeicherung unter verwendung virtueller objekte
WO2024021475A1 (zh) 一种容器调度方法及装置
CN116820324A (zh) 存储扩容方法及装置、存储介质和电子设备

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20201228

Address after: 518000 Baoan District Xin'an street, Shenzhen, Guangdong, No. 625, No. 625, Nuo platinum Plaza,

Patentee after: SHENZHEN SHANGGE INTELLECTUAL PROPERTY SERVICE Co.,Ltd.

Address before: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen

Patentee before: HUAWEI TECHNOLOGIES Co.,Ltd.

Effective date of registration: 20201228

Address after: 256600 rope net world, Lizhuang Town, Huimin County, Binzhou City, Shandong Province

Patentee after: Huimin Yisheng Yiwang Information Technology Co.,Ltd.

Address before: 518000 Baoan District Xin'an street, Shenzhen, Guangdong, No. 625, No. 625, Nuo platinum Plaza,

Patentee before: SHENZHEN SHANGGE INTELLECTUAL PROPERTY SERVICE Co.,Ltd.

TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20210602

Address after: 102600 Room 403, unit 1, 15th floor, changfengyuan, Huangcun Town, Daxing District, Beijing

Patentee after: Du Lihong

Address before: 256600 rope net world, Lizhuang Town, Huimin County, Binzhou City, Shandong Province

Patentee before: Huimin Yisheng Yiwang Information Technology Co.,Ltd.

TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20210625

Address after: Room 801-807, 8 / F, building 2, Yuhua living room, 109 software Avenue, Yuhuatai District, Nanjing City, Jiangsu Province, 210012

Patentee after: Nanjing Mingyuan Development Software Co.,Ltd.

Address before: 102600 Room 403, unit 1, 15th floor, changfengyuan, Huangcun Town, Daxing District, Beijing

Patentee before: Du Lihong