CN103986789A - Method for realizing dual redundant of NFS (network file system) nodes in HADOOP HA (home address) cluster based on NFS - Google Patents
Method for realizing dual redundant of NFS (network file system) nodes in HADOOP HA (home address) cluster based on NFS Download PDFInfo
- Publication number
- CN103986789A CN103986789A CN201410246212.8A CN201410246212A CN103986789A CN 103986789 A CN103986789 A CN 103986789A CN 201410246212 A CN201410246212 A CN 201410246212A CN 103986789 A CN103986789 A CN 103986789A
- Authority
- CN
- China
- Prior art keywords
- nfs
- hadoop
- server
- resource
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The invention discloses a method for realizing dual redundant of NFS (network file system) nodes in an HADOOP HA (home address) cluster based on an NFS and belongs to the technical field of computer servers. The method for realizing the dual redundant of the NFS nodes in the HADOOP HA cluster based on the NFS comprises the following steps: 1) mounting a DRBD (distributed replicated block device) on two servers respectively by virtue of network interaction, so that /dev/sdb1 real-time synchronization of the two servers is realized; 2) configuring NFS service, and then configuring heartbeat; 3) mounting NFS sharing on two servers used as Namenodes, and setting automatic mounting of the NFS; 4) installing HADOOP, and configuring HA of each Namenode; 5) when an NFS server monitors that resource of a main NFS server is abnormal through heatbeat, automatically taking over IP resource and DRDB resource of the NFS from the NFS server, and starting the NFS service. The method for realizing the dual redundant of the NFS nodes in the HADOOP HA cluster based on the NFS has the advantages that the stability of the HADOOP HA cluster is improved and continuous operation of a service is guaranteed.
Description
Technical field
The present invention relates to computer server technical field, specifically a kind of method of NFS node dual-computer redundancy in HADOOP HA cluster of realizing based on NFS.
Background technology
Just as we all know, NameNode is the important component part of cloud computing technology at Hadoop(hadoop, it is the most popular and the most stable instrument in current cloud computing, large data solution, be one and can carry out to mass data the software frame of distributed treatment) there is Single Point of Faliure problem in system, for addressing this problem, Hadoop 2.0 has released namenode node HA function, one is active node, one is standby node, and Active Node externally provides service as Primary NameNode.Standby Node, in Safe mode pattern, preserves the up-to-date metadata information of Primary NameNode in internal memory.Active Node and Standby Node share storage by NFS and carry out mutual edits.DataNode sends Block location information to Active Node and Standby Node simultaneously.After keeper determines that Primary NameNode breaks down, Standby Node is switched to Primary NameNode.Owing to having preserved the up-to-date information of all metadata in Standby Node internal memory, therefore can directly externally provide service.The shortcoming of this sets of plan is, it is a single-point that NFS shares storage, when NFS(is provided, is writing a Chinese character in simplified form of Network File System, i.e. NFS) during the mechanical disorder of service, the whole Hadoop cluster machine of will delaying.
Summary of the invention
Technical assignment of the present invention is to provide the method for NFS node dual-computer redundancy in a kind of HADOOP HA cluster of realizing based on NFS.
Technical assignment of the present invention is realized in the following manner, and the method step is as follows:
1) two-server is by the network interconnection, and DRBD is installed, realize two machines /dev/sdb1 real-time synchronization;
2) configuration NFS service, configuration heartbeat after completing;
3) two server carry NFS for namenode share, and the automatic carry of NFS is set;
4) Hadoop is installed, and configures the HA of namenode;
5) when monitoring the resource exception of main nfs server from nfs server by heatbeat, from nfs server, take over NFS ip resource and drbd resource, start NFS service.
Described network is Ethernet.
Described namenode node carries out carry by NFS ip, when NFS master server fault, can confirm that NFS carry is normal by df order, thereby guaranteed the normal operation of Hadoop cluster on namenode.
The HA resource of described configuration heartbeat comprises ip, the drbd resource of NFS service.
In a kind of HADOOP HA cluster of realizing based on NFS of the present invention the method for NFS node dual-computer redundancy compared to the prior art, improved cluster stability, guaranteed the continuous operation of business.
Accompanying drawing explanation
Accompanying drawing 1 is nfs server failover flow chart.
Accompanying drawing 2 is Hadoop cluster topology graph.
Embodiment
Embodiment 1:
The method step is as follows:
1) two-server is by the network interconnection, DRBD(Distributed Replicated Block Device be installed be one with software realize, without share, the storage replication solution of mirror image block device content between server), realize two machines /dev/sdb1 real-time synchronization;
2) configuration NFS service, configuration heartbeat after completing;
3) two server carry NFS for namenode share, and the automatic carry of NFS is set;
4) Hadoop is installed, and configures the HA of namenode;
5) when monitoring the resource exception of main nfs server from nfs server by heatbeat, from nfs server, take over NFS ip resource and drbd resource, start NFS service.
Embodiment 2:
The method step is as follows:
1) two-server (is referred to by Xerox company and creates and by Xerox by Ethernet, the baseband LAN standard that Intel and DEC develop jointly) interconnected, DRBD(Distributed Replicated Block Device be installed be one with software realize, without share, the storage replication solution of mirror image block device content between server), realize two machines /dev/sdb1 real-time synchronization;
2) configuration NFS service, configuration heartbeat after completing;
3) two server carry NFS for namenode share, and the automatic carry of NFS is set;
4) Hadoop is installed, and configures the HA of namenode;
5) when monitoring the resource exception of main nfs server from nfs server by heatbeat, from nfs server, take over NFS ip resource and drbd resource, start NFS service.
Embodiment 3:
The method step is as follows:
1) two-server (is referred to by Xerox company and creates and by Xerox by Ethernet, the baseband LAN standard that Intel and DEC develop jointly) interconnected, DRBD(Distributed Replicated Block Device be installed be one with software realize, without share, the storage replication solution of mirror image block device content between server), realize two machines /dev/sdb1 real-time synchronization;
2) configuration NFS service, configuration heartbeat after completing, the HA resource of configuration heartbeat comprises ip, the drbd resource of NFS service;
3) two server carry NFS for namenode share, and the automatic carry of NFS is set;
4) Hadoop is installed, and configures the HA of namenode;
5) when monitoring the resource exception of main nfs server from nfs server by heatbeat, from nfs server, take over NFS ip resource and drbd resource, start NFS service; Namenode node carries out carry by NFS ip, when NFS master server fault, can confirm that NFS carry is normal by df order, thereby guaranteed the normal operation of Hadoop cluster on namenode.
Claims (4)
1. a method for NFS node dual-computer redundancy in the HADOOP HA cluster of realization based on NFS, is characterized in that, the method step is as follows:
1) two-server is by the network interconnection, and DRBD is installed, realize two machines /dev/sdb1 real-time synchronization;
2) configuration NFS service, configuration heartbeat after completing;
3) two server carry NFS for namenode share, and the automatic carry of NFS is set;
4) Hadoop is installed, and configures the HA of namenode;
5) when monitoring the resource exception of main nfs server from nfs server by heatbeat, from nfs server, take over NFS ip resource and drbd resource, start NFS service.
2. the method for NFS node dual-computer redundancy in a kind of HADOOP HA cluster of realizing based on NFS according to claim 1, is characterized in that, described network is Ethernet.
3. the method for NFS node dual-computer redundancy in a kind of HADOOP HA cluster of realizing based on NFS according to claim 1, it is characterized in that, described namenode node carries out carry by NFS ip, when NFS master server fault, on namenode, can confirm that NFS carry is normal by df order, thereby guarantee the normal operation of Hadoop cluster.
4. the method for NFS node dual-computer redundancy in a kind of HADOOP HA cluster of realizing based on NFS according to claim 1, is characterized in that, the HA resource of described configuration heartbeat comprises ip, the drbd resource of NFS service.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410246212.8A CN103986789A (en) | 2014-06-05 | 2014-06-05 | Method for realizing dual redundant of NFS (network file system) nodes in HADOOP HA (home address) cluster based on NFS |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410246212.8A CN103986789A (en) | 2014-06-05 | 2014-06-05 | Method for realizing dual redundant of NFS (network file system) nodes in HADOOP HA (home address) cluster based on NFS |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103986789A true CN103986789A (en) | 2014-08-13 |
Family
ID=51278612
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410246212.8A Pending CN103986789A (en) | 2014-06-05 | 2014-06-05 | Method for realizing dual redundant of NFS (network file system) nodes in HADOOP HA (home address) cluster based on NFS |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103986789A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104636441A (en) * | 2015-01-07 | 2015-05-20 | 浪潮(北京)电子信息产业有限公司 | Network file system realization method and device |
| CN104679907A (en) * | 2015-03-24 | 2015-06-03 | 新余兴邦信息产业有限公司 | Realization method and system for high-availability and high-performance database cluster |
| CN105554130A (en) * | 2015-12-18 | 2016-05-04 | 深圳中兴网信科技有限公司 | Distributed storage system-based NameNode switching method and switching device |
| CN106708881A (en) * | 2015-11-17 | 2017-05-24 | 华为技术有限公司 | Interaction method and device based on network file system |
| CN107229539A (en) * | 2017-05-31 | 2017-10-03 | 郑州云海信息技术有限公司 | A kind of processing method and system for disk mirroring high-availability cluster diskless |
| CN109218386A (en) * | 2018-06-28 | 2019-01-15 | 中译语通科技股份有限公司 | A kind of high availability method managing Hadoop NameSpace |
| CN109889561A (en) * | 2017-12-25 | 2019-06-14 | 新华三大数据技术有限公司 | A kind of data processing method and device |
| CN115314361A (en) * | 2022-08-05 | 2022-11-08 | 苏州浪潮智能科技有限公司 | Server cluster management method and related components thereof |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102655460A (en) * | 2012-01-05 | 2012-09-05 | 中国工商银行股份有限公司 | Redundancy backup method and system of production server |
| CN103095837A (en) * | 2013-01-18 | 2013-05-08 | 浪潮电子信息产业股份有限公司 | Method achieving lustre metadata server redundancy |
| CN103327105A (en) * | 2013-06-26 | 2013-09-25 | 北京汉柏科技有限公司 | Automatic slave node service recovering method of hadoop system |
| CN103840961A (en) * | 2012-11-23 | 2014-06-04 | 景幂机械（上海）有限公司 | Dual-computer hot backup system |
-
2014
- 2014-06-05 CN CN201410246212.8A patent/CN103986789A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102655460A (en) * | 2012-01-05 | 2012-09-05 | 中国工商银行股份有限公司 | Redundancy backup method and system of production server |
| CN103840961A (en) * | 2012-11-23 | 2014-06-04 | 景幂机械（上海）有限公司 | Dual-computer hot backup system |
| CN103095837A (en) * | 2013-01-18 | 2013-05-08 | 浪潮电子信息产业股份有限公司 | Method achieving lustre metadata server redundancy |
| CN103327105A (en) * | 2013-06-26 | 2013-09-25 | 北京汉柏科技有限公司 | Automatic slave node service recovering method of hadoop system |
Non-Patent Citations (2)
| Title |
|---|
| 张晓波: ""基于高性能集群计算的并行文件系统关键技术研究"", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 李江昀等: ""基于Linux平台的过程控制双机热备综合解决方案"", 《计算机工程与应用》 * |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104636441A (en) * | 2015-01-07 | 2015-05-20 | 浪潮(北京)电子信息产业有限公司 | Network file system realization method and device |
| CN104636441B (en) * | 2015-01-07 | 2018-02-06 | 浪潮(北京)电子信息产业有限公司 | NFS implementation method and device |
| CN104679907A (en) * | 2015-03-24 | 2015-06-03 | 新余兴邦信息产业有限公司 | Realization method and system for high-availability and high-performance database cluster |
| WO2016150050A1 (en) * | 2015-03-24 | 2016-09-29 | 新余兴邦信息产业有限公司 | Method and system for implementing high-availability, high-performance database cluster |
| CN106708881B (en) * | 2015-11-17 | 2020-08-25 | 华为技术有限公司 | Interaction method and device based on network file system |
| CN106708881A (en) * | 2015-11-17 | 2017-05-24 | 华为技术有限公司 | Interaction method and device based on network file system |
| WO2017084587A1 (en) * | 2015-11-17 | 2017-05-26 | 华为技术有限公司 | Interactive method and apparatus based on network file system |
| CN105554130A (en) * | 2015-12-18 | 2016-05-04 | 深圳中兴网信科技有限公司 | Distributed storage system-based NameNode switching method and switching device |
| CN107229539A (en) * | 2017-05-31 | 2017-10-03 | 郑州云海信息技术有限公司 | A kind of processing method and system for disk mirroring high-availability cluster diskless |
| CN109889561A (en) * | 2017-12-25 | 2019-06-14 | 新华三大数据技术有限公司 | A kind of data processing method and device |
| CN109218386A (en) * | 2018-06-28 | 2019-01-15 | 中译语通科技股份有限公司 | A kind of high availability method managing Hadoop NameSpace |
| CN115314361A (en) * | 2022-08-05 | 2022-11-08 | 苏州浪潮智能科技有限公司 | Server cluster management method and related components thereof |
| CN115314361B (en) * | 2022-08-05 | 2023-08-22 | 苏州浪潮智能科技有限公司 | Server cluster management method and related components thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103986789A (en) | Method for realizing dual redundant of NFS (network file system) nodes in HADOOP HA (home address) cluster based on NFS | |
| US9912538B2 (en) | Synchronization of configuration file of virtual application distribution chassis | |
| US9983957B2 (en) | Failover mechanism in a distributed computing system | |
| US20190205315A1 (en) | System and method for synchronizing data between communication devices in a networked environment without a central server | |
| CN106331098B (en) | Server cluster system | |
| EP3016316B1 (en) | Network control method and apparatus | |
| CN102404390B (en) | Intelligent dynamic load balancing method for high-speed real-time database | |
| EP3288269B1 (en) | Method and system for cloud storage of video, and method and system for previewing cloud-stored video | |
| WO2015192661A1 (en) | Method, device, and system for data synchronization in distributed storage system | |
| WO2022037359A1 (en) | Configuration data access method, apparatus, and device, configuration center, and storage medium | |
| CN104320401A (en) | Big data storage and access system and method based on distributed file system | |
| CN105656653A (en) | Network access method of newly added node in distributed coordination system, device and system | |
| CN103825975A (en) | Cdn node distribution server and system | |
| CN105338095A (en) | Conversation data processing method and device | |
| WO2016177231A1 (en) | Dual-control-based active-backup switching method and device | |
| WO2020057445A1 (en) | Communication system, method, and device | |
| CN106790563A (en) | Distributed memory system and method | |
| CN111865632A (en) | Switching method of distributed data storage cluster and switching instruction sending method and device | |
| WO2016101409A1 (en) | Data switching method, device and system | |
| Aglan et al. | Reliability and scalability in SDN networks | |
| CN109587062B (en) | Load balancing information synchronization method, device and processing equipment | |
| CN112073499A (en) | Dynamic service method of multi-machine type cloud physical server | |
| WO2016065925A1 (en) | Controller replacing method and device | |
| WO2016164061A1 (en) | Big data transfer | |
| CN107819619A (en) | A kind of continual method of access for realizing NFS |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20140813 |
|
| WD01 | Invention patent application deemed withdrawn after publication |