CN107454165A - Access method and device of a kind of hadoop cluster to ceph clusters - Google Patents
Access method and device of a kind of hadoop cluster to ceph clusters Download PDFInfo
- Publication number
- CN107454165A CN107454165A CN201710660984.XA CN201710660984A CN107454165A CN 107454165 A CN107454165 A CN 107454165A CN 201710660984 A CN201710660984 A CN 201710660984A CN 107454165 A CN107454165 A CN 107454165A
- Authority
- CN
- China
- Prior art keywords
- ceph
- reference address
- clusters
- address
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
- H04L41/0668—Management of faults, events, alarms or notifications using network fault recovery by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The access method of ceph clusters is included the invention discloses a kind of hadoop cluster：When ceph nodes are malfunctioning node corresponding to the first reference address of target ceph clusters, it is determined that the ctdb clusters of the first reference address of storage；Second reference address for being different from the first reference address is chosen from ctdb clusters, it is hosting node to determine ceph nodes corresponding to the second reference address；The corresponding relation of hosting node and the first reference address is established, so that hadoop cluster passes through the first reference address access target ceph clusters.It can be seen that access method disclosed by the invention, when there is nodes break down, ctdb clusters can dispatch other enabled nodes and take over original reference address, realize the high availability that hadoop+ceph docking are deployed separately framework.A kind of access mechanism the invention also discloses hadoop cluster to ceph clusters, can equally realize above-mentioned technique effect.
Description
Technical field
The present invention relates to configuration management field, more specifically to a kind of access of hadoop cluster to ceph clusters
Method and device.
Background technology
Hadoop is a distributed system architecture developed by Apache funds club.User can not know about
In the case of distributed low-level details, distributed program is developed, makes full use of the power of cluster to carry out high-speed computation and storage.
Hadoop architecture is mainly that the basic-level support of distributed storage is realized by HDFS, is realized and divided by MapReduce
The program of cloth parallel task processing is supported.
Ceph storage systems are a kind of high-performance, high reliability, the storage system of enhanced scalability.Hadoop+ceph's
Docking, parallel task powerful MapReduce in the powerful distributed storage ability of ceph storage systems and hadoop is handled
Ability blends, the unstructured database in combination with HBASE towards row, so as to provide higher reliability, stronger property
Can, bigger capacity utilization and more easily big data platform.
In the case where existing hadoop+ceph docking are deployed separately framework, hadoop cluster can pass through core-site.xml texts
The ceph mon ip of part configuration access ceph clusters, but because network failure, node failure, MON extensions are waited indefinitely and cause hadoop collection
Group can not access ceph clusters, and docking framework of the prior art has Single Point of Faliure.
Therefore, how to realize that the high availability that hadoop+ceph docking are deployed separately under framework is those skilled in the art
Need to solve the problems, such as.
The content of the invention
Access method and device it is an object of the invention to provide a kind of hadoop cluster to ceph clusters, are realized
Hadoop+ceph docking are deployed separately the high availability under framework.
To achieve the above object, the access method the embodiments of the invention provide a kind of hadoop cluster to ceph clusters,
Including：
Judge whether ceph nodes are malfunctioning node corresponding to the first reference address of target ceph clusters；
If, it is determined that the ctdb clusters of storage first reference address；
One is chosen from the ctdb clusters and is different from the second reference address of first reference address, and determines institute
It is hosting node to state ceph nodes corresponding to the second reference address；
The corresponding relation of the hosting node and first reference address is established, so that the hadoop cluster passes through institute
State the first reference address and access the target ceph clusters.
Wherein it is determined that ceph nodes corresponding to second reference address be hosting node after, in addition to：
Judge whether the hosting node is enabled node；
If so, the step of then performing the corresponding relation for establishing the hosting node and first reference address；
If it is not, then choosing one from the ctdb clusters again is different from first reference address and second visit
The 3rd reference address of address is asked, when ceph nodes corresponding to the 3rd reference address are enabled node, determines described
Ceph nodes corresponding to three reference address are hosting node.
Wherein, establish after the corresponding relation of the hosting node and first reference address, in addition to：
Judge whether the failure ceph nodes recover；
If so, then re-establish the corresponding relation of the failure ceph nodes and first reference address.
Wherein, ceph nodes corresponding to first reference address for judging target ceph clusters whether be malfunctioning node it
Before, in addition to：
Ceph clusters are disposed, are each node distribution reference address in the ceph clusters；
All reference address in the ceph clusters are stored into a ctdb cluster, configuring the reference address is
The determination address of the ctdb clusters, to determine to store the ctdb clusters of the reference address according to the determination address；
Hadoop cluster is disposed, configures the ground that the reference address accesses the ceph clusters for the hadoop cluster
Location.
Wherein, ceph nodes corresponding to first reference address for judging target ceph clusters whether be malfunctioning node it
Before, in addition to：
The access request that the hadoop cluster is sent is received, the access request includes the first of target ceph clusters
Reference address.
To achieve the above object, the access mechanism the embodiments of the invention provide a kind of hadoop cluster to ceph clusters,
Including：
First judge module, for judging whether ceph nodes corresponding to the first reference address of target ceph clusters are event
Hinder node；
Cluster module is determined, for when ceph nodes are malfunctioning node corresponding to first reference address, it is determined that depositing
Store up the ctdb clusters of first reference address；
First determining module, it is different from the second of first reference address for choosing one from the ctdb clusters
Reference address, and determine that ceph nodes corresponding to second reference address are hosting node；
First update module, for establishing the corresponding relation of the hosting node and first reference address, with toilet
State hadoop cluster and the target ceph clusters are accessed by first reference address.
Wherein, in addition to：
Second judge module, for it is determined that ceph nodes corresponding to second reference address be hosting node after,
Judge whether the hosting node is enabled node, if so, first determining module is then triggered, if it is not, then triggering second is true
Cover half block；
Second determining module, it is different from first access for choosing one from the ctdb clusters again
Location and the 3rd reference address of second reference address, when ceph nodes corresponding to the 3rd reference address are enabled node
When, it is hosting node to determine ceph nodes corresponding to the 3rd reference address.
Wherein, in addition to：
3rd judge module, for judging whether the failure ceph nodes recover；
Second update module, for when the failure ceph nodes recover, re-establish the failure ceph nodes with
The corresponding relation of first reference address.
Wherein, in addition to：
Ceph cluster modules are disposed, are that each node distribution in the ceph clusters accesses for disposing ceph clusters
Address；
Ctdb cluster modules are disposed, for all reference address in the ceph clusters to be stored to a ctdb cluster
In, the determination address that the reference address is the ctdb clusters is configured, described in determining storage according to the determination address
The ctdb clusters of reference address；
Hadoop cluster module is disposed, for disposing hadoop cluster, it is the hadoop collection to configure the reference address
Group accesses the address of the ceph clusters.
Wherein, in addition to：
Receiving module, the access request sent for receiving the hadoop cluster, the access request include target
First reference address of ceph clusters.
By above scheme, access method bag of the hadoop cluster provided in an embodiment of the present invention to ceph clusters
Include：Judge whether ceph nodes are malfunctioning node corresponding to the first reference address of target ceph clusters；If, it is determined that storage
The ctdb clusters of first reference address；One is chosen from the ctdb clusters and is different from the of first reference address
Two reference address, and determine that ceph nodes corresponding to second reference address are hosting node；Establish the hosting node with
The corresponding relation of first reference address, so that the hadoop cluster accesses the target by first reference address
Ceph clusters.
Hadoop cluster provided in an embodiment of the present invention is to the access methods of ceph clusters, by all of same ceph clusters
Reference address is stored on a ctdb cluster, and when there is nodes break down, ctdb clusters can be according to other visits of itself storage
Address is asked, other corresponding enabled nodes of other reference address is dispatched and takes over its original reference address.As can be seen here, it is of the invention
The access method that embodiment provides, realize the high availability that hadoop+ceph docking are deployed separately under framework.The present invention also public affairs
A kind of access mechanism of hadoop cluster to ceph clusters has been opened, can equally realize above-mentioned technique effect.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
There is the required accompanying drawing used in technology description to be briefly described, it should be apparent that, drawings in the following description are only this
Some embodiments of invention, for those of ordinary skill in the art, on the premise of not paying creative work, can be with
Other accompanying drawings are obtained according to these accompanying drawings.
Fig. 1 is flow chart of a kind of hadoop cluster to the access method of ceph clusters disclosed in the embodiment of the present invention；
Fig. 2 is flow chart of another hadoop cluster to the access method of ceph clusters disclosed in the embodiment of the present invention；
Fig. 3 is another hadoop cluster disclosed in the embodiment of the present invention to the flow chart of the access method of ceph clusters；
Fig. 4 is structure chart of a kind of hadoop cluster to the access mechanism of ceph clusters disclosed in the embodiment of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation describes, it is clear that described embodiment is only part of the embodiment of the present invention, rather than whole embodiments.It is based on
Embodiment in the present invention, those of ordinary skill in the art are obtained every other under the premise of creative work is not made
Embodiment, belong to the scope of protection of the invention.
A kind of access method the embodiment of the invention discloses hadoop cluster to ceph clusters, realizes hadoop+
Ceph docking are deployed separately the high availability under framework.
Referring to Fig. 1, a kind of hadoop cluster disclosed in the embodiment of the present invention to the flow chart of the access method of ceph clusters,
As shown in figure 1, including：
S101：Judge whether ceph nodes are malfunctioning node corresponding to the first reference address of target ceph clusters, if so,
Then enter S102, if it is not, then the hadoop cluster accesses the target ceph clusters by first reference address；
When receiving module receives the access request for including target ceph cluster reference address of hadoop cluster transmission, need
Judge whether ceph nodes break down corresponding to the access request, can directly pass through the ceph node visit mesh if not
Ceph clusters are marked, if so, then needing to follow the steps below.
S102：It is determined that the ctdb clusters of storage first reference address；
In specific implementation, the reference address of each node is stored in a ctdb cluster in same ceph clusters, is owned
Reference address can be stored on a node in ctdb clusters, can also be stored on different nodes, not make herein
It is specific to limit.
For example, node A, node B and node C in ceph clusters A be present, reference address mon ip1, mon are corresponded to respectively
Ip2 and mon ip3, it can be stored in the node A1 in ctdb clusters A, node A1, node A2 can also be respectively stored in
In node A3.
The determination address of the ctdb clusters be the ceph clusters in all nodes reference address, i.e. a ctdb cluster
Determination address with the reference address stored in the cluster be numerically it is equal, can by any one reference address
It is determined that store its ctdb clusters.
In the above example, if reference address mon ip1, mon ip2 and mon ip3 are stored in ctdb clusters A
Node A1 in, then mon ip1, mon ip2 and mon ip3 are ctdb clusters A determination addresses.When node A breaks down
When, by determining that address mon ip1 can determine that storage reference address mon ip1 ctdb clusters A.
S103：Second reference address for being different from first reference address is chosen from the ctdb clusters, and
It is hosting node to determine ceph nodes corresponding to second reference address；
In the above example, when node A breaks down, the mon in addition to mon ip1 is chosen in ctdb clusters A
Ip2 is as the second reference address, when node B corresponding to mon ip2 is enabled node, i.e., using node B as hosting node.
S104：The corresponding relation of the hosting node and first reference address is established, so as to the hadoop cluster
The target ceph clusters are accessed by first reference address.
In specific implementation, hadoop cluster passes through node by ceph clusters reference address and the corresponding relation of node
The ceph clusters belonging to the node are accessed, that is, pass through the first address now corresponding hosting node access target ceph clusters.
In the above example, when node A breaks down, hadoop cluster can pass through mon ip1 now corresponding sections
Point B accesses ceph clusters A.
Hadoop cluster provided in an embodiment of the present invention is to the access methods of ceph clusters, by all of same ceph clusters
Reference address is stored on a ctdb cluster, and when there is nodes break down, ctdb clusters can be according to other visits of itself storage
Address is asked, other corresponding enabled nodes of other reference address is dispatched and takes over its original reference address.As can be seen here, it is of the invention
The access method that embodiment provides, realize the high availability that hadoop+ceph docking are deployed separately under framework.
A kind of access method the embodiment of the invention discloses hadoop cluster to ceph clusters, implement relative to upper one
Example, the present embodiment have made further instruction and optimization to technical scheme.Specifically：
Referring to Fig. 2, flow of another hadoop cluster provided in an embodiment of the present invention to the access method of ceph clusters
Figure, as shown in Fig. 2 including：
S201：Judge whether ceph nodes are malfunctioning node corresponding to the first reference address of target ceph clusters, if so,
Then enter S202, if it is not, then the hadoop cluster accesses the target ceph clusters by first reference address；
S202：It is determined that the ctdb clusters of storage first reference address；
S231：Second reference address for being different from first reference address is chosen from the ctdb clusters, and
It is hosting node to determine ceph nodes corresponding to second reference address；
S232：Judge whether the hosting node is enabled node, if so, then entering S204, if it is not, then entering S233；
S233：Again one is chosen from the ctdb clusters and is different from first reference address and second access
3rd reference address of address, when ceph nodes corresponding to the 3rd reference address are enabled node, determine the described 3rd
Ceph nodes corresponding to reference address are hosting node；
When ceph nodes break downs corresponding to the first reference address, it is necessary to find ceph corresponding to the second reference address
Node is as hosting node, but hosting node must available can just play a part of the reference address of adapter first, Suo Youxu
The corresponding relation of the hosting node and first reference address could be established when hosting node is enabled node, if institute
State that hosting node is unavailable, then need to choose again from ceph nodes corresponding to the reference address in ctdb clusters one it is available
Node is as hosting node.
S204：Establish the corresponding relation of the hosting node and first reference address；
S251：Judge whether the failure ceph nodes recover, if so, then entering S252, if it is not, the then hadoop collection
Group accesses the target ceph clusters by first reference address；
S252：The corresponding relation of the failure ceph nodes and first reference address is re-established, so as to described
Hadoop cluster accesses the target ceph clusters by first reference address.
A kind of access method the embodiment of the invention discloses hadoop cluster to ceph clusters, implement relative to upper one
Example, the present embodiment have made further instruction and optimization to technical scheme.Specifically：
Referring to Fig. 3, flow of another hadoop cluster to the access method of ceph clusters disclosed in the embodiment of the present invention
Figure, as shown in figure 3, including：
S311：The access request that the hadoop cluster is sent is received, the access request includes target ceph clusters
The first reference address；
When hadoop cluster asks access target ceph clusters, being sent to receiving module, which includes target ceph clusters, accesses
The access request of address, when ceph nodes do not break down corresponding to the access request, you can directly saved by the ceph
Point access target ceph clusters.
S312：Judge whether ceph nodes are malfunctioning node corresponding to the first reference address of target ceph clusters, if so,
Then enter S302, if it is not, then the hadoop cluster accesses the target ceph clusters by first reference address；
S302：It is determined that the ctdb clusters of storage first reference address；
S331：Second reference address for being different from first reference address is chosen from the ctdb clusters, and
It is hosting node to determine ceph nodes corresponding to second reference address；
S332：Judge whether the hosting node is enabled node, if so, then entering S304, if it is not, then entering S333；
S333：Again one is chosen from the ctdb clusters and is different from first reference address and second access
3rd reference address of address, when ceph nodes corresponding to the 3rd reference address are enabled node, determine the described 3rd
Ceph nodes corresponding to reference address are hosting node；
S304：Establish the corresponding relation of the hosting node and first reference address；
S351：Judge whether the failure ceph nodes recover, if so, then entering S352, if it is not, then rejudging described
Whether failure ceph nodes recover；
S352：The corresponding relation of the failure ceph nodes and first reference address is re-established, so as to described
Hadoop cluster accesses the target ceph clusters by first reference address.
On the basis of above-described embodiment, preferably, also include before the S311：
Ceph clusters are disposed, are each node distribution reference address in the ceph clusters；
There are Monitor nodes, MDS nodes (optional, to be stored for file), at least two OSD to keep in one ceph cluster
Shield process.A reference address is distributed for each node in ceph clusters, so that hadoop cluster passes through in ceph clusters
The reference address of node accesses ceph clusters.
All reference address in the ceph clusters are stored into a ctdb cluster, configuring the reference address is
The determination address of the ctdb clusters, to determine to store the ctdb clusters of the reference address according to the determination address；
Ctdb clusters are that a kind of Cluster Database of lightweight is realized, many application collection can be realized based on ctdb clusters
Group.Ctdb clusters are not HA solutions in itself, but are combined with cluster file system, and it can provide a kind of simple and effective
The solution of cluster high availability.
The reference address of all nodes in one ceph cluster is stored into a ctdb cluster, and by all access
Determination address of the address all as the ctdb clusters, so as in nodes break down corresponding to a reference address, it is determined that depositing
The ctdb clusters of the reference address are stored up, the reference address is taken over so as to dispatch other nodes in same ceph clusters.
Hadoop cluster is disposed, configures the ground that the reference address accesses the ceph clusters for the hadoop cluster
Location.
In the core-site.xml files of hadoop cluster, the reference address of each node in ceph clusters is configured to
The reference address of the ceph clusters,
The access mechanism of ceph clusters is introduced a kind of hadoop cluster provided in an embodiment of the present invention below, under
A kind of hadoop cluster of text description is to the access mechanism of ceph clusters with a kind of above-described hadoop cluster to ceph collection
The access method of group can be with cross-referenced.
Referring to Fig. 4, a kind of hadoop cluster provided in an embodiment of the present invention to the structure chart of the access mechanism of ceph clusters,
As shown in figure 4, including：
First judge module 401, for whether judging ceph nodes corresponding to the first reference address of target ceph clusters
For malfunctioning node；
Cluster module 402 is determined, for when ceph nodes are malfunctioning node corresponding to first reference address, it is determined that
Store the ctdb clusters of first reference address；
First determining module 403, it is different from first reference address for choosing one from the ctdb clusters
Second reference address, and determine that ceph nodes corresponding to second reference address are hosting node；
First update module 404, for establishing the corresponding relation of the hosting node and first reference address, so as to
The hadoop cluster accesses the target ceph clusters by first reference address.
Hadoop cluster provided in an embodiment of the present invention is to the access mechanisms of ceph clusters, by all of same ceph clusters
Reference address is stored on a ctdb cluster, and when there is nodes break down, ctdb clusters can be according to other visits of itself storage
Address is asked, other corresponding enabled nodes of other reference address is dispatched and takes over its original reference address.As can be seen here, it is of the invention
The access mechanism that embodiment provides, realize the high availability that hadoop+ceph docking are deployed separately under framework.
On the basis of above-described embodiment, preferably, in addition to：
Second judge module, for it is determined that ceph nodes corresponding to second reference address be hosting node after,
Judge whether the hosting node is enabled node, if so, first determining module is then triggered, if it is not, then triggering second is true
Cover half block；
Second determining module, it is different from first access for choosing one from the ctdb clusters again
Location and the 3rd reference address of second reference address, when ceph nodes corresponding to the 3rd reference address are enabled node
When, it is hosting node to determine ceph nodes corresponding to the 3rd reference address.
On the basis of above-described embodiment, preferably, in addition to：
3rd judge module, for judging whether the failure ceph nodes recover；
Second update module, for when the failure ceph nodes recover, re-establish the failure ceph nodes with
The corresponding relation of first reference address.
On the basis of above-described embodiment, preferably, in addition to：
Ceph cluster modules are disposed, are that each node distribution in the ceph clusters accesses for disposing ceph clusters
Address；
Ctdb cluster modules are disposed, for all reference address in the ceph clusters to be stored to a ctdb cluster
In, the determination address that the reference address is the ctdb clusters is configured, described in determining storage according to the determination address
The ctdb clusters of reference address；
Hadoop cluster module is disposed, for disposing hadoop cluster, it is the hadoop collection to configure the reference address
Group accesses the address of the ceph clusters.
On the basis of above-described embodiment, preferably, in addition to：
Receiving module, the access request sent for receiving the hadoop cluster, the access request include target
First reference address of ceph clusters.
Each embodiment is described by the way of progressive in this specification, what each embodiment stressed be and other
The difference of embodiment, between each embodiment identical similar portion mutually referring to.
The foregoing description of the disclosed embodiments, professional and technical personnel in the field are enable to realize or using the present invention.
A variety of modifications to these embodiments will be apparent for those skilled in the art, as defined herein
General Principle can be realized in other embodiments without departing from the spirit or scope of the present invention.Therefore, it is of the invention
The embodiments shown herein is not intended to be limited to, and is to fit to and principles disclosed herein and features of novelty phase one
The most wide scope caused.
Claims (10)
1. a kind of hadoop cluster is to the access method of ceph clusters, it is characterised in that including：
Judge whether ceph nodes are malfunctioning node corresponding to the first reference address of target ceph clusters；
If, it is determined that the ctdb clusters of storage first reference address；
One is chosen from the ctdb clusters and is different from the second reference address of first reference address, and determines described the
Ceph nodes corresponding to two reference address are hosting node；
The corresponding relation of the hosting node and first reference address is established, so that the hadoop cluster passes through described
One reference address accesses the target ceph clusters.
2. access method according to claim 1, it is characterised in that determine that ceph corresponding to second reference address is saved
Point be hosting node after, in addition to：
Judge whether the hosting node is enabled node；
If so, the step of then performing the corresponding relation for establishing the hosting node and first reference address；
If it is not, then choosing one from the ctdb clusters again is different from first reference address and second access
3rd reference address of location, when ceph nodes corresponding to the 3rd reference address are enabled node, determine that the described 3rd visits
It is hosting node to ask ceph nodes corresponding to address.
3. access method according to claim 2, it is characterised in that establish the hosting node and first access
After the corresponding relation of location, in addition to：
Judge whether the failure ceph nodes recover；
If so, then re-establish the corresponding relation of the failure ceph nodes and first reference address.
4. access method according to claim 3, it is characterised in that first access for judging target ceph clusters
Before whether ceph nodes are malfunctioning node corresponding to location, in addition to：
Ceph clusters are disposed, are each node distribution reference address in the ceph clusters；
All reference address in the ceph clusters are stored into a ctdb cluster, it is described to configure the reference address
The determination address of ctdb clusters, to determine to store the ctdb clusters of the reference address according to the determination address；
Hadoop cluster is disposed, configures the address that the reference address accesses the ceph clusters for the hadoop cluster.
5. according to the access method described in claim any one of 1-4, it is characterised in that described to judge the of target ceph clusters
Before whether ceph nodes are malfunctioning node corresponding to one reference address, in addition to：
The access request that the hadoop cluster is sent is received, the access request includes the first access of target ceph clusters
Address.
6. a kind of hadoop cluster is to the access mechanism of ceph clusters, it is characterised in that including：
First judge module, for judging whether ceph nodes corresponding to the first reference address of target ceph clusters are failure section
Point；
Cluster module is determined, for when ceph nodes are malfunctioning node corresponding to first reference address, it is determined that storage institute
State the ctdb clusters of the first reference address；
First determining module, for choosing second access for being different from first reference address from the ctdb clusters
Address, and determine that ceph nodes corresponding to second reference address are hosting node；
First update module, for establishing the corresponding relation of the hosting node and first reference address, so as to described
Hadoop cluster accesses the target ceph clusters by first reference address.
7. access mechanism according to claim 6, it is characterised in that also include：
Second judge module, for it is determined that ceph nodes corresponding to second reference address be hosting node after, judge
Whether the hosting node is enabled node, if so, first determining module is then triggered, if it is not, then triggering second determines mould
Block；
Second determining module, for again from the ctdb clusters choose one be different from first reference address and
3rd reference address of second reference address, when ceph nodes corresponding to the 3rd reference address are enabled node,
It is hosting node to determine ceph nodes corresponding to the 3rd reference address.
8. access mechanism according to claim 7, it is characterised in that also include：
3rd judge module, for judging whether the failure ceph nodes recover；
Second update module, for when the failure ceph nodes recover, re-establish the failure ceph nodes with it is described
The corresponding relation of first reference address.
9. access mechanism according to claim 8, it is characterised in that also include：
Ceph cluster modules are disposed, are each node distribution reference address in the ceph clusters for disposing ceph clusters；
Ctdb cluster modules are disposed, for all reference address in the ceph clusters to be stored into a ctdb cluster,
The determination address that the reference address is the ctdb clusters is configured, to determine to store the access according to the determination address
The ctdb clusters of address；
Hadoop cluster module is disposed, for disposing hadoop cluster, the reference address is configured and is visited for the hadoop cluster
Ask the address of the ceph clusters.
10. according to the access mechanism described in claim any one of 6-9, it is characterised in that also include：
Receiving module, the access request sent for receiving the hadoop cluster, the access request include target ceph
First reference address of cluster.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710660984.XA CN107454165A (en) | 2017-08-04 | 2017-08-04 | Access method and device of a kind of hadoop cluster to ceph clusters |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710660984.XA CN107454165A (en) | 2017-08-04 | 2017-08-04 | Access method and device of a kind of hadoop cluster to ceph clusters |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107454165A true CN107454165A (en) | 2017-12-08 |
Family
ID=60490527
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710660984.XA Pending CN107454165A (en) | 2017-08-04 | 2017-08-04 | Access method and device of a kind of hadoop cluster to ceph clusters |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107454165A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107562547A (en) * | 2017-08-30 | 2018-01-09 | 郑州云海信息技术有限公司 | A kind of CTDB group systems and creation method, create system |
| CN108491163A (en) * | 2018-03-19 | 2018-09-04 | 腾讯科技（深圳）有限公司 | A kind of big data processing method, device and storage medium |
| CN109446178A (en) * | 2018-11-13 | 2019-03-08 | 郑州云海信息技术有限公司 | A kind of Hadoop object storage high availability method, system, device and readable storage medium storing program for executing |
| US20230136274A1 (en) * | 2021-11-04 | 2023-05-04 | Softiron Limited | Ceph Media Failure and Remediation |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110107135A1 (en) * | 2009-11-02 | 2011-05-05 | International Business Machines Corporation | Intelligent rolling upgrade for data storage systems |
| CN102868754A (en) * | 2012-09-26 | 2013-01-09 | 北京联创信安科技有限公司 | High-availability method, node device and system for achieving cluster storage |
| CN103905531A (en) * | 2014-03-14 | 2014-07-02 | 浪潮电子信息产业股份有限公司 | Method for converting load balancing among nodes in cluster storage system |
| CN104090992A (en) * | 2014-08-06 | 2014-10-08 | 浪潮电子信息产业股份有限公司 | Method for high-availability configuration between conversion nodes in cluster storage system |
| CN105262820A (en) * | 2015-10-29 | 2016-01-20 | 浪潮电子信息产业股份有限公司 | Cluster multi-computer mutual backup method based on Linux operating system |
-
2017
- 2017-08-04 CN CN201710660984.XA patent/CN107454165A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110107135A1 (en) * | 2009-11-02 | 2011-05-05 | International Business Machines Corporation | Intelligent rolling upgrade for data storage systems |
| CN102868754A (en) * | 2012-09-26 | 2013-01-09 | 北京联创信安科技有限公司 | High-availability method, node device and system for achieving cluster storage |
| CN103905531A (en) * | 2014-03-14 | 2014-07-02 | 浪潮电子信息产业股份有限公司 | Method for converting load balancing among nodes in cluster storage system |
| CN104090992A (en) * | 2014-08-06 | 2014-10-08 | 浪潮电子信息产业股份有限公司 | Method for high-availability configuration between conversion nodes in cluster storage system |
| CN105262820A (en) * | 2015-10-29 | 2016-01-20 | 浪潮电子信息产业股份有限公司 | Cluster multi-computer mutual backup method based on Linux operating system |
Non-Patent Citations (2)
| Title |
|---|
| 李昌隆: "云存储系统中数据访问和存储接口的研究与实现", 《中国优秀硕士论文全文数据库信息科技辑》 * |
| 李淼淼: "GlusterFS分布式存储系统在云渲染平台中的应用研究", 《中国优秀硕士论文全文数据库信息科技辑》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107562547A (en) * | 2017-08-30 | 2018-01-09 | 郑州云海信息技术有限公司 | A kind of CTDB group systems and creation method, create system |
| CN108491163A (en) * | 2018-03-19 | 2018-09-04 | 腾讯科技（深圳）有限公司 | A kind of big data processing method, device and storage medium |
| CN109446178A (en) * | 2018-11-13 | 2019-03-08 | 郑州云海信息技术有限公司 | A kind of Hadoop object storage high availability method, system, device and readable storage medium storing program for executing |
| US20230136274A1 (en) * | 2021-11-04 | 2023-05-04 | Softiron Limited | Ceph Media Failure and Remediation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109492040B (en) | System suitable for processing mass short message data in data center | |
| Verma et al. | A survey on network methodologies for real-time analytics of massive IoT data and open research issues | |
| Tran et al. | Task placement on fog computing made efficient for IoT application provision | |
| US20170310596A1 (en) | Load distribution in data networks | |
| CN105933448B (en) | A kind of the micro services framework and its method of servicing of Self management | |
| CN103401917B (en) | A kind of mixing cloud computing system and its implementation method based on cloud bus | |
| CN106131213B (en) | Service management method and system | |
| US7894372B2 (en) | Topology-centric resource management for large scale service clusters | |
| CN101207550B (en) | Load balancing system and method for multi business to implement load balancing | |
| CN107454165A (en) | Access method and device of a kind of hadoop cluster to ceph clusters | |
| CN106663033B (en) | System and method for supporting a wraparound domain and proxy model and updating service information for cross-domain messaging in a transactional middleware machine environment | |
| CN110262902A (en) | Information processing method and system, medium and calculating equipment | |
| CN110266815B (en) | Micro service network system based on block chain | |
| CN108737527A (en) | One kind being suitable for platform large-scale consumer access method and system | |
| CN107172214A (en) | A kind of service node with load balancing finds method and device | |
| CN102882979A (en) | System and method for collecting, gathering and distributing data on basis of cloud computing system | |
| CN102812445A (en) | Hierarchical Multi-core Processor, Multi-core Processor System, And Control Program | |
| CN103140851B (en) | System including middleware machine environment | |
| Ranchal et al. | RADical Strategies for engineering web-scale cloud solutions | |
| He et al. | Research on architecture of internet of things platform based on service mesh | |
| CN106209680A (en) | Information processor and method | |
| CN114615251B (en) | Signal testing instrument cloud control system and method for realizing acquisition and calculation separation | |
| CN113824801B (en) | Intelligent integration terminal unified access management component system | |
| CN112351106B (en) | Service grid platform containing event grid and communication method thereof | |
| CN110457307A (en) | Metadata management system, user's cluster creation method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20171208 |
|
| RJ01 | Rejection of invention patent application after publication |