CN107547252B

CN107547252B - Network fault processing method and device

Info

Publication number: CN107547252B
Application number: CN201710515775.6A
Authority: CN
Inventors: 马春燕; 陈杰
Original assignee: Hangzhou H3C Technologies Co Ltd
Current assignee: Hangzhou H3C Technologies Co Ltd
Priority date: 2017-06-29
Filing date: 2017-06-29
Publication date: 2020-12-04
Anticipated expiration: 2037-06-29
Also published as: CN107547252A

Abstract

The invention discloses a network fault processing method and device. The distributed storage system comprises a service network and a storage network which are separately deployed, a cluster control node MON and a plurality of object storage devices OSD are connected to the service network for communication, and a plurality of OSD are also connected to the storage network for communication, the method is applied to the OSD and comprises the following steps: detecting a storage network connection link between the OSD and the OSD matched with the OSD to obtain link state information of the OSD side; if the link state of the OSD side is detected to be abnormal, the daemon process of the OSD is actively stopped, and the OSD is prevented from sending OSD fault information matched with the OSD to the MON through a service network. The network fault processing method is applied to the OSD of the storage system, and when the link state of one side of the network is detected to be abnormal, the daemon process of the network is actively stopped, other OSD faults are prevented from being reported by mistake, and the problem of OSD oscillation in the storage network is solved.

Description

Network fault processing method and device

Technical Field

The present invention relates to the field of communications technologies, and in particular, to a method and an apparatus for processing a network fault.

Background

Ceph is an open source distributed storage system, has become one of the most common storage systems at present, and is one of the most popular open source storage items at present. The Ceph has the characteristics of high performance, high reliability, high scalability, and the like, and includes an Object Storage Device (OSD) and a cluster monitoring node (Monitor, MON). The OSD is used for providing storage resources, storage can be provided when the state is normal (up state), normal reading and writing cannot be performed when the state is abnormal (down state), and the OSD has an own daemon process (OSD decoder) which is used for completing all logic functions of the OSD, including communication with MON and other OSDs to maintain and update the system state and the like. The MON is used for receiving the status report reported by the OSD, and updating and diffusing the OSD status information (OSDMap). To maintain the global state of the entire Ceph cluster.

However, when the Ceph cluster is applied to a production environment, network connection is one of the very important factors affecting the work thereof. The network architecture proposed in the Ceph Cluster is to separately deploy a service network (Public network) and a storage network (Cluster network). The service network mainly carries user actual data, OSD and MON, and state information and heartbeat communication between MON and MON, OSD, and the storage network is mainly used for heartbeat communication between OSD and cluster internal data, such as recovery, copy, scrubbing, etc. During robustness and reliability tests, network flash or other network problems can trigger an OSD oscillation (paging) phenomenon of the storage network, which is represented by that part or all of the OSD is repeatedly set to an up state or a down state, so that service interruption is caused.

Disclosure of Invention

The application provides a network fault processing method and device, which are used for solving the problem of OSD oscillation of a storage network caused by network flash or other network problems in Ceph.

According to an aspect of the present application, there is provided a network failure handling method, in a distributed storage system, including a service network and a storage network separately deployed, a cluster control node MON and a plurality of object storage devices OSD are connected to the service network for communication, and the plurality of OSDs are further connected to the storage network for communication, the method applied to the OSDs includes:

detecting a storage network connection link between the OSD and the OSD matched with the OSD to obtain link state information of the OSD side;

if the link state of the OSD side is detected to be abnormal, the daemon process of the OSD is actively stopped, so that the OSD is prevented from sending the information of the OSD fault matched with the OSD to the MON through a service network.

According to another aspect of the present application, there is provided a network failure processing apparatus, in a distributed storage system, including a service network and a storage network separately deployed, a cluster control node MON and a plurality of object storage devices OSD are connected to the service network for communication, and the plurality of OSDs are further connected to the storage network for communication, the apparatus is applied to the OSDs, and includes:

the link detection unit is used for detecting a storage network connection link between the OSD where the device is located and the OSD matched with the OSD where the device is located to obtain link state information of the OSD side where the device is located;

and the processing unit is used for actively stopping the daemon process of the OSD when the link state of the OSD side is detected to be abnormal so as to prevent the OSD from sending the OSD fault information matched with the OSD to the MON through the service network.

The beneficial effect of this application is: the network fault processing method is applied to the OSD of the Ceph storage network, the link state information of the OSD side is obtained by detecting the storage network connection link between the OSD and the OSD matched with the OSD, and when the link state of the OSD side is detected to be abnormal, the daemon process of the OSD is actively stopped, so that the state recovery can not occur any more, the OSD fault matched with the OSD is prevented from being mistakenly reported, and the problem of OSD oscillation in the storage network is solved.

Drawings

FIG. 1 is a schematic diagram of a Ceph network architecture and communication;

FIG. 2 is a diagram illustrating a communication process between peer OSD;

fig. 3 is a schematic flowchart of a network fault handling method according to an embodiment of the present application;

fig. 4 is a schematic diagram of a network path for connection between peer OSDs through multi-level routing;

fig. 5 is a schematic structural diagram of a network fault handling apparatus according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a network fault handling apparatus according to another embodiment of the present application;

fig. 7 is a schematic structural diagram of a network fault handling apparatus according to another embodiment of the present application;

fig. 8 is a schematic structural diagram of OSD hardware according to an embodiment of the present application.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with aspects of the present application.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.

In order to make those skilled in the art better understand the technical solution of the present application, first, the structure and operation principle of the Ceph open source distributed storage system applied in the present application are introduced. As shown in fig. 1, a service network (i.e., Front in the figure) and a storage network (i.e., Back in the figure) of a Ceph cluster are separately deployed. The service network mainly bears user actual data, OSD and MON, and Map information and heartbeat communication between MON and MON, OSD, and the storage network is mainly used for heartbeat communication between OSD and cluster internal data, such as recovery, copy, scrubbing, etc.

The OSD status check flow is described as follows:

1. OSD actively reporting its state to MON

For example, the OSD reports once in a default of 900s, so that the OSD waits too long for its own report to the MON after a fault occurs. Moreover, if the reporting time is shortened, the load of reporting the OSD state processed by the MON increases linearly with the increase of the cluster size, but generally, only the OSD with the problem needs to be concerned, and most of the reporting time has little actual effect, so the effect of shortening the reporting time is not good.

2. OSD detecting OSD status of its pairing (peer)

For example, a peer relationship is established between OSDs carrying the same Placement Group (PG), or a peer relationship is established between OSDs before and after their own ID. And reporting the state of the peer OSD according to the state of the heartbeat communication.

Specifically, the detection strategy between peer OSDs is as follows:

and each OSD starts a thread, and sends heartbeat messages to each peer OSD at regular intervals (0.5-5.9 s), and if the cluster is configured with public _ network and cluster _ network, each heartbeat message is sent on two links at the same time.

As shown in fig. 2, osd.a sends a heartbeat message of PING (carrying a sending timestamp and osdmap version number of osd.a) to osd.b, which is a peer OSD, and osd.b replies a heartbeat message of PING (carrying a timestamp sent by osd.a and an osdmap version number of osd.b) to osd.a under normal conditions, and osd.a records the heartbeat information of osd.b when receiving the REPLY message of osd.b.

If the osd.a cannot receive the REPLY message of the osd.b within the preset time (default 20s, available), the osd.b is added into a failure _ queue and reported to the MON. If an OSD is reported 3 times by its peer OSD, then the MON updates osdmap and sets the state of the OSD down.

And then, the OSD self-check with down tries to bind the network card again, if the binding is successful, the state is up, if the binding is unsuccessful, the OSD is further down and out, which indicates that the OSD has completely failed and does not bear any PG any more.

However, there is a problem in the above detection strategy that Ceph can detect the hardware failure of the local network card, but cannot detect the abnormal situation of the link in the cluster network, such as pulling out the network cable, or network flash. Taking the network fault of the node B in fig. 1 as an example, the PING _ REPLY messages between osd.a and osd.b, and osd.b and osd.c that do not receive peer OSD report each other down, for example, when MON receives osd.b that is reported down by peer OSD three times continuously, OSDmap is updated, and osd.b is set down. After being down, the OSD.b starts self-checking and tries to re-bind the network card, if the network card is in failure, the binding is failed, and the state of the OSD.b cannot be changed; however, if the network cable is pulled out, or the network is flashed, or the network port of the opposite end (one end of the connection switch) fails and the network cable is not connected well, the OSD.b successfully binds the network card again at the moment, and the OSD.b state is re-up. Similarly, the osd.a and osd.c are also reported down by the osd.b, and then the network card is bound again, and the state up is realized. Therefore, multiple OSDs in the cluster are repeatedly up/down, causing long-time OSD oscillation, which inevitably causes upper layer service interruption.

Based on this, the technical idea of this application lies in: aiming at the problem of OSD oscillation caused by network flash or other network problems in the Ceph storage network in the prior art, a link detection mechanism is added in the OSD of the Ceph storage network, the link state of one side of the OSD is detected, link state information is obtained, and when the link state of one side of the OSD is detected to be abnormal, the self daemon process is actively stopped. The method comprises the steps of finding out one part of OSD which really has faults through a link detection mechanism added in the OSD, and stopping the daemon process of the part of OSD, so that the OSD can not generate state recovery any more, the matched OSD faults of the part of OSD are prevented from being reported by mistake, and the problem of OSD oscillation in a storage network is solved.

The following describes an implementation process of the network failure handling scheme according to the present application with reference to an embodiment.

Fig. 3 is a flowchart illustrating a network failure processing method according to an embodiment of the present application. In a distributed storage system, including a service network and a storage network deployed separately, a cluster control node MON and a plurality of object storage devices OSD are connected to the service network for communication, and the plurality of OSDs are also connected to the storage network for communication, the method is applied to the OSDs, and includes the following steps:

step S110, detecting a storage network connection link between the OSD and an OSD paired with the OSD to obtain link status information of the OSD side.

Step S120, if it is detected that the link status of the OSD side is abnormal, actively terminating the daemon process of the OSD to prevent the OSD from sending OSD fault information paired with the OSD to the MON through the service network.

Through the state self-checking of the OSD, after the link state of the OSD is detected to be abnormal, the daemon process of the OSD is actively stopped, and the state recovery of the OSD is prevented, so that the peer OSD network abnormality of the OSD is avoided, the problem of OSD oscillation is avoided, and the normal issuing of upper-layer services and the effective migration of data are ensured.

In the solution of the present application, the link detection includes detection of a network port and detection of a router on a network path.

For simpler networking, for example, only one level of router is arranged between each OSD, and all the OSD are connected to the same router, the state of the network port is judged, and the detection of the link state can be realized. At this time, detecting a storage network connection link between the OSD and an OSD paired with the OSD to obtain link state information of the OSD side, including:

and detecting the network port of the OSD to obtain the network port state information of the OSD.

Correspondingly, if it is detected that the link state of the OSD side is abnormal, the method actively terminates the daemon process of the OSD to prevent the OSD from sending OSD fault information paired with the OSD to the MON through the service network includes:

through the self-checking of the network port state of the OSD, if the network port state of the OSD is detected to be abnormal, the daemon process of the OSD is actively stopped, so that the OSD is prevented from sending the OSD fault information matched with the OSD to the MON through the service network.

Specifically, a timing network detection mechanism may be added to the OSD, and the OSD detects whether the network port is normal (for example, a network card drive failure, a network cable is not plugged or damaged, a switch end is pulled out, etc.) before detecting the heartbeat communication timeout every 6 seconds. When the OSD detects that the network port of the OSD is abnormal, namely the state of a direct link is abnormal, the OSD actively quits the process, so that the abnormality of Peer OSD heartbeat communication cannot be misreported, and the OSD Flapping problem can be solved.

In addition, for large-scale clusters, multi-level routing is required to be adopted for networking so as to isolate faults. Thus, the heartbeat between OSDs is no longer a single router connection, possibly spanning multiple routers. As shown in fig. 4, the OSD1 is connected to the central router C via the router a, the OSD2 is connected to the central router C via the router B, i.e., there are three levels of routers in the network path between the OSD1 and the OSD2, and the router C is the central router commonly connecting the OSD1 and the OSD 2. When the router A has a fault, the OSD1 can not receive the heartbeat response of the OSD2, and after the time is out, the Mon reports the message that the OSD2 is Down; after receiving the message, Mon issues OSDMap and sets OSD2 as Down; at this time, the OSD2 finds that its own portal is normal, reports MON, updates the state to up, and sends heartbeat message to the OSD1, however, due to the failure of the router a, the OSD2 cannot receive the heartbeat response of the OSD1, after timeout, the OSD2 reports the message that the OSD1 is Down to MON, and after receiving the message, MON sets the OSD1 to Down, and so on, the problem of flagping is generated.

The key of the problem lies in finding out the real fault party, and if we know the number of the router stages, the real fault OSD can be determined by which router the heartbeat message is blocked, so as to solve the problem of false alarm. Based on this, in some embodiments of the present application, the method further comprises:

storing cluster network topology information on the OSD; the cluster network topology information comprises the router level number of the cluster network and the position information of the central router; and the OSD acquires the position of a central router connected with the OSD and the OSD matched with the OSD together according to the topology information of the cluster network, thereby confirming which stages of routers are positioned at one side of the OSD.

The detecting a storage network connection link between the OSD and an OSD paired with the OSD to obtain link state information of the OSD side further includes:

after the OSD and the OSD heartbeat communication matched with the OSD are abnormal, the OSD sends an IP message step by step and receives messages returned by each level of router, if the messages returned by a certain level of router cannot be received, the router is judged to have a fault.

Correspondingly, if it is detected that the link state at the OSD side is abnormal, the method actively terminates the daemon process of the OSD to prevent the OSD from sending the OSD fault information paired with the OSD to the MON through the service network further includes:

and if the OSD detects that the router with the fault is between the OSD and the central router according to the cluster network topology information, the OSD actively terminates the daemon process of the OSD so as to prevent the OSD from sending the OSD fault information matched with the OSD to the MON through the service network. By adding the cluster network topology information, the OSD which really has a fault is detected, so that the fault OSD is kicked out from the cluster work, the OSD is prevented from misrepresenting other OSD as Down, and the OSD flipping problem is solved.

The cluster network topology message is created when networking is carried out. And after the hierarchical relationship of the router of the cluster network changes, if there is a newly added OSD in the cluster or there is an OSD replaced, the topology information of the cluster network is updated, based on this situation, the method further includes:

receiving and storing updated cluster network topology information sent by the OSD matched with the OSD through a service network, and sending the updated cluster network topology information to the OSD matched with the OSD; and/or receiving and storing the updated cluster network topology information sent by the MON through a service network.

Preferably, in some embodiments of the present application, an internet control message protocol ICMP protocol may be used to discover routers on a network path stage by stage. Specifically, the IP packet is an IP packet complying with the internet control message protocol ICMP protocol, and a Time To Live (TTL) field of the IP packet describes a limit value of the number of devices that can be experienced before the IP packet is discarded in the transmission process.

The step of sending the IP packet step by step and receiving the message returned by each step of router includes:

and setting the initial value of the survival time TTL of the IP message to be 1, sending, adding 1 to the TTL value of the IP message after receiving the message returned by the primary router, and sending again.

Taking OSD1 in fig. 4 as an example, OSD1 sends an IP message with a TTL initial value of 1 (actually, 3 messages of 40 bytes are sent each time, including the source address, the destination address and the time stamp of message sending) to the destination OSD 2. When the first router a on the path receives the message, it subtracts TTL from 1, at this time, TTL becomes 0, so router a will drop the message and send back an ICMP time exceeded message (including the source address of the sending IP message, all contents of the IP message and the IP address of the router), OSD1 knows that this router a exists on this path after receiving this message, and then sends out another IP message whose TTL is 2, and finds the 2 nd central router C. By this route tracking, the TTL of the sent IP packet is added by 1 each time to discover another router.

Assuming that the router a is in failure, when the OSD1 sends the first IP packet (TTL is 1) time out, according to the topology information of the cluster network, it can know that the time out route is the router a and is at one end of the OSD1, so that the OSD1 cancels the message that the OSD2 reports Down, and exits the daemon. Meanwhile, the OSD2 sends an IP message with TTL of 1 to the OSD1, the router B returns a message, the OSD2 sends an IP message with TTL of 2 again, the router C returns successfully, the OSD2 continues to send a message with TTL of 3, the IP message returns overtime due to the fault of the router A, the OSD2 knows that the central router is the central router C and has no fault according to the topology information of the cluster network, so that the OSD1 end fault can be known, the message that the OSD1 is Down is reported to the Mon, the Mon sends OSDMap after receiving the message, and the OSD1 is set as Down. Therefore, the OSD1 can actively quit the daemon process when the own link fails, thereby avoiding the error report of the peer OSD being Down, and solving the problem of OSD flagging.

Corresponding to the above method, the present application further discloses a network fault handling apparatus, in a distributed storage system, including a service network and a storage network separately deployed, where a cluster control node MON and a plurality of object storage devices OSD are connected to the service network for communication, and the plurality of OSDs are further connected to the storage network for communication, and the apparatus is applied to the OSDs, as shown in fig. 5, and is functionally divided, where the network fault handling apparatus 200 includes:

the link detection unit 210 is configured to detect a storage network connection link between the OSD where the device is located and the OSD paired with the OSD where the device is located, and obtain link state information of the OSD where the device is located.

The processing unit 220 is configured to, when it is detected that the link state of the OSD side where the OSD is located is abnormal, actively terminate the daemon process of the OSD, so as to prevent the OSD from sending the OSD fault information paired with the OSD through the service network to the MON.

Further, referring to fig. 6, in another embodiment of the present application, the link detection unit 210 includes:

the network port detecting unit 211 is configured to detect whether the network port of the OSD is normal, so as to obtain the network port status information of the OSD.

The processing unit 220 is specifically configured to, when detecting that the state of the OSD portal is abnormal, actively terminate the daemon of the OSD, so as to prevent the OSD from sending the OSD fault information paired with the OSD to the MON through the service network.

As further shown with reference to fig. 7, in another embodiment of the present application, the apparatus further comprises:

a storage unit 230, configured to store cluster network topology information; the cluster network topology information includes the number of router levels of the cluster network and the position information of the central router. The device acquires the position of a central router connected with the OSD where the device is located and the OSD matched with the OSD where the device is located together according to the topology information of the cluster network, thereby confirming which levels of routers are located on one side of the OSD where the device is located.

The link detection unit 210 further includes:

and the router detection unit 212 is configured to send an IP packet step by step and receive messages returned by the routers at each stage after the OSD and the OSD heartbeat communication paired with the OSD are abnormal, and determine that the router fails if the messages returned by a certain stage of router cannot be received.

The processing unit 220 is further configured to determine, according to the topology information of the cluster network, a location of a router in which a fault occurs; when the router with the fault is detected to be between the located OSD and the central router, the daemon process of the located OSD is actively terminated to prevent the located OSD from sending the information of the OSD fault paired with the located OSD to the MON through the service network.

Referring again to fig. 7, in some embodiments of the present application, the apparatus further comprises:

an updating unit 240, configured to receive, through a service network, updated cluster network topology information sent by an OSD paired with the located OSD after a router hierarchical relationship of the cluster network changes, send the updated cluster network topology information to the storage unit 230 for storage, and send the updated cluster network topology information to the OSD paired with the located OSD; and/or receiving the updated cluster network topology information sent by the MON through a service network, and sending the updated cluster network topology information to the storage unit 230 for storage.

Specifically, the IP packet sent by the router detection unit 212 is an IP packet complying with the internet control packet protocol ICMP protocol. The router detection unit 212 sets the TTL initial value of the IP packet to 1, and sends the IP packet, adds 1 to the TTL value of the IP packet after receiving a message returned by the first-level router, and sends the IP packet again. And when the message returned by a certain level of router cannot be received, judging that the router fails.

The network fault processing device provided by the application can be realized by software, or can be realized by hardware or a combination of hardware and software. For example, in the case of a software implementation, machine-executable instructions in the nonvolatile memory 850 corresponding to the network fault handling apparatus 200 may be read into the memory 840 by the processor 810 and executed. From a hardware aspect, as shown in fig. 8, the hardware structure of the apparatus of the present application is a structure diagram of hardware, except for the processor 810, the internal bus 820, the network interface 830, the memory 840, and the nonvolatile memory 850 shown in fig. 8, other hardware may be included according to the actual function of the OSD, which is not described again.

In various embodiments, the non-volatile memory 850 may be: a storage drive (e.g., hard disk drive), a solid state drive, any type of storage disk (e.g., compact disk, DVD, etc.), or similar storage medium, or a combination thereof. The memory 840 may be: RAM (random Access Memory), volatile Memory, nonvolatile Memory, and flash Memory.

Further, the non-volatile storage 850 and the memory 840 serve as machine-readable storage media on which machine-executable instructions corresponding to the network fault handling device 200 executed by the processor 810 may be stored.

For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims

1. A network fault processing method is characterized in that in a distributed storage system, a service network and a storage network are separately deployed, a cluster control node MON and a plurality of object storage devices OSD are connected to the service network for communication, the plurality of OSD are also connected to the storage network for communication, and the method is applied to the OSD and comprises the following steps:

if the link state of the OSD side is detected to be abnormal, actively stopping the daemon process of the OSD so as to prevent the OSD from sending OSD fault information matched with the OSD to the MON through a service network;

the method further comprises the following steps:

storing cluster network topology information; the cluster network topology information comprises the router level number of the cluster network and the position information of the central router; and the OSD acquires the position of a central router connected with the OSD and the OSD matched with the OSD together according to the topology information of the cluster network, thereby confirming which stages of routers are positioned at one side of the OSD.

2. The method of claim 1, wherein detecting a storage network connection link between the OSD and an OSD paired with the OSD to obtain link status information of the OSD side comprises:

detecting the network port of the OSD to obtain the network port state information of the OSD;

if the abnormal state of the network port of the OSD is detected, the daemon process of the OSD is actively stopped, so that the OSD is prevented from sending the information of the OSD fault matched with the OSD to the MON through a service network.

3. The method according to claim 1 or 2,

after the OSD and the OSD heartbeat communication matched with the OSD are abnormal, an IP message is sent step by step and a message returned by each level of router is received, if the message returned by a certain level of router cannot be received, the router is judged to be in fault;

correspondingly, if the link state of the OSD side is detected to be abnormal, the daemon process of the OSD is actively terminated to prevent the OSD from sending OSD fault information paired with the OSD to the MON through the service network, and the method further includes:

and according to the cluster network topology information, if the router with the fault is detected to be between the OSD and the central router, actively stopping the daemon process of the OSD so as to prevent the OSD from sending the OSD fault information matched with the OSD to the MON through a service network.

4. The method of claim 3, further comprising: after the router hierarchy of the clustered network changes,

receiving and storing updated cluster network topology information sent by the OSD matched with the OSD through a service network, and sending the updated cluster network topology information to the OSD matched with the OSD;

and/or receiving and storing the updated cluster network topology information sent by the MON through a service network.

5. The method according to claim 3, wherein the IP packet is an IP packet complying with the ICMP protocol; the step-by-step sending of the IP message and the step-by-step receiving of the message returned by each level of router comprises:

6. A network failure handling apparatus, in a distributed storage system, including a service network and a storage network separately deployed, a cluster control node MON and a plurality of object storage devices OSD are connected to the service network for communication, and the plurality of OSDs are also connected to the storage network for communication, the apparatus is applied to the OSDs, and includes:

the processing unit is used for actively stopping the daemon process of the OSD when the link state of the OSD side is detected to be abnormal so as to prevent the OSD from sending OSD fault information matched with the OSD to the MON through a service network;

the apparatus further comprises:

the storage unit is used for storing the cluster network topology information; the cluster network topology information comprises the router level number of the cluster network and the position information of the central router; the device acquires the position of a central router connected with the OSD where the device is located and the OSD matched with the OSD where the device is located together according to the topology information of the cluster network, thereby confirming which levels of routers are located on one side of the OSD where the device is located.

7. The apparatus of claim 6, wherein the link detection unit comprises:

the network port detection unit is used for detecting whether the network port of the OSD is normal or not to obtain the network port state information of the OSD;

the processing unit is specifically configured to, when it is detected that the state of the OSD portal is abnormal, actively terminate a daemon process of the OSD, so as to prevent the OSD from sending OSD fault information paired with the OSD to the MON through the service network.

8. The apparatus according to claim 6 or 7,

the link detection unit further includes:

the router detection unit is used for sending IP messages step by step and receiving messages returned by receiving routers at all levels after abnormal communication between the OSD at the position and the OSD heartbeat matched with the OSD at the position, and judging that the router fails if the messages returned by a certain level of router cannot be received;

and the processing unit is further used for actively terminating the daemon process of the located OSD when the router with the fault is detected to be located between the located OSD and the central router according to the cluster network topology information so as to prevent the located OSD from sending the information of the OSD fault matched with the located OSD to the MON through the service network.

9. The apparatus of claim 8, further comprising:

an updating unit, configured to, after a router hierarchical relationship of the cluster network changes,

receiving updated cluster network topology information sent by the OSD matched with the OSD through a service network, sending the updated cluster network topology information to the storage unit for storage, and sending the updated cluster network topology information to the OSD matched with the OSD;

and/or receiving the updated cluster network topology information sent by the MON through a service network, and sending the updated cluster network topology information to the storage unit for storage.

10. The apparatus according to claim 8, wherein the IP packet sent by the router detection unit is an IP packet complying with internet control packet protocol ICMP protocol; specifically, the router detection unit sets the TTL initial value of the IP packet to 1, and sends the IP packet, and adds 1 to the TTL value of the IP packet and sends the IP packet again after receiving a message returned by the first-level router.