CN116170294B

CN116170294B - Network anomaly detection method and system for distributed system

Info

Publication number: CN116170294B
Application number: CN202310144498.8A
Authority: CN
Inventors: 柯杰伟; 徐文豪; 王弘毅; 张凯
Original assignee: SmartX Inc
Current assignee: SmartX Inc
Priority date: 2023-02-21
Filing date: 2023-02-21
Publication date: 2023-07-11
Anticipated expiration: 2043-02-21
Also published as: CN116170294A

Abstract

The invention relates to the technical field of network anomaly detection, and provides a network anomaly detection method of a distributed system, which comprises the following steps: s1: starting an L2Ping service to acquire cluster configuration information; s2: generating a plurality of random MAC addresses for each network port based on each network port, taking the random MAC addresses as head MAC addresses of L2Ping detection packets generated through L2Ping service, detecting the current network port and other network nodes for a plurality of times by adopting the L2Ping detection packets comprising different head MAC addresses for each network port, and acquiring a group of sending end MAC addresses and receiving end M AC addresses meeting the requirement of self-sending or self-sending of the current network port from detection results for the plurality of times; s3: and each network port in each network node periodically transmits an L2Ping detection packet to all other network nodes by adopting an L2Ping service, and the health state of the network port is evaluated. The method can adapt to network anomaly detection of different configuration scenes and support detection of various network anomaly types such as incapability of receiving and transmitting, high network delay, high packet loss rate and the like.

Description

Network anomaly detection method and system for distributed system

Technical Field

The present invention relates to the field of network anomaly detection technologies, and in particular, to a method and a system for detecting network anomalies in a distributed system.

Background

In a distributed system, a plurality of physical nodes in a cluster communicate by means of a network, the quality of which affects the availability of the whole system. When a network fault occurs, a method is needed to accurately and rapidly locate the abnormal component so as to repair or isolate the abnormal component and reduce the adverse effect on the whole system.

As shown in fig. 1, which is a typical network topology, a plurality of nodes of a cluster are connected to the same physical switch or logical switch (two physical switches are combined into one logical switch by using a stacking technology in the figure), and directly communicate under the same two-layer network. Each node may configure two portal constituent network bindings (hereinafter also referred to as binding) to support portal high availability and load balancing.

From the perspective of each node, the abnormal performance of the network comprises that the network port cannot transmit and receive normally, the network port transmits and receives with higher delay or packet loss rate, and the like. The network abnormality may be caused by a failure of network port hardware or drive, a failure of a network cable module, a failure or congestion of an upstream link connected to the network port, etc.

The existing network anomaly detection method can be divided into in-band detection and out-of-band detection. In-band detection typically occurs by monitoring existing traffic in the system, and out-of-band detection discovers anomalies by generating new traffic from outside. The invention belongs to an out-of-band detection method, and can discover potential abnormality in advance. The network anomaly detection method can be divided into a method of being deployed on a physical node and a method of being deployed on a physical switch according to the deployment location. The invention belongs to a method for deployment at a physical node, and is more suitable for a scene with limited access rights to a physical switch. Only the network anomaly detection method deployed on the physical node will be discussed below.

The most common detection method is to determine whether the device can communicate with the outside through a ping tool, but the device is not applicable to a scene of multi-network port network binding, because the ping flow can travel through one network port, and the ping flow cannot be controlled to detect the health state of the other network port. Aiming at the network binding of multiple network ports, different network binding technologies provide respective anomaly detection methods.

(1) VMware's Teaming supports two network failure detection strategies

and a, linkstatusonly (only judging up and down of a network link), detecting the physical link state of a network port, and detecting the scenes such as network line pulling out, direct connection physical switch power failure and the like. The undetectable scenario includes ports of the physical switch being blocked by STP; VLAN error configuration; link failure of an upstream physical switch, etc.

Beaconprobing (each physical network port periodically transmits a broadcast signal message to realize network link detection). The same host computer forms a plurality of network ports of the telecommunication to mutually send a beacon probe packet, and at least three network ports are needed. If one network port can not receive the replies of the other two network ports, the network port can be judged to be abnormal, and isolation is carried out. The detection packet mode can detect scenes which cannot be detected by the Link status policy. But does not support the scenario that the switch configures a PortChannel (multiple switch ports are combined into one logical port), because the beaconinprobe probe packet is a two-layer packet, the destination address is the MAC of the other network port, and when the probe packet is sent out from the PortChannel port, the switch will not forward it back to the same PortChannel port, and the other two network ports cannot receive the probe packet.

(2) Bonding of Linux also supports two network failure detection strategies

MIImonitor. The physical link state of the portal is detected, similar to the link ta tune of VMware.

ARPmonitor. The user may designate a plurality of target host IPs as the purpose of ARP detection, and the L inuxbinding driver periodically transmits ARP detection packets from each of the ports, and checks whether each of the ports receives any packets in one cycle, thereby determining whether the ports have a link problem. In the scenario that the switch is configured with PortChannel, ARPmontor requires the physical switch to adopt a RoundRobin load balancing mode, otherwise, when the traffic is idle, all replies of possible ARPs are received by the same network port, and other network ports can be disabled by mistake. However, none of the switches in the main stream support the RoundRobin, so this strategy cannot be applied to the PortChannel scenario.

(3) LACP anomaly detection

The LACP is a protocol for implementing link dynamic aggregation and deaggregation, and the switch port and the host network port may be configured in active or passive mode, where at least one party needs to be configured as active. The active party will actively send heartbeat to the other party, and the passive party will only return a reply. Through heartbeat, the LACP can detect whether the network port fails.

For the network binding technology in the prior art, the following defects exist when network anomaly detection is performed:

(1) And cannot adapt to different configuration scenes. For example, ping does not support a multi-portal network binding scenario; ARPmonnitor of Beaconprobing and Linux of VMware does not support PortChannel scene; LACP can only support PortChannel scenarios.

(2) It is not possible to detect a richer type of network anomaly. For example, higher network delays or packet loss rates may affect system traffic operation, but none of the above network binding techniques support detection of network delays, packet loss rates.

Disclosure of Invention

In view of the above problems, an object of the present invention is to provide a method and a system for detecting network anomalies in a distributed system, which can adapt to network anomalies detection in various different configuration scenarios, and support detection of multiple network anomalies types such as incapability of transceiving, high network delay, high packet loss rate, and the like.

The above object of the present invention is achieved by the following technical solutions:

a network anomaly detection method of a distributed system comprises the following steps:

s1: starting an L2Ping service, and acquiring cluster configuration information from a cluster configuration service, wherein the L2Ping service is a detection tool which operates inside each network node in a cluster and is used for detecting the health state of each network port on the network node, and the cluster configuration service is a service for storing the configuration information of all the network nodes in the cluster;

s2: generating a plurality of random MA (media access control) addresses for each network port based on each network port in each network node, taking the random MAC addresses as head MAC addresses of L2Ping detection packets generated by the L2Ping service, detecting the network port between the current network port and other network nodes for a plurality of times by adopting the L2Ping detection packets comprising different head MAC addresses, and acquiring a group of sending end MAC addresses and receiving end MAC addresses meeting the requirement of the current network port for self-sending or self-sending and self-receiving of the network port from detection results for the plurality of times as the sending end MAC addresses and the receiving end MAC addresses for detecting by adopting the L2Ping service subsequently;

S3: and each network port in each network node adopts the L2Ping service to periodically send the L2Ping detection packet to all other network nodes, and the health state of the network port is evaluated.

Further, in step S1, the cluster configuration information is obtained from the cluster configuration service, specifically:

when the L2Ping service is started for the first time, the cluster configuration information is acquired from the cluster configuration service, wherein the cluster configuration information comprises information including all network port MAC, network binding type, network port list associated with network binding and VLANID of the network node.

Further, in step S2, a set of the sender MAC address and the receiver MAC address that satisfy the current self-sending or self-receiving of the portal is obtained, and the set of the sender MAC address and the receiver MAC address that are detected by subsequently adopting the L2Ping service are specifically:

s21: for each network port in each network node, the L2Ping service generates a plurality of random MAC addresses for each network port, and the random MAC addresses are used as the head MAC addresses of the L2Ping detection packets generated by the L2Ping service;

S22: for the current network port, the L2Ping service sends the L2Ping detection packets with different head MAC addresses to other network nodes for a plurality of times by taking a plurality of random MAC addresses of the current network port as the sending end MAC address;

s23: after the network ports on other network nodes receive the L2Ping detection packet sent by the current network port, selecting any one of the random MAC addresses of the network ports to reply to the current network port from the network port receiving the L2Ping detection packet;

s24: after receiving replies of the network ports on other network nodes, the current network ports take a plurality of random MAC addresses of the current network ports as the MAC addresses of the sending end, take the random MAC addresses replied by the network ports on other network nodes as the MAC addresses of the receiving end, and send the L2Ping detection packets with different head MAC addresses to other network nodes again;

s25: after the network ports on other network nodes receive the L2Ping detection packet sent by the current network port again, replying the current network port from the network port receiving the L2Ping detection packet by using the MAC address of the receiving end;

S26: after receiving replies of the network ports on other network nodes, the network port acquires a group of sending end MAC addresses and receiving end MAC addresses which meet the requirement of self-sending or self-sending and receiving of the network port in a plurality of detection results, and the group of sending end MAC addresses and receiving end MAC addresses are used as the sending end MAC addresses and the receiving end MAC addresses which are detected by adopting the L2Ping service subsequently.

Further, in step S3, each of the network ports in each of the network nodes periodically sends the L2Ping probe packet to all other network nodes by using the L2Ping service, so as to evaluate the health status of the network port, specifically:

s31: each network port in each network node adopts the L2Ping service to periodically send the L2Ping detection packet to all other network nodes, and counts the receiving and transmitting conditions of the L2Ping detection packet;

s32: judging whether the current network port is isolated, if so, judging whether the current network port is recovered to be healthy, if so, releasing the isolation of the network port, if not, judging the health state of the current network port, and when the current network port is abnormal, performing operations including resetting, isolating and alarming.

Further, in step S31, the receiving and transmitting conditions of the L2Ping probe packet are counted, specifically:

configuring the waiting reply timeout time of the L2Ping detection packet, and judging that the L2Ping detection packet is lost if replies of other network nodes are not received after timeout;

configuring a detection period of the L2Ping detection packet, sending the L2Ping detection packet to other network nodes at a fixed time interval within the detection period, and counting the round trip time RTT;

and acquiring a packet loss rate according to the packet loss condition of the L2Ping detection packet, and acquiring average delay according to the counted round trip time RTT.

Further, in step S32, if the current portal is not isolated, the health status of the current portal is determined, and when the current portal is abnormal, operations including reset, isolation, and alarm are performed, specifically:

when at least one abnormal feature including incapability of completing receiving and transmitting with all other network nodes, exceeding a delay threshold value by the average delay of the network nodes and the packet loss rate of all other network nodes and exceeding a packet loss threshold value is generated in the L2Ping detection packet sent by the current network port to all other network nodes, and the abnormal feature lasts for a plurality of detection periods, judging that the current network port is abnormal;

When the current network port is abnormal, performing operations including resetting, isolation and alarming on the network port.

Further, in the operation of the L2Ping service, periodically querying the cluster configuration information, and when the cluster configuration information is changed, re-detecting the sender MAC address and the receiver MAC address of the L2Ping probe packet for health status evaluation between each current network port and other network nodes according to the new cluster configuration information; and when the L2Ping service acquires the cluster configuration information each time, caching the cluster configuration information into a local file, and when network abnormality occurs, reading the cluster configuration information from the local file.

A network anomaly detection system of a distributed system for performing the network anomaly detection method of the distributed system as described above, comprising:

the cluster configuration information acquisition module is used for starting an L2Ping service and acquiring cluster configuration information from the cluster configuration service, wherein the L2Ping service is a detection tool which operates inside each network node in a cluster and is used for detecting the health state of each network port on the network node, and the cluster configuration service is a service for storing the configuration information of all the network nodes in the cluster;

The MAC address detection module is used for generating a plurality of random MAC addresses for each network port based on each network port in each network node, taking the random MAC addresses as head MAC addresses of L2Ping detection packets generated by the L2Ping service, detecting the L2Ping detection packets between the current network port and other network nodes for a plurality of times by adopting the head MAC addresses, and acquiring a group of sending end MAC addresses and receiving end MAC addresses which meet the requirement of self-sending or self-sending of the current network port from the detection results for the plurality of times as the sending end MAC addresses and the receiving end MAC addresses which are detected by adopting the L2Ping service subsequently;

and the health state evaluation module is used for providing the L2Ping detection packet to each network port in each network node, adopting the L2Ping service to periodically send the L2Ping detection packet to all other network nodes, and evaluating the health state of the network port.

A computer device comprising a memory and one or more processors, the memory having stored therein computer code which, when executed by the one or more processors, causes the one or more processors to perform a method as described above.

A computer readable storage medium storing computer code which, when executed, performs a method as described above.

Compared with the prior art, the invention has at least one of the following beneficial effects:

(1) The method is used for detecting the sending end MAC address and the receiving end MAC address of the L2Ping detection packet which are sent to all other network nodes by adopting the L2Ping service regularly in a mode of randomly generating a plurality of groups of random MAC addresses for each network port of each network node so as to acquire the sending end MAC address and the receiving end MAC address which meet the requirement of self-sending or self-sending and receiving of the other party, and the method can be suitable for different network configuration scenes, particularly the network configuration scene with PortChannel configured at a switch end, and solves the limitation that other existing anomaly detection methods can only be suitable for individual network configuration scenes. Meanwhile, when the configuration is changed, the method can adapt in a self-adaptive mode, and the sending end MAC address and the receiving end MAC address which meet the detection requirements are found according to the new configuration, so that the abnormality detection is continued.

(2) According to good L2Ping protocol design, RTT statistics of replied detection packets of L2Ping can support detection of various network anomaly types such as incapability of receiving and transmitting, high network delay, high packet loss rate and the like in a single-network-port or multi-network-port scene, and other existing detection methods can only detect incapability of receiving and transmitting anomalies in the multi-network-port scene generally.

(3) The L2Ping service provided by the invention can quickly and accurately discover common network abnormality in a distributed system, can timely assist operation and maintenance personnel to locate network ports with network abnormality, and has certain self-repairing and fault isolation capabilities.

Drawings

FIG. 1 is a schematic diagram of a typical network topology;

FIG. 2 is a schematic diagram of the network topology of the "self-sending, self-receiving" scene and the "self-sending, opposite-receiving" scene of the present invention;

FIG. 3 is a schematic diagram of a network architecture of a network anomaly detection method of a distributed system according to the present invention

FIG. 4 is a general flow chart of a network anomaly detection method of the distributed system of the present invention;

FIG. 5 is a schematic diagram of obtaining cluster configuration information according to the present invention;

FIG. 6 is a schematic diagram of a detection PortChannel configuration of the present invention;

FIG. 7 is a schematic diagram of L2Ping detection and exception handling according to the present invention;

fig. 8 is an overall configuration diagram of a network anomaly detection system of the distributed system of the present invention.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

The invention particularly provides a network anomaly detection method and system of a distributed system, which are used for detecting the health state of a network port. The method has the following two advantages:

(1) The method can adapt to various different network configuration scenes, and typical network configuration scenes comprise: different network binding types (only one network card is used in common types, when the network card is damaged, a multi-network card binding active-backup strategy of the other network card is activated, a multi-network card binding balance strategy based on a selected transmission hash strategy, a link aggregation control protocol LACP and the like), whether PortChannel, portChannel is configured, different loadbandwidth modes (common fields such as MAC, IP, UDP/TCP ports according to a source end and a destination end are used for hash) are adopted, and the like. When the configuration is changed, the self-adaptive adaptation can be realized.

(2) And the detection of various network anomalies such as incapability of receiving and transmitting, high network delay, high packet loss rate and the like is supported.

The detection principle of the network anomaly detection method of the distributed system is briefly described as follows:

in order to detect the health status of the network port of the network node, a detection packet needs to be sent from the network port, and whether the network port is healthy is judged according to the condition that a reply is received. If no reply can be received, the network port receiving and transmitting is considered to be abnormal. If the average delay of the received reply is high or only a part of the reply of the detection packet is received, the network port has high network delay or abnormal high packet loss rate.

When the switch configures a PortChannel, for the same network node, a probe packet is sent from one portal, and a corresponding reply may be received from another portal. Existing three-layer ping tools cannot support processing replies received by other portals, and thus a new ping tool is needed.

The invention realizes the detection tool L2Ping (Layer-2 Ping) on a two-Layer network, is convenient for constructing different MAC, VLAN, IP fields and the like, and supports wider detection scenes. L2Ping creates an AF_PACKET socket for each network port based on a network data PACKET capturing function PACKET under a system library Libpcap (unix/linux platform) and based on which most network monitoring software is used for receiving and transmitting the network port. The L2Ping is regulated to use the private L2 protocol number 0x9898, and only the data packet conforming to the protocol number is limited to be received through the bpf filter supported by the Libpcap, so that the processing cost can be reduced. After the L2Ping is started, a detection packet is sent to all other network nodes of the cluster periodically, corresponding replies are received, and meanwhile, the detection packets sent by the other network nodes are responded and the replies are sent. By counting RTT (RoundTripTime) of each probe packet, the L2Ping can evaluate the health status of each network port of the network node.

Under the PortChannel configuration, for a probe packet sent by the same network node in the cluster, the probe packet may be sent from one network port of the network node, and a corresponding reply is received from another network port of the network port node, if the reply is not received, how to determine which network port has an abnormality? An anomaly may occur in a probe packet transmission process or a reply reception process. It is desirable to be able to predict in advance from which portal a reply is received to be able to determine an abnormal portal. Note that the receiving is mainly controlled by loadbandwidth policy of PortChannel, and the switch hashes the data packet according to the loadbandwidth field set by the user (usually according to MAC, IP, UDP/TCP ports of the source end and the destination end, etc.), and selects to forward the packet to one of the ports. In the implementation of the switch, the hash function selected by PortChannel is basically fixed, i.e. the reply is received by which portal, i.e. stable. Only when the user changes the policy of Loadbalance will a short time change be brought about. Therefore, the portal capable of receiving the reply can be presumed from the history of the reception situation.

Taking two network ports as an example, the corresponding relation of receiving and transmitting has 3 combinations, namely: self-transmitting and self-receiving, self-transmitting and self-receiving by one network port and self-transmitting and self-receiving by the other network port. When the receiving network port is consistent with the sending network port, the health state of the single network port can be evaluated more conveniently. By randomly generating a plurality of groups of source and destination end MAC (media access control) to try to send the detection packet, a group of source and destination end MAC meeting the requirement of 'self-sending and self-receiving' can be found according to the reply. When the hash field selected by the PortChannel is not the MAC, all replies are hashed to the same network port, so that one network port can be ensured to send and receive by itself, and the other network port can send and receive by itself. It can be seen that we can always guarantee that there is at least one portal to "send and receive by oneself". And determining which network port the reply corresponding to each network port detection packet is received from, and judging the abnormality.

As shown in the left half of fig. 2, the predicted portal 1 is a "self-sending, self-receiving" scene. If the network port 1 does not receive the reply, indicating that the network port has a fault; if a reply is received, its RTT reflects the state of the portal and can be used to determine whether there is a high delay or a high packet loss rate.

As shown in the right half of fig. 2, the prediction is that the portal 1 "sends itself, and the other receives" the scene. It is necessary to determine whether the network port 2 has a fault, and the network port 2 may perform determination according to the scenario of "self-sending and self-receiving". If the network port 2 has a fault, the network port 1 can be correspondingly recovered or isolated without checking. Only when the network port 2 is healthy, if the network port 1 does not receive a reply, the network port 1 is indicated to have a fault; if the network port 1 receives the reply, the RTT thereof may be roughly estimated as (RTT 1-1/2RTT 2) ×2, where RTT1 and RTT2 represent RTTs of the probe packets of the network port 1 and the network port 2, respectively, and may be used to determine whether the network port 1 has a high delay or a high packet loss rate.

The overall architecture of the network anomaly detection method of the distributed system is as follows:

for descriptive convenience we will only exemplify one communication plane (e.g. traffic, storage, management), but the invention supports separate detection of different communication planes, as long as they are within a two-tier network. Fig. 3 is a schematic architecture diagram of a network anomaly detection method of a distributed system according to an embodiment of the present invention, including:

Each network node (in fig. 3, the network node is a physical server) that forms a cluster, and one or more ports are configured in a single communication plane. These ports are connected to the same two-layer network, and PortChannel may be configured at the switch end. And the physical server runs the L2Ping service, and sends L2Ping detection packets from each network port to all other network nodes so as to detect whether the current network port fails. The L2Ping service relies on the library of the libpcap system to send and receive packets.

And the cluster configuration information service provides a query service of cluster configuration information. The service may have multiple instances deployed in several network nodes in the cluster, typically only one service instance is in an operational state and the other instances are in a standby state.

The protocol format of the L2Ping detection packet of the invention is as follows:

l2Ping builds on top of the Ethernet (Ethernet) layer with the message format as follows.

Wherein, dstttunmac and srcttunmac are a sender MAC address and a receiver MAC address, respectively. They are typically randomly generated to satisfy "self-send self-receive" as much as possible. 802.1Q contains VLAN information and need only be carried when VLAN configuration is involved. EtherType is Ethernet type, fixed is 0x 9898. L2PingHeader is header information of the L2Ping protocol.

The header information of the L2Ping protocol includes the following fields:

version: for distinguishing version numbers from subsequent protocol upgrades.

Operation: indicating whether an L2Ping request or reply.

Flag: the L2PingProbe option indicates whether the probe packet is used to probe a PortChannel configuration.

Seq: for finding the corresponding request when a reply is received. The Seq of the request and reply are identical.

ExpectedTunMac: indicating which TunMac address the sender wishes the receiver's reply to send using. If the value is 0, the receiving end should randomly use a Tunmac to send a reply; otherwise, the receiving end should send a reply using the specified TunMac.

Checksum: aiming at the checksum of the header of the L2Ping message, the sending end calculates, and the receiving end is responsible for checking to confirm that the message is not tampered. The checksum calculation method is not limited and may be conventional binary inverse summation.

SrcIP: and the source end IP is used for identifying the network node host.

dstIP: the destination IP is used for identifying the network node host.

First embodiment

As shown in fig. 4, the embodiment provides a network anomaly detection method of a distributed system, which specifically includes the following steps:

s1: and starting an L2Ping service, and acquiring cluster configuration information from a cluster configuration service, wherein the L2Ping service is a detection tool which is operated in each network node in a cluster and is used for detecting the health state of each network port on the network node, and the cluster configuration service is a service for storing the configuration information of all the network nodes in the cluster.

As shown in fig. 5, the cluster configuration information is obtained from the cluster configuration service, specifically:

when the L2Ping service is started for the first time, the cluster configuration information is acquired from the cluster configuration service, wherein the cluster configuration information comprises information including all network port MAC, network binding type, network port list associated with network binding and VLANID of the network node. The network port MAC in the cluster configuration information is set by a manufacturer when the network card leaves the factory, and can be used for uniquely identifying one network card. The random MAC address mentioned below is an address that we randomly generate, and is used to find a set of sender MAC address and receiver MAC address that satisfy "self-send self-receive".

S2: based on each network port in each network node, generating a plurality of random MA C addresses for each network port, taking the random MAC addresses as head MAC addresses of L2Ping detection packets generated by the L2Ping service, detecting the network port between the current network port and other network nodes for a plurality of times by adopting the L2Ping detection packets comprising different head MAC addresses, and acquiring a group of sending end MAC addresses and receiving end MAC addresses meeting the requirement of the current network port for self-sending or self-sending and self-receiving of the network port from detection results for the plurality of times as the sending end MAC addresses and the receiving end MAC addresses for detecting by adopting the L2Ping service subsequently.

The purpose of this step is to detect whether a detection PortChannel configuration exists in the cluster, and find a suitable set of sender MAC address and receiver MAC address of each other network node for each network port, for subsequent L2Ping detection. As shown in fig. 6, specifically:

s21: for each network port in each network node, the L2Ping service generates a plurality of random MAC addresses for each network port, and uses the random MAC addresses as the header MAC addresses of the L2Ping probe packets generated by the L2Ping service.

Specifically, for each portal of a network node in the network binding management, the L2Ping service may randomly generate a number of random MAC addresses (e.g. 10) for them, which may be used as the ethernet header MAC address of the L2Ping probe packet. The collision probability for randomly generating 38000 MAC addresses is calculated to be 0.001%, which is negligible. In order to ensure that the same random MAC address is always used by the host computer every time the host computer is started, a large number of MACTableentry occupying the switch is avoided, and the seed for generating random numbers by each network node host computer can be controlled to be unchanged.

S22: for the current network port, the L2Ping service sends the L2Ping detection packets with different head MAC addresses to other network nodes for a plurality of times by taking a plurality of random MAC addresses of the current network port as the sending end MAC address.

Specifically, for example, the number of random MAC addresses of the current network port is 10, 10 random MAC addresses of each network port are used as a sender MAC address, and the network port MACs of other network nodes are used as receiver MAC addresses, so as to send L2Ping probe packets to other network nodes. The ExpectedTunMac of the L2Pin gcader of these probe packets is set to 0 in order for the target network node to randomly select a random M AC address for reply. And sets the L2PingProbe option.

S23: after the network ports on other network nodes receive the L2Ping detection packet sent by the current network port, selecting any one of the random MAC addresses of the network ports from the network port receiving the L2Ping detection packet to reply to the current network port.

Specifically, for example, after receiving the probe packet, the L2Ping service of the target network node randomly selects from 10 random MAC addresses of itself to reply to the L2Ping, and sends the reply from the packet receiving network port.

S24: after receiving replies of the network ports on other network nodes, the current network ports take a plurality of random MAC addresses of the current network ports as the MAC addresses of the sending end, take the random MAC addresses replied by the network ports on other network nodes as the MAC addresses of the receiving end, and send the L2Ping detection packets with different head MAC addresses to other network nodes again for a plurality of times.

Specifically, for example, after knowing the random MAC address of the destination network node, the L2Ping of the current network port takes 10 random MAC addresses of the local network port as the MAC of the transmitting end, takes the random MAC address replied by the receiving end as the MAC address of the receiving end, and sends an L2Ping probe packet to the receiving end. The ExpectedTunMac of the L2ping header of the probe packet is set to the random MAC address that the receiver replies to, so that the receiver uses the random MAC address for reply. And sets the L2PingProbe option.

S25: after the network ports on other network nodes receive the L2Ping detection packet sent by the current network port again, replying the current network port from the network port receiving the L2Ping detection packet by using the MAC address of the receiving end.

Specifically, the L2Ping service at the receiving end receives the probe packet and sends a reply with an ExpectedTunMac. After the current network port receives the reply, the network port from which reply is received under the random MA C addresses of the current sending end and the receiving end can be determined. And (3) integrating the receiving and transmitting conditions of each network port, wherein the transmitting end L2Ping preferentially selects a group of source and destination TunMACs (namely the transmitting end MAC address and the receiving end MAC address) meeting the requirement of 'self-transmitting and self-receiving', and if the source and destination TunMACs are not met, the transmitting end L2Ping then falls back and then finds a group of source and destination TunMACs (namely the transmitting end MAC address and the receiving end MAC address) meeting the requirement of 'self-transmitting and opposite receiving'.

S3: and each network port in each network node adopts the L2Ping service to periodically send the L2Ping detection packet to all other network nodes, and the health state of the network port is evaluated. As shown in fig. 7, specifically:

s31: each network port in each network node adopts the L2Ping service to periodically send the L2Ping detection packet to all other network nodes, and counts the receiving and sending conditions of the L2Ping detection packet, which specifically includes:

and configuring the waiting reply timeout time (such as 500 ms) of the L2Ping detection packet, and judging that the L2Ping detection packet is lost if replies of other network nodes are not received after timeout.

And configuring a detection period (such as 10 s) of the L2Ping detection packet, sending the L2Ping detection packet to other network nodes at a fixed time interval (1 s) inside the detection period, and counting the round trip time RTT.

If the current network port is not isolated, judging the health state of the current network port, and performing operations including resetting, isolating and alarming when the current network port is abnormal, specifically:

when the L2Ping detection packet sent by the current network port to all other network nodes appears at least one abnormal feature including incapability of completing receiving and sending with all other network nodes, the average delay of the detection packet with all other network nodes exceeding a delay threshold value and the packet loss rate of all other network nodes exceeding a packet loss threshold value, and the abnormal feature lasts for a plurality of detection periods (such as 3), judging that the current network port is abnormal;

when the current network port is abnormal, performing operations including resetting, isolating and alarming on the network port, specifically:

the recovery action is generally to reset the network port, which can cope with the situation that the hardware of the network card or the driver is continuously in an abnormal state, and the internal states of the driver and the hardware can be favorable for the network port to recover the transceiving function after being reset.

If the network port is still abnormal after being reset, the network port can be isolated, and the influence on the network of the network node is avoided. The isolation is only aimed at a multi-port network binding scene, and on the premise that the other port is normal, the abnormal port is temporarily removed from the bonding, so that the bonding is in a single port state. If bonding only leaves one network port, after the network port is abnormal, only resetting is tried, and isolation is not carried out any more. If the removed portal subsequently recovers, the removed portal is automatically added back to the bonding.

For the LACP mode, if the network port is removed from the bonding, the switch finds that the heartbeat of the network port cannot be received, the corresponding switch port is suspended, and the L2Ping cannot continuously detect whether the network port is healthier or not. Therefore, the limitation is that the network port is not allowed to be isolated in the LACP mode, and only a reset recovery or alarm mode is adopted.

Second embodiment

As shown in fig. 8, the present embodiment provides a network anomaly detection system of a distributed system for performing a network anomaly detection method of the distributed system as in the first embodiment, including:

the cluster configuration information acquisition module 1 is used for starting an L2Ping service and acquiring cluster configuration information from the cluster configuration service, wherein the L2Ping service is a detection tool which operates inside each network node in a cluster and is used for detecting the health state of each network port on the network node, and the cluster configuration service is a service for storing the configuration information of all the network nodes in the cluster;

the MAC address detection module 2 is configured to generate, for each network port based on each network port in each network node, a plurality of random MAC addresses, and use the random MAC addresses as header MAC addresses of L2Ping detection packets generated by the L2Ping service, where each network port performs detection between the current network port and other network nodes for several times by using the L2Ping detection packets including different header MAC addresses, and obtain, in a detection result of several times, a set of sender MAC addresses and receiver MAC addresses that satisfy the current network port's own sending and receiving or own sending and receiving, as the sender MAC addresses and receiver MAC addresses that subsequently use the L2Ping service to perform detection;

And the health state evaluation module 3 is configured to provide the health state evaluation module to each network port in each network node, and periodically send the L2Ping detection packet to all other network nodes by adopting the L2Ping service to evaluate the health state of the network port.

A computer readable storage medium storing computer code which, when executed, performs a method as described above. Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be implemented by a program to instruct related hardware, the program may be stored in a computer readable storage medium, and the storage medium may include: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.

The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above examples, and all technical solutions belonging to the concept of the present invention belong to the protection scope of the present invention. It should be noted that modifications and adaptations to the present invention may occur to one skilled in the art without departing from the principles of the present invention and are intended to be within the scope of the present invention.

The technical features of the above-described embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above-described embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

It should be noted that the above embodiments can be freely combined as needed. The foregoing is merely a preferred embodiment of the present invention and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which are intended to be comprehended within the scope of the present invention.

Claims

1. The network anomaly detection method of the distributed system is characterized by comprising the following steps of:

2. The method for detecting network anomalies in a distributed system according to claim 1, characterized in that in step S1, the cluster configuration information is obtained from the cluster configuration service, in particular:

3. The network anomaly detection method of the distributed system according to claim 1, wherein in step S2, a set of the sender MAC address and the receiver MAC address that satisfy the current self-sending or self-sending-receiving of the portal is obtained as the sender MAC address and the receiver MAC address that are subsequently detected by using the L2Ping service, specifically:

4. The method for detecting network anomalies of a distributed system according to claim 1, wherein in step S3, each of the network ports in each of the network nodes periodically sends the L2Ping probe packet to all other network nodes using the L2Ping service, and the health status of the network port is evaluated, specifically:

5. The method for detecting network anomalies in a distributed system according to claim 4, wherein in step S31, the statistics of the transmission and reception of the L2Ping probe packet are specifically as follows:

configuring a detection period of the L2Ping detection packet, sending the L2Ping detection packet to other network nodes at a fixed time interval within the detection period, and counting round trip time RTT;

6. The method for detecting network anomalies of a distributed system according to claim 5, wherein in step S32, if the current portal is not isolated, the health status of the current portal is determined, and operations including resetting, isolating, and alarming are performed when the current portal is anomalous, specifically:

7. The network anomaly detection method of a distributed system of claim 1, further comprising:

During the operation of the L2Ping service, periodically inquiring the cluster configuration information, and when the cluster configuration information is changed, detecting the sending end MAC address and the receiving end MAC address of the L2Ping detection packet for carrying out health status evaluation between each current network port and other network nodes according to the new cluster configuration information;

and when the L2Ping service acquires the cluster configuration information each time, caching the cluster configuration information into a local file, and when network abnormality occurs, reading the cluster configuration information from the local file.

8. A network anomaly detection system of a distributed system for performing the network anomaly detection method of the distributed system according to claims 1 to 7, comprising:

9. A computer device comprising a memory and one or more processors, the memory having stored therein computer code that, when executed by the one or more processors, causes the one or more processors to perform the method of any of claims 1-7.

10. A computer readable storage medium storing computer code which, when executed, performs the method of any one of claims 1 to 7.