CN113691388A - Data acquisition system and method based on LVS and SNMP protocol - Google Patents

Abstract

The invention discloses a data acquisition system based on LVS and SNMP protocols, which comprises a monitoring server cluster, a monitored equipment group and an LVS module; the supervision server cluster comprises a plurality of servers, and each server comprises an Snmp request module and an Snmp response module; snmp request module: the SNMP request message is used for sending the SNMP request message of the information to be acquired to the monitored equipment according to the instruction of the user; the monitored equipment: the SNMP request module is used for receiving an SNMP request message sent by the Snmp request module, replying a corresponding response message and sending the response message to the LVS module; an LVS module: the monitoring device is used for receiving a response message sent by the monitored equipment, analyzing and finding a server with low load, and sending the response message to the corresponding Snmp response module; snmp response module: and the monitoring server is used for receiving and processing the response message sent by the LVS module to realize load balance among the monitoring server clusters.

Description

Data acquisition system and method based on LVS and SNMP protocol

Technical Field

The invention relates to a data acquisition system and a data acquisition method based on LVS and SNMP protocols, and belongs to the technical field of equipment supervision.

Background

Under normal conditions, two log collection modes are provided, wherein one mode is that SYSLOG logs are sent out by hardware equipment and reported to a collection server, and finally sampling matching is carried out according to a regular expression; the other mode is that the hardware actively detects the information by the acquisition server, and the hardware replies the corresponding information to the acquisition server. The SNMP protocol is just the second way to collect data. SNMP is a standard protocol specifically designed for managing network nodes (servers, workstations, routers, switches, HUBS, etc.) in an IP network, which is an application layer protocol. SNMP enables a network administrator to manage network performance, discover and solve network problems, and plan network growth. The network management system receives the random message (and the event report) through the SNMP to know that the network has problems.

The internal conditions of the network can be effectively managed and monitored by acquiring data through the SNMP protocol, because of the internal structure in the SNMP protocol package, the data acquired through the SNMP protocol still needs to be fed back to the monitoring server, the existing methods for acquiring data through the SNMP protocol include get, getnext, getresult and the like, and the implementation processes of the methods are that the monitoring server sends an acquisition instruction to the monitored equipment, and the monitored equipment returns response information to the monitoring server. When the number of the network internal devices to be monitored and managed is large, the interactive data between the acquisition server and the monitored device is also large, so that the load of the monitoring server is large, and even the server is crashed or crashed; meanwhile, the efficiency of data acquisition of the supervision equipment is reduced, and the more the number of the supervision equipment is, the lower the efficiency is.

Disclosure of Invention

The invention aims to provide a data acquisition system and a data acquisition method based on LVS and SNMP protocols, which aim to overcome the defect that the prior art is single server and has large load when a large number of devices in a network are monitored and managed.

A data acquisition system based on LVS and SNMP protocol comprises a monitoring server cluster, a monitored equipment group and an LVS module; the supervision server cluster comprises a plurality of servers, and each server comprises an Snmp request module and an Snmp response module;

snmp request module: the SNMP request message is used for sending the SNMP request message of the information to be acquired to the monitored equipment according to the instruction of the user;

the monitored equipment: the SNMP request module is used for receiving an SNMP request message sent by the Snmp request module, replying a corresponding response message and sending the response message to the LVS module;

an LVS module: the monitoring device is used for receiving a response message sent by the monitored equipment, analyzing and finding a server with low load, and sending the response message to the corresponding Snmp response module;

snmp response module: and the monitoring server is used for receiving and processing the response message sent by the LVS module to realize load balance among the monitoring server clusters.

Further, the LVS module comprises:

PREROUTING chain: the IP address acquisition module is used for checking whether a target IP of the discovery data packet is a local IP; if yes, sending the data packet to an INPUT chain;

an IPVS module: and comparing whether the server receiving the data packet is a cluster or not, if so, modifying the target IP address of the data packet into one real server IP in the cluster server according to the algorithm, and then sending the data packet to a POSTROUTING chain.

Further, the request message includes: IP header, UDP message header and SNMP message;

the IP in the IP header is a local IP; the UDP header comprises a source port, a destination port, a UPD message length and a checksum, wherein an original port is a server sending port where the LVS module is located; the destination port is a port for receiving the reply message by the Snmp request; the SNMP message comprises a protocol identifier, a message length, a protocol type, an improved PDU unit, a community, a request message length and a request message, wherein the protocol identifier is the SNMP protocol message for identifying the section of the log; the protocol type is the version of SNMP protocol, and the improved PDU unit types include three types: get-request, get-next-request and set-request.

A data acquisition method based on LVS and SNMP protocols, the method comprising:

the monitored equipment group receives an SNMP request sent by an Snmp request module in the server;

the monitored equipment group receives the corresponding SNMP request message, gives a response message and sends the response message to the LVS module to analyze the server with low load;

and the LVS module sends the RESPONSE message to an SNMP RESPONSE module in the server with the corresponding low load for processing, so that the load balance among the cluster servers is realized.

Further, the SNMP request message transmission content includes:

packaging the request oid into a request message in a TVL mode, and calculating the length of the request message;

encapsulating community keywords negotiated and set between the Snmp request module and the supervised device group;

the request mode for packaging and sending the message, namely a PDU unit, comprises the following three modes: get-request, get-next-request and set-request;

encapsulating the protocol types of the Snmp request module and the supervised device group;

encapsulating message body lengths sent over SNMP protocols

Packaging the SNMP protocol identifier;

packaging a PDU message header, and setting a source port as a random port and a destination port as a 161 port;

and packaging the IP message header.

Further, the response packet further includes a check, and the check includes:

checking whether a target IP in the IP header is a local IP or not;

taking out a destination port in the UDP header and checking the port;

analyzing the SNMP message, and checking whether the protocol identifier is an SNMP protocol;

checking whether the message body lengths of the SNMP message lengths are consistent;

comparing with local configuration, and checking the correctness of the protocol type;

comparing with local configuration, and checking the correctness of the community;

checking whether the length of the response message is consistent with the length of the body of the response message;

checking whether an error is reported or not, and returning error content;

comparing with local configuration, checking oid format correctness.

Further, the LVS module analysis method comprises the following steps:

after receiving the response data packet, the supervised device sends the response data packet to a PREROUTING chain of a kernel space;

the PREROUTING chain checks that the target IP of the data packet is a local machine, and sends the data packet to the INPUT chain;

comparing whether the server receiving the data packet is a cluster or not, if so, modifying the target IP address of the data packet into one real server IP in the cluster server according to an algorithm, and then sending the data packet to a POSTROUTING chain;

modifying an IP header in a data packet message into an RIP;

the snmp response module receives the data packet and analyzes the data in the data packet.

Compared with the prior art, the invention has the following beneficial effects: in the invention, a PDU unit of a SNMP protocol is split and transformed, the PDU unit is divided into two parts which are respectively applied to an SNMP-REQUEST module and an SNMP-RESPONSE module, the two modules are mutually independent, and a monitoring server is transformed into a cluster server by using an LVS technology, thereby solving the bottleneck problem of the performance of a single server; the error reporting mode of the SNMP is improved, and only the SNMP-REQUEST module has the error reporting of the SNMP.

Drawings

FIG. 1 is a working model of the data acquisition system of the present invention;

FIG. 2 is a message format sent by a snmp request module according to the present invention;

FIG. 3 is a functional principle of the present invention interacting with a terminal;

FIG. 4 is a diagram of a monitoring model of a server cluster implemented by LVS technology according to the present invention;

fig. 5 is a message format received by the snmp response module of the present invention.

Detailed Description

In order to make the technical means, the creation characteristics, the achievement purposes and the effects of the invention easy to understand, the invention is further described with the specific embodiments.

The invention relates to a method for monitoring equipment by a cluster server, which comprises the steps of improving PDU units and error states in an SNMP protocol, simultaneously modifying an IP header and a UDP header to cut off the relation between data REQUESTs and RESPONSEs, simultaneously deploying an SNMP REQUEST module and an SNMP RESPONSE module on each server, when the cluster server A sends the SNMP REQUESTs through the SNMP REQUEST module, receiving corresponding SNMP REQUEST messages by monitored equipment, then sending the RESPONSE messages to an LVS module of the cluster server A by the monitored equipment, analyzing and finding servers with low load through an LVS technology, sending the RESPONSE messages to the SNMP RESPONSE module of the corresponding server for processing, thereby achieving the purpose of monitoring the equipment by the cluster server, realizing load balance among the cluster servers, and processing a system model as shown in figure 1.

In a first aspect: a data acquisition system based on LVS and SNMP protocol comprises a monitoring server cluster, a monitored equipment group and an LVS module; the supervision server cluster comprises a plurality of servers, and each server comprises an Snmp request module and an Snmp response module;

snmp request module: the SNMP request message is used for sending the SNMP request message of the information to be acquired to the monitored equipment according to the instruction of the user;

the monitored equipment: the SNMP request module is used for receiving an SNMP request message sent by the Snmp request module, replying a corresponding response message and sending the response message to the LVS module;

an LVS module: the monitoring device is used for receiving a response message sent by the monitored equipment, analyzing and finding a server with low load, and sending the response message to the corresponding Snmp response module;

snmp response module: and the monitoring server is used for receiving and processing the response message sent by the LVS module to realize load balance among the monitoring server clusters.

As shown in fig. 4, the LVS module includes: PREROUTING chain: the IP address acquisition module is used for checking whether a target IP of the discovery data packet is a local IP; if yes, sending the data packet to an INPUT chain;

an IPVS module: comparing whether the server receiving the data packet is a cluster or not, if so, modifying the target IP address of the data packet into one real server IP in the cluster server according to an algorithm, and then sending the data packet to a POSTROUTING chain, wherein the specific processing process comprises the following steps: the IP load balancing technology of the LVS is realized through an IPVS module, the IPVS is core software of the LVS cluster system, and the main functions of the IPVS module are as follows: is installed on a Director Server, which is one of the cluster servers, and an IP address is virtually represented on the Director Server, through which the command originator must access the service. This Virtual IP is commonly referred to as the VIP of the LVS, the Virtual IP. The access request first reaches the load scheduler through the VIP, load judgment is performed on each Server in the cluster Server by calling an internal algorithm, and then a suitable Server is selected from a Real Server cluster (Real Server cluster) list by the load scheduler to be processed with response packet data, as shown in fig. 4, the specific flow is as follows.

After receiving the response data packet of the supervised device, the supervision server sends the data packet to a PREROUTING chain of a kernel space, wherein a source IP of the message is named CIP (ClientIP), and a target IP is named VIP (VirtualIP);

the PREROUTING chain checks that the target IP of the data packet is a local machine, and sends the data packet to the INPUT chain;

and comparing whether the server receiving the data packet is a cluster by an IPVS (selection server module), if so, modifying the target IP address of the data packet into one real server IP in the cluster server according to an algorithm, and then sending the data packet to a POSTROUTING chain. At the moment, the source IP of the message is CIP (ClientIP), and the target IP is RIP (RealIP);

one real server in the server cluster receives the data packet sent by the POSTROUTING link, modifies the IP header in the data packet message, and modifies the destination IP in the data packet to RIP (RealIP) when the destination IP is the server IP which sends the request.

And a snmp response module in the cluster server receives the data packet and analyzes the data in the data packet to finally obtain a response result which is wanted by people.

As shown in fig. 2, in this embodiment, the request message includes: IP header, UDP message header and SNMP message; the IP in the IP header is a local IP; the UDP header comprises a source port, a destination port, a UPD message length and a checksum, wherein an original port is a server sending port where the LVS module is located; the destination port is a port for receiving the reply message by the Snmp request; the Snmp message comprises a protocol identifier, a message length, a protocol type, an improved PDU unit, a community, a request message length and a request message, wherein the protocol identifier identifies that the section of log is an SNMP protocol message; the protocol type is the version of SNMP protocol, and the improved PDU unit types include three types: get-request, get-next-request and set-request; the advantage of this modification is that the request and the response are separated and independent, and there is no dependency relationship between them; the community is a plaintext password between a management process and an agent process, and the communities of the cluster servers need to be configured with the same community; the request message is divided into two parts: the method comprises the following steps of reporting error conditions and a variable binding list, wherein the error conditions are divided into two types: data caused by network inaccessibility is not sent to the monitored equipment, and the sent request message has incorrect format or illegal characters; the OID is mainly in the variable binding list.

In a second aspect: a data acquisition method based on LVS and SNMP protocols, the method comprising:

the monitored equipment group receives an SNMP request sent by an Snmp request module in the server;

the monitored equipment group receives the corresponding SNMP request message, gives a response message and sends the response message to the LVS module to analyze the server with low load;

and the LVS module sends the RESPONSE message to an SNMP RESPONSE module in the server with the corresponding low load for processing, so that the load balance among the cluster servers is realized.

In this embodiment, the sending of the SNMP request message includes:

packaging the request oid into a request message in a TVL mode, and calculating the length of the request message;

setting community keywords for the request message;

setting a PDU unit request mode;

setting the protocol types of the Snmp request module and the supervised device group;

calculating the length of SNMP message, and marking the packed SNMP protocol;

packaging a PDU message header, and setting a source port as a random port and a destination port as a 161 port;

and packaging the IP message header.

In this embodiment, as shown in fig. 5, the response packet further includes a check, where the check includes:

(1) checking whether a target IP in the IP header is a local IP or not;

(2) taking out a destination port in the UDP header, and checking whether the port is 10161;

(3) analyzing the SNMP message, and checking whether the protocol identifier is an SNMP protocol;

(4) checking whether the message body lengths of the SNMP message lengths are consistent;

(5) comparing with local configuration, and checking the correctness of the protocol type;

(6) checking whether the PDU unit is get-response;

(7) comparing with local configuration, and checking the correctness of the community;

(8) checking whether the length of the response message is consistent with the length of the body of the response message;

(9) checking whether an error is reported or not, and returning error content;

in this embodiment, the LVS module analysis method includes: the IP load balancing technology of the LVS module is realized through an IPVS module, the IPVS is core software of the LVS cluster system, and the IPVS module mainly has the following functions: is installed on a Director Server, which is one of the cluster servers, and an IP address is virtually represented on the Director Server, through which the command originator must access the service. This Virtual IP is commonly referred to as the VIP of the LVS, the Virtual IP. The access request first reaches the load scheduler through the VIP, load judgment is performed on each Server in the cluster Server by calling an internal algorithm, and then a suitable Server is selected from a Real Server (Real Server cluster) list by the load scheduler to be processed corresponding to the response packet data, as shown in fig. 4, the specific flow is as follows:

(1) after receiving the response data packet, the supervised device sends the response data packet to a PREROUTING chain of a kernel space, wherein a source IP of the message is named CIP (ClientIP), and a target IP is named VIP (VirtualIP);

(2) the PREROUTING chain checks that the target IP of the data packet is a local machine, and sends the data packet to the INPUT chain;

(3) and comparing whether the server receiving the data packet is a cluster by an IPVS (selection server module), if so, modifying the target IP address of the data packet into one real server IP in the cluster server according to an algorithm, and then sending the data packet to a POSTROUTING chain. At the moment, the source IP of the message is CIP (ClientIP), and the target IP is RIP (RealIP);

(4) one real server in the server cluster receives the data packet sent by the POSTROUTING link, modifies the IP header in the data packet message, and modifies the destination IP in the data packet to RIP (RealIP) when the destination IP is the server IP which sends the request.

(5) And a snmp response module in the cluster server receives the data packet and analyzes the data in the data packet to finally obtain a response result which is wanted by people.

In this embodiment, the core of the LVS technology is the IPVS module, and each device of the cluster server is analyzed according to a specific algorithm, and finally, a server with a lower load is selected to process data. IPVS realizes load balancing model mechanism has three kinds, be NAT, TUN and DR respectively, what this patent adopted is DR model mechanism. In the set of acquisition system, the IPVS module selects a dynamic scheduling algorithm: the shortest expected delay. The scheduler can judge the busy degree of the server cluster and then dynamically obtain the dispatching request according to a scheduling algorithm.

The Shortest Expected Delay scheduling (short Expected Delay) algorithm is based on the WLC algorithm, and the simple algorithm formula is as follows:

（active+1)*256/weight

where active is the number of active connections and weight is the weight.

The algorithm can effectively detect the real-time busy degree of each server in a server cluster, simultaneously considers the performance of the server to set the weight, and finally, one server with lower load can be effectively selected according to formula calculation.

The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.

Claims (7)

1. A data acquisition system based on LVS and SNMP protocol is characterized by comprising a monitoring server cluster, a monitored equipment group and an LVS module; the supervision server cluster comprises a plurality of servers, and each server comprises an Snmp request module and an Snmp response module;

snmp request module: the SNMP request message is used for sending the SNMP request message of the information to be acquired to the monitored equipment according to the instruction of the user;

the monitored equipment: the SNMP request module is used for receiving an SNMP request message sent by the Snmp request module, replying a corresponding response message and sending the response message to the LVS module;

an LVS module: the monitoring device is used for receiving a response message sent by the monitored equipment, analyzing and finding a server with low load, and sending the response message to the corresponding Snmp response module;

snmp response module: and the monitoring server is used for receiving and processing the response message sent by the LVS module to realize load balance among the monitoring server clusters.

2. The LVS and SNMP protocol based data acquisition system according to claim 1, wherein said LVS module comprises:

PREROUTING chain: the IP address acquisition module is used for checking whether a target IP of the discovery data packet is a local IP; if yes, sending the data packet to an INPUT chain;

an IPVS module: and comparing whether the server receiving the data packet is a cluster or not, if so, modifying the target IP address of the data packet into one real server IP in the cluster server according to the algorithm, and then sending the data packet to a POSTROUTING chain.

3. The LVS and SNMP protocol based data collection system according to claim 1, wherein said request message comprises: IP header, UDP message header and SNMP message;

the IP in the IP header is a local IP; the UDP header comprises a source port, a destination port, a UPD message length and a checksum, wherein an original port is a server sending port where the LVS module is located; the destination port is a port for receiving the reply message by the Snmp request; the SNMP message comprises a protocol identifier, a message length, a protocol type, an improved PDU unit, a community, a request message length and a request message, wherein the protocol identifier is the SNMP protocol message for identifying the section of the log; the protocol type is the version of SNMP protocol, and the improved PDU unit types include three types: get-request, get-next-request and set-request.

4. A data acquisition method based on LVS and SNMP protocol is characterized by comprising the following steps:

the monitored equipment group receives an SNMP request sent by an Snmp request module in the server;

the monitored equipment group receives the corresponding SNMP request message, gives a response message and sends the response message to the LVS module to analyze the server with low load;

and the LVS module sends the RESPONSE message to an SNMP RESPONSE module in the server with the corresponding low load for processing, so that the load balance among the cluster servers is realized.

5. The LVS and SNMP protocol based data collection method according to claim 4, wherein said SNMP request message sending content comprises:

packaging the request oid into a request message in a TVL mode, and calculating the length of the request message;

encapsulating community keywords negotiated and set between the Snmp request module and the supervised device group;

the request mode for packaging and sending the message, namely a PDU unit, comprises the following three modes: get-request, get-next-request and set-request;

encapsulating the protocol types of the Snmp request module and the supervised device group;

encapsulating message body lengths sent over SNMP protocols

Packaging the SNMP protocol identifier;

packaging a PDU message header, and setting a source port as a random port and a destination port as a 161 port;

and packaging the IP message header.

6. The LVS and SNMP protocol based data collection method according to claim 1, wherein said reply message further comprises a check, said check comprising:

checking whether a target IP in the IP header is a local IP or not;

taking out a destination port in the UDP header and checking the port;

analyzing the SNMP message, and checking whether the protocol identifier is an SNMP protocol;

checking whether the message body lengths of the SNMP message lengths are consistent;

comparing with local configuration, and checking the correctness of the protocol type;

comparing with local configuration, and checking the correctness of the community;

checking whether the length of the response message is consistent with the length of the body of the response message;

checking whether an error is reported or not, and returning error content;

comparing with local configuration, checking oid format correctness.

7. The LVS and SNMP protocol based data collection method according to claim 1, wherein the LVS module analysis method comprises:

after receiving the response data packet, the supervised device sends the response data packet to a PREROUTING chain of a kernel space;

the PREROUTING chain checks that the target IP of the data packet is a local machine, and sends the data packet to the INPUT chain;

comparing whether the server receiving the data packet is a cluster or not, if so, modifying the target IP address of the data packet into one real server IP in the cluster server according to an algorithm, and then sending the data packet to a POSTROUTING chain;

modifying an IP header in a data packet message into an RIP;

the snmp response module receives the data packet and analyzes the data in the data packet.

Images