CN108270599B

CN108270599B - Data analysis processing method and system based on SNMP (simple network management protocol)

Info

Publication number: CN108270599B
Application number: CN201611260525.4A
Authority: CN
Inventors: 刘路; 高建军
Original assignee: Beijing Boco Inter Telecom Technology Co ltd
Current assignee: Beijing Boco Inter Telecom Technology Co ltd
Priority date: 2016-12-30
Filing date: 2016-12-30
Publication date: 2021-05-11
Anticipated expiration: 2036-12-30
Also published as: CN108270599A

Abstract

A data analysis processing method based on SNMP protocol. The method comprises the following steps: 1) reading a message of an SNMP protocol by a data acquisition platform; 2) transposing the read SNMP message to change the longitudinal message into a transverse message; 3) and processing the horizontal message. In addition, the invention also relates to a data analysis processing system based on the SNMP protocol. Therefore, redundant information in the SNMP message is removed, and the analysis efficiency of the SNMP message is obviously improved.

Description

Data analysis processing method and system based on SNMP (simple network management protocol)

Technical Field

The invention relates to a data analysis processing method based on an SNMP protocol. The invention further relates to a system for carrying out the method according to the invention.

Background

Simple Network Management Protocol (SNMP) is an application layer Protocol in the TCP/IP suite, which is currently the most widely used Network Management Protocol in data networks, especially access networks. The SNMP protocol has the advantages of simplicity, flexibility and easy development and use. In the SNMP protocol, data Information is defined by a Management Information Base (MIB). Any one managed resource is represented as an object, also referred to as a managed object. The MIB is a collection of managed objects, which describes management data of the managed objects using a hierarchical tree structure. The SNMP protocol groups the various parameters of the devices in a tree structure, starting from the root of the tree, each hierarchical node has a code, and the codes of the hierarchy are concatenated with ". multidot.m. as separators to form a string of codes called Object Identifiers (OIDs), by means of which the parameters represented by the OIDs can be manipulated.

In the prior art, SNMP messages collected by a device are often output line by line over time, so that a plurality of collected SNMP messages are arranged longitudinally. The parsing of these messages usually needs to be realized by means of complex codes. This has the disadvantage of being costly at later maintenance and presenting problems that are difficult to locate.

Due to the wide use of routers, switches and firewalls in various fields and the popularization of fiber to the home in recent years, the number of managed devices is increasing, which leads to a wider application range of the SNMP protocol. However, SNMP parsing is highly dependent on professional and highly skilled developers. Nowadays, how to simplify the parsing process, improve the reuse degree, and improve the efficiency is especially important. The SNMP protocol has a complex structure, the analysis program of the SNMP protocol is also very complex, the SNMP protocol needs to be specially and complexly developed, and the analysis program has no universality. The existing analysis mode has a black box on the code, namely, the code analysis process is opaque and difficult to understand, and the analysis process can be accurately understood only by analyzing the code. This has the disadvantage of long cycle time for software development related to SNMP protocol. The analysis by means of the codes has larger development difficulty and longer period compared with the mode of processing files by means of a database and scripts. In addition, when the index bit number and the relevant field of the relevant OID need to be changed and the analysis logic is added, the codes cannot be unified and cannot be reused. In addition, the SNMP message headers have the same value for the same device, and there is a lot of redundant information in the case of a very large number of SNMP messages, which causes a disadvantage of low processing efficiency because a programming language such as Java must read and parse data row by row.

Disclosure of Invention

Therefore, the invention aims to provide a data analysis processing method based on the SNMP protocol. The method transposes the longitudinal message into the transverse message on the basis of regularly summarizing the SNMP message, and then processes the transverse message. By the method, the understanding and the later maintenance of the message analysis are easy, the transverse message can be directly placed in the database for processing, the code analysis is not needed any more, and the data is processed in the database, so that the message analysis efficiency is obviously improved.

The method transposes the message based on the SNMP protocol, automatically reads the OID as a header and finally readjusts the structure of the SNMP original report. The method is simpler and clearer for analysis development, improves the analysis efficiency, makes later maintenance more convenient and further processes data more flexibly. The transverse message makes the corresponding relation between each information and the OID to which the information belongs clear at a glance. The horizontal message may be processed directly or imported into a database for further processing. In this way, the data parsing process can be realized very flexibly and efficiently. The relevant OID is embodied in the header, which is very convenient for later maintenance and for field addition.

The invention provides a data analysis processing method based on an SNMP protocol. The method comprises the following steps:

-reading messages based on the SNMP protocol by means of a data acquisition platform;

-transposing the read SNMP message so that the longitudinal message becomes the lateral message;

-processing said lateral messages.

Firstly, reading an SNMP message from a data port by a data acquisition platform.

According to an advantageous embodiment of the invention, the transposition is based on the knowledge of the table OID, the field OID and the number of index bits. The field OID is the OID of the leaf node in the MIB, which is a field representing only one object. The index number of bits is the number of bits counted bit by bit from right to left to the field OID. For example, the SNMP message read is entries priorities.17409.3.3.5.1.1.1.16781569.0.1, and if the field OID of the SNMP message is entries priorities.17409.3.3.5.1.1.1, the index bit number is 3, that is, the index bit number is 16781569.0.1. The table OID is the OID of the table corresponding to the leaf node, i.e., the parent node of the field OID. Thus, the relationship between table OID and field OID is: table OID is field OID minus the last bit. In this example, the table OID is entries documents.17409.3.3.5.1.1.

In practice, if there are a plurality of SNMP messages, the plurality of SNMP messages read are arranged longitudinally line by line. The table OID and the field OID are obtained by detecting the numerical value from which bit in the SNMP messages starts to change, the mode is that the SNMP messages are detected from left to right bit by bit to determine which bit changes, all bits before the bit are the table OID, and the table OID and the bit form the field OID. Therefore, before transposing, the step of obtaining the index bit number can be subdivided into the following steps:

-determining a table OID and a field OID by detecting the SNMP message bit by bit from left to right, determining from which bit a value has changed, wherein the number of bits of the table OID is the sum of all the number of bits preceding the changed bit, and the field OID is the number of bits of the table OID plus 1;

-determining the number of bits of the index according to the field OID, wherein the number of bits of the index is the total number of bits of the SNMP message minus the number of bits of the field OID.

The transposing according to the invention is understood to mean that a plurality of SNMP messages arranged vertically are arranged horizontally. According to an advantageous embodiment of the invention, in the step of transposing the SNMP message, the read first SNMP message is kept motionless and the information value contained in the next SNMP message with the same field OID is inserted between the information value and the index of the first SNMP message. In other words, in the case of reserving the read field OID of the first SNMP message, the information value in the SNMP message having the same table OID is inserted into the original first SNMP. Therefore, the steps realize the transposition of the SNMP message from the longitudinal direction to the transverse direction, and realize the removal of redundant information after the transposition.

According to an advantageous embodiment of the invention, the transposed SNMP message can be associated in a system for organizing, storing and managing data, for example in a database. An association in the sense of the present invention is understood to be a summary of the management information of the same device. First, the transposed SNMP message is imported into the database through a data loading tool, for example, through Oracle SQL Loader. The associated horizontal messages are composed of field headers, which are composed of table OIDs, and value structures, which are composed of information values contained in the individual SNMP messages. Advantageously, depending on the type of database, suitable separators can be added between the individual values during transposition in order to simplify further processing in the database afterwards. The separator may be, for example, a vertical line "|", or the like. Data can be accessed in a database and the database system can be queried, updated and managed by usual database access methods, such as the structured query language SQL.

After the steps, the redundant information in the SNMP message is removed, and the useful information in the SNMP message becomes clearer and easier to read. The horizontal message can be directly imported into a database for further processing, and the corresponding relation between the field OID and the value is very clear. The transposing process and the subsequent data processing can also be automated directly by scripting, which can further increase the efficiency of data processing.

According to another aspect of the present invention, a data parsing processing system based on the SNMP protocol is proposed, which implements the above method according to the present invention. The system comprises the following features:

-a message collection unit for collecting SNMP messages from the data port;

-a message transpose unit for changing the read SNMP message from portrait to landscape;

-a data analysis processing unit for analyzing and processing the lateral messages.

According to an advantageous embodiment of the invention, the message transpose unit is designed to insert the information value contained in the next SNMP message between the information value and the index of the first SNMP message, keeping the read first SNMP message intact.

According to an advantageous embodiment of the invention, the data analysis processing unit can be arranged in a database. The data conversion unit and the data analysis processing unit may be implemented in hardware or software. They may have interfaces which may be constructed in hardware and/or in software. When constructed in hardware, the interface may be part of a so-called ASIC system, for example, which contains the different functions of the above-mentioned units. However, the interface may also be a separate integrated circuit or be at least partly composed of discrete components. When configured in software, the interface may be a software module that co-exists with other software modules in a database or microcontroller, for example.

The dependent claims are advantageous embodiments of the invention. Although the present invention has been described by means of preferred embodiments, it is not limited thereto but can be modified in many ways.

Drawings

FIG. 1: a flow chart of a data analysis processing method based on an SNMP protocol;

FIG. 2: a flow chart for transposing the read SNMP message;

FIG. 3: original SNMP message;

FIG. 4: an original SNMP message with a tag;

FIG. 5: a plurality of exemplary diagrams before data transposing;

FIG. 6: a plurality of example graphs after data rotation;

FIG. 7: a data analysis processing system structure schematic diagram based on an SNMP protocol;

FIG. 8: a working principle schematic diagram of a data analysis processing system based on an SNMP protocol;

FIG. 9: the data analysis processing system based on the SNMP protocol is a working principle schematic diagram.

Detailed Description

Fig. 1 shows a flowchart of a data parsing processing method based on the SNMP protocol.

And step S11, reading the message based on the SNMP protocol by the data acquisition platform.

SNMP message: each instance of a column object may be viewed as an element in the table that specifies the corresponding position of the column. The position of an element in the table may be represented by a row coordinate and a column coordinate, and correspondingly, the row coordinate of a column object may be represented by a RowIndexValue and the column coordinate by a ColumOID. The same column element has the same ColumnOID and the same row element has the same RowIndexValue. The elements in the table may be identified by the following formula: oid = ColumnOID + RowIndexValue.

ColumnOID: a regular device parameter encoding. The snmp protocol groups various parameters of equipment according to a tree structure, each level node has a code from the root of the tree, the level codes are used as separators, a string of codes formed by splicing the level codes is called an OID, and the parameters represented by the OID can be operated through the OID. Hereinafter referred to as OID.

RowIndex: it is the primary key for all objects, considering that its methods (physical and logical) for all interfaces are divided and assigned values. This value assignment activates the device during and may not be changed.

The following gives examples of SNMP messages:

SNMPv2-SMI::enterprises.17409.3.3.1.2.1.1.1.1 = INTEGER: 1

SNMPv2-SMI::enterprises.17409.3.3.1.2.1.1.2.1 = STRING: "OLT"

according to the characteristics of the SNMP message, the SNMP message sequentially comprises a table OID, a field index and a specific field value.

Step S12, transposing the read SNMP message to change the vertical message into the horizontal message.

After the SNMP message is transferred, the longitudinal message is changed into the transverse message, and the values corresponding to the messages with the same index are arranged in sequence by separating characters. After the output, a relatively intuitive standard message is formed, and the standard message can be directly put into a database for subsequent processing through some query statements such as SQLLOADER.

And step S13, processing the horizontal message.

Preferably, in order to enable the processed SNMP message to be better applied, the embodiment of the present invention may further include the following step S14 or step S15:

step S14: and processing the horizontal message into a desired file format by means of a regular expression.

Step S15: and importing the horizontal message into a database, and associating the information with the same field OID, namely summarizing the management information of the same equipment.

Fig. 2 shows a flow chart of transposing a read SNMP message.

Step S21: and determining the table OID and the field OID by detecting the change of the numerical value of the SNMP message from the head of the SNMP message bit by bit. The number of bits of the table OID is the sum of all the bits before the changed bit, and the field OID is the number of bits of the table OID plus 1.

Step S22: the number of bits of the index is determined according to the field OID. The bit number of the index is the total bit number of the SNMP message minus the bit number of the field OID.

Step S23: and keeping the read first SNMP message still, and inserting an information value contained in a next SNMP message with the same field OID between the information value and the index of the first SNMP message.

After the message traversing, the transverse message is composed of a field header and a value structure, wherein the field header is composed of a field OID, and the value structure is composed of information values contained in the SNMP message. Fig. 3 shows the original SNMP message collected from a PON (Passive Optical Network: Passive Optical Network) port. The SNMP messages are output line by line along with time change, and a plurality of collected SNMP messages are longitudinally arranged. It can be seen that these SNMP messages contain different information. Now, looking bit by bit to the right from the leftmost bit of the message, the value of the SNMP message changes from the bit shown in the left box of fig. 4, then all bits before this bit are table OIDs, this bit is a field OID bit identifying field OID, and the bits after this bit are index bits, as shown in the right box. In transpose, the information value "OLT" of the second line of messages is placed to the right of the value "1" of the first line of messages and separated by the symbol "|", forming a new line of messages. Subsequently, the string value "P3616-2 TE more card" of the third line message is placed behind the new message, i.e., behind the string value "OLT". By analogy, the values of all subsequent SNMP messages in this example are placed into the first row, and these values are distributed between the first row message value and OID index "1" in turn. Finally, the value structure of the horizontal message is "1 | OLT | P3616-2TE move card |1|5370047|1|1|1|1 |" with no field header shown. By the method, the SNMP messages which are longitudinally arranged are transposed into the horizontal direction, and the redundant information in the SNMP messages is removed.

Fig. 5 shows an exemplary diagram of pieces of data. As can be seen from fig. 5, the 8 th bit from the left of the SNMP message starts to change, and thus the total of 8 bits including this bit is a field bit and the rightmost three bits are index bits. Thus, these SNMP messages change every four rows, from 1 to 9. It can be considered that there are 8 fields, four lines of data. In the case of a plurality of pieces of data, transposition is performed similarly to the case of a single piece of data. That is, the information values in

lines

5, 9, 13, 17, 21, 25, 29 are sequentially inserted between the information value of line 1 and the index, forming a new line 1 with a separator (|) added between the respective information values. Similarly, the information in lines 6, 10, 14, 18, 22, 26, 30 is inserted between the information value and the index of line 2 in sequence to form a new line 2, and a separator (|) is also added between each information value. After applying the method to the 3 rd and 4 th rows in sequence, the resulting transposed message is as shown in fig. 6. In fig. 6, line 1 is the header of the translated horizontal table structure, and the remaining 4 lines are the value structure of the translated horizontal table structure, which contains all the information in the original SNMP messages. As can be seen from a comparison between fig. 5 and fig. 6, the data size of the SNMP message is significantly reduced after the transposition, and the network administrator can easily recognize useful information in the SNMP message.

Fig. 7 shows a data parsing processing system based on the SNMP protocol by means of the method according to the invention, said system comprising:

a message collection unit 1 for reading messages based on the SNMP protocol by means of a data collection platform.

A message transpose unit 2 for transposing a vertically arranged message into a horizontal direction by sequentially inserting an information value contained in each SNMP message having the same field OID between an information value and an index of a first SNMP message, keeping the first SNMP message still.

A data analysis processing unit 3 for processing the horizontal message.

The processing of the horizontal messages comprises processing the horizontal messages into a desired file format by means of a regular expression or importing the horizontal messages into a database, and associating information with the same field OID, namely summarizing management information of the same device.

The data analysis processing system based on the SNMP protocol is based on the data analysis processing method based on the SNMP protocol, the principle of the data analysis processing system is similar to that of the method, the specific examples and the operation principle of each part are recorded in detail in the method, and the detailed description is omitted.

Fig. 8 shows a workflow of a data parsing processing system based on the SNMP protocol. The SNMP message sent by the managed object is acquired by the message acquisition unit. In the SNMP message transposing module, the SNMP messages arranged in the vertical direction are transposed into the horizontal direction, and the respective information values are separated by separators in the process of transposing. The data can be processed on the one hand by means of regular expressions into any output file format desired. On the other hand, the data can also be imported into the database through the database import module. In the database, the association module may associate data with the same device ID, that is, aggregate information values with the same device ID. Finally, the associated information is added to the table.

Fig. 9 shows a schematic representation of the operating principle of a further data analysis processing system based on the SNMP protocol by means of the method according to the invention. The task of the system is to obtain performance information for a particular device. The data acquisition system obtains information related to the CPU and information related to the memory through the SNMP message. In the steps of CPU table transposition and memory table transposition, the SNMP messages related to the two pieces of information are transposed respectively, and then data arranged transversely are obtained. The data is imported into a database after being subjected to file processing, and is associated in the database, that is, information about the CPU and information about the memory are aggregated into performance information of the device.

Claims

1. A data analysis processing method based on SNMP protocol is characterized by comprising the following steps:

-processing said lateral messages;

the transposing the read SNMP message comprises the following steps:

-determining a table OID and a field OID by detecting from which bit starting from the header of the SNMP message the value of the SNMP message has changed bit by bit, wherein the number of bits of the table OID is the sum of all the number of bits preceding the changed bit and the field OID is the number of bits of the table OID plus 1;

-determining the number of bits of an index according to the field OID, wherein the number of bits of the index is the total number of bits of the SNMP message minus the number of bits of the field OID;

-keeping the read first SNMP message intact, inserting the information value contained in the next SNMP message with the same field OID between the information value and the index of said first SNMP message.

2. The SNMP-based data parsing processing method according to claim 1, wherein the horizontal packet is composed of a field header and a value structure, wherein the field header is composed of a field OID and the value structure is composed of an information value contained in the SNMP packet.

3. A data parsing processing method based on SNMP protocol according to claim 1, wherein in the step of processing, the horizontal packet is processed into a desired file format by means of regular expression.

4. The SNMP protocol-based data parsing processing method according to claim 2, wherein in the processing step, the horizontal packet can be imported into a database, and information with the same field OID is associated, i.e. management information of the same device is aggregated.

5. The SNMP protocol-based data parsing processing method according to claim 1, wherein the method is capable of being automatically implemented by a program.

6. The SNMP protocol-based data parsing processing method according to claim 1, wherein a delimiter is added between information values included in a next SNMP message having a same field OID when the information value is inserted between the information value and an index of the first SNMP message.

7. The SNMP protocol-based data parsing processing method according to claim 6, wherein the delimiter is a vertical line.

8. The SNMP protocol-based data parsing processing method according to claim 1, wherein the data parsing processing method is adapted to a plurality of SNMP messages having different field OIDs.

9. A data parsing processing system based on the SNMP protocol for implementing the method according to any of the preceding claims, comprising the following features:

-a message collection unit for reading messages based on the SNMP protocol by means of a data collection platform;

-a message transposing unit for transposing a vertically arranged message into a landscape orientation, the transposing of a vertically arranged message into a landscape orientation comprising the steps of:

-keeping the read first SNMP message intact, inserting the information value contained in the next SNMP message with the same field OID between the information value and the index of said first SNMP message;

-a data analysis processing unit for processing the horizontal packets; the processing of the horizontal messages comprises processing the horizontal messages into a desired file format by means of a regular expression or importing the horizontal messages into a database, and associating information with the same field OID, namely summarizing management information of the same device.