CN117014527A

CN117014527A - Data processing method and device, storage medium and electronic equipment

Info

Publication number: CN117014527A
Application number: CN202210494020.3A
Authority: CN
Inventors: 杨子尧; 张德智; 汤健; 梁真铭
Original assignee: China Telecom Corp Ltd
Current assignee: China Telecom Corp Ltd
Priority date: 2022-04-29
Filing date: 2022-04-29
Publication date: 2023-11-07

Abstract

The disclosure provides a data processing method, a data processing device, a storage medium and electronic equipment; relates to the technical field of communication. The method comprises the following steps: determining a data type of the acquired data, wherein the data type comprises one of a difference data type, an abnormal data type, a dynamic length field type, a fixed length field type and a length separation type; encoding the acquired data according to the data type of the acquired data to obtain encoded data; and sending the encoded data to a collector so that the collector analyzes the encoded data. According to the method and the device, the data types of the acquired data are expanded, the difference data types and the abnormal data types are increased, and flexible processing of the acquired data is realized, so that the processing efficiency of the acquired data is improved.

Description

Data processing method and device, storage medium and electronic equipment

Technical Field

The present disclosure relates to the field of communication technologies, and in particular, to a data processing method, a data processing apparatus, a computer readable storage medium, and an electronic device.

Background

With increasing device sizes of SDN (Software Defined Network, software defined networking) networks, more and more services are carried, and users have raised higher requirements on intelligent operation and maintenance of SDN networks. For example, the accuracy of the monitoring data needs to be higher so that the micro burst traffic can be detected and quickly adjusted in time, and meanwhile, the monitoring process has little influence on the functions and performances of the device itself so that the utilization rate of the device and the network can be improved. Based on this, telemet technology has been developed.

The technology is a technology for remotely collecting data from physical equipment or virtual equipment at a high speed, and adopts an active push mode, for example, information such as interface flow statistics, CPU (Central Processing Unit ) or memory data of the equipment can be periodically and actively sent to the collector through the push mode. Moreover, the technology of Telemetry supports structured data, has higher execution efficiency and more real-time sub-second acquisition precision, has small influence on functions and performances of equipment, can provide important big data analysis basis for quick positioning of network problems and optimization and adjustment of network quality by combining SDN application, and meets the operation and maintenance requirements of refinement, visualization and intelligent monitoring.

It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the present disclosure and thus may include information that does not constitute prior art known to those of ordinary skill in the art.

Disclosure of Invention

The present disclosure provides a data processing method, a data processing apparatus, a computer-readable storage medium, and an electronic device, thereby overcoming, at least to some extent, the problem of low processing efficiency for collected data due to the limitations of the related art.

According to a first aspect of the present disclosure, there is provided a data processing method comprising:

determining a data type of the acquired data, wherein the data type at least comprises one of a difference data type, an abnormal data type, a dynamic length field type, a fixed length field type and a length separation type;

encoding the acquired data according to the data type of the acquired data to obtain encoded data;

and sending the encoded data to a collector so that the collector analyzes the encoded data.

In an exemplary embodiment of the present disclosure, the encoding the collected data according to the data type of the collected data to obtain encoded data includes:

the coded data is composed of a data identification block and a data content block, wherein the data identification block comprises a data tag and a data type, and the data content block at least comprises one of data content, a data difference value and an abnormality reason identification.

In an exemplary embodiment of the present disclosure, the data type of the collected data is a difference data type; the step of encoding the collected data according to the data type of the collected data to obtain encoded data comprises the following steps:

Determining a data difference value of the current acquired data according to the data content of the last acquired data;

and encoding the data tag, the difference data type and the data difference value of the current acquired data to obtain the encoded data.

In an exemplary embodiment of the present disclosure, the encoding the data tag of the current acquired data, the difference data type, and the data difference value to obtain the encoded data includes:

and performing binary coding on the data tag of the current acquired data, the difference data type and the data difference value to obtain the coded data.

In an exemplary embodiment of the present disclosure, the data type of the collected data is an abnormal data type; the step of encoding the collected data according to the data type of the collected data to obtain encoded data comprises the following steps:

determining an abnormality reason identifier of the current acquired data;

and encoding the data tag of the current acquired data, the abnormal data type and the abnormal reason identifier to obtain the encoded data.

According to a second aspect of the present disclosure, there is provided a data processing method comprising:

The network equipment determines the data type of the acquired data, wherein the data type at least comprises one of a difference data type, an abnormal data type, a dynamic length field type, a fixed length field type and a length separation type;

the network equipment encodes the acquired data according to the data type of the acquired data to obtain encoded data, and sends the encoded data to a collector;

the collector receives the coded data and matches corresponding message analysis rules according to the data type of the coded data;

and the collector analyzes the coded data according to the message analysis rule to obtain the collected data.

In an exemplary embodiment of the present disclosure, before the collector receives the encoded data, the method further comprises:

presetting a message analysis rule corresponding to the data type of the acquired data.

In an exemplary embodiment of the present disclosure, the collector receives the encoded data, and matches a corresponding packet parsing rule according to a data type of the encoded data, including:

the collector receives the encoded data and extracts the data type, the data tag and the data content block of the encoded data;

When the data type of the coded data is a difference data type, the corresponding data content block is a data difference value, and a first message analysis rule is obtained according to the difference data type;

when the data type of the encoded data is an abnormal data type, the corresponding data content block is an abnormal reason identifier, and a second message analysis rule is obtained according to the abnormal data type matching.

In an exemplary embodiment of the present disclosure, the collector parses the encoded data according to the message parsing rule to obtain the collected data, including:

the collector determines the data content of the last encoded data according to the data tag of the encoded data based on the first message parsing rule;

and determining the current acquired data according to the data content of the last encoded data and the data difference value.

the collector determines the current collected data according to the abnormal reason identification of the coded data based on the second message analysis rule, wherein the current collected data is the reason that the data content corresponding to the data tag of the coded data is the abnormal data.

According to a third aspect of the present disclosure, there is provided a data processing apparatus comprising:

the data type determining module is used for determining the data type of the acquired data, wherein the data type at least comprises one of a difference data type, an abnormal data type, a dynamic length field type, a fixed length field type and a length separation type;

the data coding module is used for coding the acquired data according to the data type of the acquired data to obtain coded data;

and the data transmitting module is used for transmitting the encoded data to a collector so that the collector analyzes the encoded data.

According to a fourth aspect of the present disclosure, there is provided a data processing apparatus comprising:

the data type determining module is used for determining the data type of the acquired data by the network equipment, wherein the data type at least comprises one of a difference data type, an abnormal data type, a dynamic length field type, a fixed length field type and a length separation type;

the data transmission module is used for encoding the acquired data according to the data type of the acquired data by the network equipment to obtain encoded data and transmitting the encoded data to the acquirer;

The data receiving module is used for receiving the coded data by the collector and matching corresponding message analysis rules according to the data type of the coded data;

and the data analysis module is used for analyzing the coded data by the collector according to the message analysis rule to obtain the collected data.

According to a fifth aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any one of the above.

According to a sixth aspect of the present disclosure, there is provided an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method of any of the above via execution of the executable instructions.

Exemplary embodiments of the present disclosure may have some or all of the following advantages:

in the data processing method provided in the exemplary embodiments of the present disclosure, by determining a data type of the collected data, the data type includes at least one of a difference data type, an abnormal data type, a dynamic length field type, a fixed length field type, and a length separation type; encoding the acquired data according to the data type of the acquired data to obtain encoded data; and sending the encoded data to a collector so that the collector analyzes the encoded data. According to the method and the device, the data types of the acquired data are expanded, the difference data types and the abnormal data types are increased, and flexible processing of the acquired data is realized, so that the processing efficiency of the acquired data is improved. The collected data can be compressed by reporting the collected data in a data difference mode, so that the redundancy of the collected data is reduced, and the processing efficiency of the collected data is improved; when the data is not acquired, the acquired data is reported in an abnormal data mode, so that the reasons of the abnormal data can be timely reported to the acquirer, the acquirer can reconfigure the network equipment conveniently, real-time acquisition of the data is guaranteed, and the processing efficiency of the acquired data is further improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure. It will be apparent to those of ordinary skill in the art that the drawings in the following description are merely examples of the disclosure and that other drawings may be derived from them without undue effort.

FIG. 1 illustrates a schematic diagram of an exemplary system architecture to which the data processing methods and apparatus of embodiments of the present disclosure may be applied;

FIG. 2 schematically illustrates a flow chart of a data processing method according to one embodiment of the present disclosure;

FIG. 3 schematically illustrates a schematic diagram of GPB encoded acquisition data, according to one embodiment of the disclosure;

FIG. 4 schematically illustrates a schematic diagram of acquisition data of one particular GPB code, according to one embodiment of the disclosure;

FIG. 5 schematically illustrates a flow chart of encoding acquired data of a difference data type, according to one embodiment of the disclosure;

FIG. 6 schematically illustrates a flow chart for encoding acquisition data of an anomalous data type in accordance with an embodiment of the disclosure;

FIG. 7 schematically illustrates a schematic diagram of a set of acquired data of light module temperature according to one embodiment of the disclosure;

FIG. 8 schematically illustrates a flow chart of a data processing method according to another embodiment of the present disclosure;

FIG. 9 schematically illustrates a block diagram of a data processing apparatus according to one embodiment of the present disclosure;

FIG. 10 schematically illustrates a block diagram of a data processing apparatus according to another embodiment of the present disclosure;

fig. 11 shows a schematic diagram of a computer system suitable for use in implementing embodiments of the present disclosure.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the present disclosure. One skilled in the relevant art will recognize, however, that the aspects of the disclosure may be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.

Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor devices and/or microcontroller devices.

FIG. 1 illustrates a schematic diagram of a system architecture of an exemplary application environment in which data processing methods and apparatus of embodiments of the present disclosure may be applied.

As shown in fig. 1, the system 100 may include one or more of the terminal devices 101, 102, 103, a network 104, and a server 105. The terminal devices 101, 102, 103 may be network devices to be monitored, and the network devices may be physical devices or virtual devices, which is not limited in this disclosure. The network 104 is a medium used to provide a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc. The server 105 may be a collector for receiving and storing collected data reported by the network device. The server 105 may be a server, a server cluster formed by a plurality of servers, a virtualization platform or a cloud computing service center. It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

The data processing method provided by the exemplary embodiments of the present disclosure is generally performed by one or more of the terminal devices 101, 102, 103, and accordingly, the data processing apparatus may also be provided in the terminal devices 101, 102, 103. For example, the terminal device 101 may sample the designated collected data, encode the collected data according to the data type to obtain encoded data, and may send the encoded data to the server 105 In a manner such as gRPC (Google Remote Procedure Call ), INT (In-band Telemetry), ERSPAN (Encapsulated Remote Switch Port Analyzer, encapsulated remote port mirror), etc., so that the server 105 parses the received encoded data, to implement real-time collection of the data. Those skilled in the art will readily appreciate that the data processing methods provided by the exemplary embodiments of the present disclosure may also be performed by server 105, and accordingly, the data processing apparatus is typically disposed in server 105. For example, the server 105 receives and stores the encoded collected data reported by the network device, and performs a data processing method to obtain the collected data, and displays the collected data to a corresponding operation and maintenance person for management, for example, the collected data may be displayed to the operation and maintenance person in a form of a graphical interface, which is not particularly limited in this exemplary embodiment.

The following describes the technical scheme of the embodiments of the present disclosure in detail:

when the Telemetry technology is used to collect data, a GPB (Google Protocol Buffer, protobuf) binary coding mode is generally adopted, and the data structure of the data is described by a proto file (a structure description file). The GPB is a lightweight and efficient structured data storage format and can be used for structured data serialization. In the data acquisition process, the network equipment can automatically generate a message packaging code based on the proto file, and the collected data is serialized automatically, namely, the collected data is packaged into a binary message, and the data is continuously pushed to the collector in a streaming mode. After the collector receives the deserialized message, an analysis code can be automatically generated based on the proto file to analyze the deserialized message.

The present disclosure may be illustrated with respect to a scenario for acquisition of access network data. A large amount of unchanged or less changed acquired data exists in the actual acquired data, and integer coding with determined precision is generally adopted when the access network equipment reports the data, so that the coded data value is larger, and a plurality of bytes are occupied. In addition, the OLT (Optical Line Terminal ) belongs to a service node side device of the access network, and generally adopts a UDP (User Datagram Protocol ) protocol to collect data, and when the data is missing, it cannot be determined whether the cause of the abnormality is packet loss of the UDP protocol or the device cannot report.

Based on one or more of the above problems, the present exemplary embodiment provides a data processing method, which may be applied to one or more of the above terminal devices 101, 102, 103, and may also be applied to the above server 105, which is not particularly limited. Referring to fig. 2, the data processing method may include the following steps S210 to S230:

s210, determining a data type of acquired data, wherein the data type at least comprises one of a difference data type, an abnormal data type, a dynamic length field type, a fixed length field type and a length separation type;

s220, coding the acquired data according to the data type of the acquired data to obtain coded data;

and S230, sending the encoded data to a collector so that the collector analyzes the encoded data.

Next, the above steps of the present exemplary embodiment will be described in more detail.

In step S210, a data type of the collected data is determined, the data type including at least one of a difference data type, an abnormal data type, a dynamic length field type, a fixed length field type, and a length separation type.

In an example embodiment of the present disclosure, the collected data may be data such as optical link information, interface traffic statistics of the access network device, and the like. Currently, 6 data types have been defined when GPB encoding the acquired data, as shown in table 1.

TABLE 1

As can be seen from table 1, protobuf classifies different data types and selects different storage formats. The classification mark corresponding to the 6 data types is 0-5, wherein the types 3 and 4 are abandoned, and the other data types are respectively a dynamic length field type, a fixed length field type and a length separation type. For the dynamic length field type, namely the varint data type, the corresponding classification is marked as '0', and the data types belonging to the dynamic length field type comprise int32, int64, uint32, uint64, sint32, sint64, bool, enumeration and the like. The fixed length field type comprises two data types of 64 bits and 32 bits, and the corresponding classification identifiers are respectively 1 and 5; the class identifier corresponding to the length separation type is "2", and the data types belonging to the length separation type include character strings, bytes, embedded messages, packed repeated fields, and the like. Type 0, type 1, and type 5 may be stored in the form of key-value, and type 2 may be stored in the form of key-length-value due to the variable data length.

It should be noted that, when using GPB binary coding, 3 bits may represent at most 8 data types, i.e., 0 to 7, and only 6 data types are currently defined. Based on this, in configuring the data types of the collected data in the exemplary embodiments of the present disclosure, two data types, that is, a difference data type and an abnormal data type, may be added, as shown in table 2. As can be seen from Table 2, the classification identifier "6" is added to identify the difference between the current acquired data content and the last acquired data content, i.e., the data difference, of the same data object. For example, the last acquired data content may be the data content of the acquired data at the previous moment, and the corresponding data difference value is the difference value between the data content of the acquired data at the current moment and the data content of the acquired data at the previous moment. Taking collected data of the temperature of the optical module as an example, when the temperature of the optical module at the previous moment is 15 ℃ and the temperature of the optical module at the current moment is 12 ℃, the corresponding data difference value is-3 ℃; when the temperature of the optical module at the previous moment is 15 ℃ and the temperature of the optical module at the current moment is still 15 ℃, the corresponding data difference value is 0 ℃; when the temperature of the optical module at the previous moment is 15 ℃ and the temperature of the optical module at the current moment is still 17 ℃, the corresponding data difference value is 2 ℃. A classification identifier "7" is added to identify data values that cannot be reported due to device reasons, i.e., an anomaly reason identifier. It can be understood that the device reasons for which the data cannot be reported can be preconfigured, and each device reason for which the data cannot be reported is encoded, so as to obtain the corresponding abnormal reason identifier. For example, when the data cannot be reported due to a register abnormality, the corresponding abnormality cause flag is "1", and when the data cannot be reported due to a CPU (Central Processing Unit ) abnormality, the corresponding abnormality cause flag is "2", and the device cause for which the data cannot be reported and the corresponding abnormality cause flag may be stored in the form of a key value pair. In other examples, after the device reason for failing to report data is preconfigured, the character string may also be used to identify each device reason for failing to report data, which is not limited in this disclosure.

TABLE 2

Classification identification	Data type classification	Marking conditions
			6	Difference data type	Data encoding using data differences
7	Abnormal data type	Data values that cannot be reported due to device reasons

For example, a data structure of the acquisition data may be described using a proto file in which an acquisition data message may be composed of at least one acquisition data, each data object in the acquisition data being configured with a data type and a data tag. For example, for the collection data of the optical module temperature, the data tag is used to identify that the temperature characteristic of the optical module is collected, and the data tag of the optical module temperature may be configured to be "2". Similarly, the data tags of the voltage, current, emitted light power, etc. of the light module may be sequentially configured as "3", "4", "5", which is not particularly limited by the present disclosure. Correspondingly, taking an optical module temperature proto file as an example, when the optical module temperature is 15 ℃, the proto file comprises a data type and a data tag, wherein the data type is int32 type, and the data tag is 2. Thus, the data type of the acquired data may be determined from the proto file of the access network device, wherein the data type may comprise at least one of a difference data type, an abnormal data type, a dynamic length field type, a fixed length field type and a length separation type.

The data type of the acquired data is expanded, so that the difference data type and the abnormal data type are increased. For a large amount of unchanged or less changed collected data, the collected data is reported in a data difference mode, so that the collected data can be compressed, the redundancy of the collected data is reduced, and the processing efficiency of the collected data is improved; when the data is not acquired, the acquired data is reported in an abnormal data mode, so that the reasons of the abnormal data can be timely reported to the acquirer, the acquirer can reconfigure the network equipment conveniently, real-time acquisition of the data is guaranteed, and the processing efficiency of the acquired data is further improved.

In step S220, the collected data is encoded according to the data type of the collected data, so as to obtain encoded data.

The collected data refers to the data such as optical link information, interface flow statistics and the like of the access network equipment which are not coded. For example, the acquired data may be binary coded, such as GPB coded, according to a data type of the acquired data, to obtain coded data. The binary code may be a code such as thread (an interface description language and binary communication protocol), SBE (Simple Binary Encoding, simple binary code), etc. The encoded data may be composed of a data identification block including a data tag and a data type, and a data content block including at least one of a data content, a data difference value, and an abnormality cause identification. In other examples, the collected data may be XML (Extensible Markup Language ) encoded, JSON (Java Script Object Notation, java Script object notation) encoded, etc. according to the data type of the collected data, which is not specifically limited in this disclosure.

Taking GPB encoding of the collected data as an example, the encoded data obtained by using GPB encoding is a binary message in the form of byte stream. Referring to fig. 3, a schematic representation of acquisition data in the proto format is shown, which may consist of two parts, a data identification block (1) and a data content block (2). The data identification block (1) and the data content block (2) may be represented by one or more consecutive bytes, 8 bits each, and the most significant bit (6) (Most Significant Bit, MSB) of each byte may be used to identify whether the data block is finished, e.g. if the most significant bit of a certain byte is 1, it indicates that the data block is not finished, i.e. the byte is further followed by other bytes, and if the most significant bit is 0, it indicates that the data block is finished, i.e. the byte is the last byte of the data block. In the data identification block (1), the last three bits (4) of the last byte are used to identify the data type, the other bits (3) than the MSB (6) are used to identify the data tag, the specific number of bits depends on the binary representation of the data tag, which is not specifically limited by the present disclosure. In the data content block (2), the remaining lower 7 bits (5) excluding the MSB (6) each represent data content.

For example, for a proto file of the optical module temperature, for example, when the optical module temperature is 15 ℃, the corresponding data type is the int32 type, which belongs to the data type 0, the data tag of the optical module temperature is "2", the binary representation only needs 2 bits, and the data tag and the data type (binary representation is 3 bits) form a data block which needs only 1 byte. The data content is 150 (unit 0.1 ℃), the binary representation requires 8 bits, and since there is one MSB per byte, a block of data content requires two bytes.

Referring to fig. 4, the GPB encoding result of the optical module with 15 ℃ of the acquired data is correspondingly given. The first byte from the left is the data identification block, the first bit (6) in the byte is the MSB, and the MSB is 0, indicating that the data identification block is finished, it can be seen that only one byte is included in the data identification block, the last three bits (4) in the byte are data types, the data types are "0", the binary system is represented as 000, the middle four bits (3) in the byte are data labels, the data labels are "2", the binary system is represented as 0010, and the binary system of the whole data identification block is represented as 00010000. The MSB of the second byte from the left is 1, which indicates that the data content block is not finished, the MSB of the third byte is 0, which indicates that the data content block is finished, and it can be seen that the second byte and the third byte constitute the data content block, the other bits (5) except the MSBs of the two bytes indicate the data content, the data content is "150", the binary representation is 10010110, and since one MSB is provided for each byte, the binary representation of "150" requires two bytes, which is 10000001_00010110. When a plurality of bytes are included in one data block, the inversion encoding is required, but the MSB of each byte is required to remain unchanged at the time of the inversion encoding, and thus, the binary representation of "150" becomes 10010110—00000001. As can be seen from fig. 4, "00010000_10010110_00000001" is the data to be collected "light module temperature: 15 ℃ and the coded data obtained after GPB coding.

In an example embodiment, when the data type of the acquired data is a difference data type, referring to fig. 5, the acquired data may be encoded according to steps S510 and S520.

In step S510, a data difference value of the current collected data is determined according to the data content of the last collected data.

For a large amount of unchanged or less changed acquired data, the acquired data can be reported in a data difference mode so as to compress the acquired data.

For example, for the collected data of the optical module temperature, the data tag of the optical module temperature is "2", and the data content is a temperature value. The data content of the last acquired data, namely the temperature value of the optical module at the last moment, can be queried according to the data tag '2' of the optical module temperature so as to determine the data type of the optical module temperature at the current moment. For example, if the temperature value of the optical module at the previous time is 15 ℃, and the temperature value of the optical module at the current time is 17 ℃, the data type of the optical module temperature at the current time may be configured as data type 0, and GPB encoding as shown in fig. 4 is performed, or the data type of the optical module temperature at the current time may be configured as data type 6, and the data difference of the current collected data, that is, the data difference at the current time, may be determined according to the temperature value of the optical module at the previous time, that is, the data difference is 2 ℃.

In step S520, the data tag of the current collected data, the difference data type and the data difference value are encoded, so as to obtain the encoded data.

Taking the collected data of the temperature of the optical module as an example, after determining the data difference value of the current moment, the data tag "2", the difference data type identifier "6" of the temperature of the optical module and the data difference value of the current moment, such as 2 ℃, can be binary coded to obtain coded data, so as to send the coded data to the collector for analysis. The binary code may be GPB, thrif, SBE or the like. In other examples, the data tag "2" of the temperature of the optical module, the difference data type identifier "6" and the data difference value of the current moment, such as 2 ℃, may be subjected to XML encoding, JSON encoding, and the like, which is not limited in this disclosure.

Specifically, when the acquired data is GPB encoded, the binary representation of the data tag "2" is 0010, the binary representation of the difference data type identifier "6" is 110, and the binary representation of the data difference 2 ℃ is 0000010. As can be seen, the collected data is "light module temperature: 17 deg.c ", encoded data of" 00010110_00000010 "is obtained when encoded in the difference data type. And when encoded in the dynamic length field type (int 32 type), the resulting encoded data is "00010000_10101010_00000001".

In this example, the data type is extended without adding bytes, and the acquired data encoded in the difference data type occupies fewer bytes. Therefore, for a large amount of unchanged or less changed collected data, the collected data is reported in a data difference mode, the collected data can be compressed, the redundancy of the collected data is reduced, and the processing efficiency of the collected data is improved.

In another example embodiment, when the data type of the acquired data is an abnormal data type, referring to fig. 6, the acquired data may be encoded according to step S610 and step S620.

In step S610, an abnormality cause identification of the current acquired data is determined.

For the situation that the data cannot be reported, the collected data can be reported in an abnormal data mode, so that the reason why the collected data cannot be reported is explained.

When the data cannot be reported, the current data type of the temperature of the optical module can be configured as the data type 7, and the data content of the acquired data is determined to be the abnormal reason identifier of the abnormal data according to the preconfigured equipment reason for the failure to report the data and the corresponding abnormal reason identifier. For example, when the equipment reason cannot report the temperature value of the optical module at the current moment, determining the abnormal reason identifier corresponding to the equipment reason by inquiring the key value pair corresponding to the equipment reason incapable of reporting the data. For example, when data cannot be reported due to a register abnormality, the abnormality cause flag can be determined to be "1" by the inquiry key pair, and when data cannot be reported due to a CPU abnormality, the abnormality cause flag can be determined to be "2" by the inquiry key pair.

In step S620, the data tag of the current collected data, the abnormal data type and the abnormal cause identifier are encoded, so as to obtain the encoded data.

Taking the collected data of the optical module temperature as an example, after determining the abnormal reason identifier of the collected data of the optical module temperature, the data tag '2', the abnormal data type identifier '7' and the corresponding abnormal reason identifier '1' of the optical module temperature can be binary coded to obtain coded data, and the coded data is sent to the collector for analysis. The binary code may be GPB, thrif, SBE or the like. In other examples, the data tag "2" of the optical module temperature, the abnormal data type identifier "7" and the corresponding abnormal cause identifier "1" may be subjected to XML encoding, JSON encoding, and the like, which is not limited in this disclosure.

Specifically, when the acquired data is GPB encoded, the binary representation of the data tag "2" is 0010, the binary representation of the abnormal data type identifier "7" is 111, and the binary representation of the abnormal cause identifier "1" is 0000001. It can be seen that when data cannot be reported, i.e. data is not collected, and the data is encoded in an abnormal data type, the obtained encoded data is "00010111_00000001". Before the abnormal data type is configured, the data cannot be reported, which means that the data cannot be collected and the coding cannot be performed.

In the example, the data type is expanded under the condition that bytes are not added, and after the acquired data is encoded by the abnormal data type, the reasons of the abnormal data can be timely reported to the collector, so that the collector can reconfigure the network equipment, the real-time acquisition of the data is ensured, and the processing efficiency of the acquired data is further improved.

Referring to fig. 7, a set of acquired data for the temperature of the optical module is schematically shown. Fig. 7 includes three continuous time acquisition data of the optical module 10, each time acquisition data includes two parts of acquisition data, the acquisition data 1 is used for identifying optical module information, including an optical module name (such as optical module 10), a data tag and a data type of the optical module, and the acquisition data 2 is used for identifying optical module temperature information, including optical module temperature data, a data tag and a data type of the optical module temperature.

For the collected data at time 1, the collected data 1 is "00001010—00001010", where the data tag (3) of the optical module is "1", the binary representation is 0001, the data type (4) of the optical module is "2", the binary representation is 010, the data content (5) is the binary representation of the optical module name, that is, the binary representation 0001010 of the optical module "10", and (6) is the byte MSB. The acquired data 2 is '00010000_10101010_00000001', wherein the data tag (3) of the optical module temperature is '2', the binary representation is 0010, the data type (4) of the optical module is '0', the binary representation is 000, and the data content (5) is the binary representation of the optical module temperature, namely, the binary representation of the optical module temperature of 15 ℃ is 10010110_00000001.

For the collected data at time 2, the collected data 1 is "00001010_00001010", the collected data 2 is "00010110_00000000", wherein the data tag (3) of the optical module temperature is "2", the binary representation is 0010, the data type (4) of the optical module is "6", the binary representation is 110, the data content (5) is a binary representation of the difference between the optical module temperature at time 2 and the optical module temperature at time 1, the temperature difference is 0000000, and the optical module temperature at time 2 is 15 ℃.

For the collected data at time 3, the collected data 1 is "00001010_00001010", the collected data 2 is "00010111_00000001", wherein the data tag (3) of the optical module temperature is "2", the binary representation is 0010, the data type (4) of the optical module is "7", the binary representation is 111, the data content (5) is the binary representation of the abnormality cause identifier, 0000001 represents the abnormality cause identifier is "1", and the reasons that the device cannot upload data can be obtained according to the abnormality cause identifier can be matched.

In this example, the difference data type and the abnormal data type are added by expanding the data type of the acquired data. For a large amount of unchanged or less changed collected data, the collected data is reported in a data difference mode, so that the collected data can be compressed, the redundancy of the collected data is reduced, and the processing efficiency of the collected data is improved; when the data is not acquired, the acquired data is reported in an abnormal data mode, so that the reasons of the abnormal data can be timely reported to the acquirer, the acquirer can reconfigure the network equipment conveniently, real-time acquisition of the data is guaranteed, and the processing efficiency of the acquired data is further improved.

In step S230, the encoded data is sent to a collector, so that the collector parses the encoded data.

The encoded data obtained by GPB encoding can be sent to the collector, and after the collector receives the encoded data, the analysis code can be automatically generated based on the proto file for analysis. For example, when the encoded data received by the collector is "00010000_10101010_00000001", the encoded data may be obtained by parsing to indicate "the optical module temperature: 15 ℃ ". By actively pushing the collected data to the collector, a more real-time, higher and more accurate network monitoring function can be provided.

On the other hand, the present exemplary embodiment also provides a data processing method, which may be applied to one or more of the above-mentioned terminal devices 101, 102, 103, and may also be applied to the above-mentioned server 105, which is not particularly limited in this exemplary embodiment. Referring to fig. 8, the data processing method may include the following steps S810 to S840:

in step S810, the network device determines a data type of the collected data, the data type including at least one of a difference data type, an abnormal data type, a dynamic length field type, a fixed length field type, and a length separation type.

The network device may be an access network device, and when the access network device actively pushes the collected data to the collector, the data type of the collected data may be determined, where the data type may include at least one of a difference data type, an abnormal data type, a dynamic length field type, a fixed length field type, and a length separation type. The present disclosure mainly describes a difference data type and an abnormal data type, and specifically, a classification identifier "6" identifies a difference between a currently acquired data content and a last acquired data content of the same data object, that is, a data difference. The classification identifier "7" identifies a data value that cannot be reported due to the device cause, i.e., an abnormality cause identifier.

In step S820, the network device encodes the collected data according to the data type of the collected data to obtain encoded data, and sends the encoded data to the collector.

For example, binary coding, such as GPB, thrif, SBE, may be performed on the collected data according to the data type of the collected data, so as to obtain coded data, and the coded data is sent to the collector, where after the collector receives the coded data, an analysis code may be automatically generated based on the proto file for analysis. The process of encoding the collected data according to the data type of the collected data is described in detail in step S220 of the data processing method, and will not be described here again.

In step S830, the collector receives the encoded data, and matches a corresponding packet parsing rule according to a data type of the encoded data.

Before the collector receives the encoded data, a message parsing rule corresponding to the data type of the collected data may be preset. For example, when the data type of the collected data is a difference data type, a first message parsing rule may be configured to parse the encoded data according to the first message parsing rule to obtain a data difference. When the data type of the acquired data is an abnormal data type, a second message analysis rule can be configured to analyze the encoded data according to the second message analysis rule, so as to obtain the reason that the equipment cannot report the data.

After the collector receives the encoded data, the data type, data tag and data content block of the encoded data may be extracted. When the data type of the encoded data is a difference data type, the corresponding data content block is a data difference, and the first message analysis rule can be obtained according to the difference data type matching. When the data type of the encoded data is an abnormal data type, the corresponding data content block is an abnormal reason identifier, and the second message analysis rule can be obtained according to the abnormal data type matching. In this example, the difference data type and the anomaly data type are added. The collected data is reported in a data difference mode, the collected data can be compressed, and corresponding analysis rules are given. The acquired data is reported in an abnormal data mode, so that the reasons of the abnormal data can be timely reported to the acquirer, corresponding analysis rules are provided, the explanation of the reasons that the data cannot be reported is realized, and the processing efficiency of the acquired data is further improved.

In step S840, the collector parses the encoded data according to the message parsing rule to obtain the sampled data.

In an example embodiment, when the data type of the encoded data is a difference data type, the corresponding data content block is a data difference, and the first message parsing rule may be obtained by matching according to the difference data type. Further, the collector may determine the data content of the last encoded data according to the data tag of the encoded data based on the first message parsing rule, for example, may determine the data content of the encoded data at the last time according to the data tag of the encoded data, and determine the current collected data according to the data content and the data difference value of the encoded data at the last time.

Illustratively, identifying the data type of the encoded data as "6" indicates that the encoded data is of a difference data type. Firstly, the data content corresponding to the data label at the last time t-1 can be found according to the data label (such as the identification optical module temperature) in the encoded data, and is' X _t-1 ", the data difference value extracted from the encoded data at the current time t is Δx _t Therefore, the data content at the current time t can be assigned as "X _t-1 +ΔX _t The value is the current acquired data. For example, the data difference value extracted from the collected data of the optical module temperature at the current time t is 2 ℃, the data label of the optical module temperature is 2, the optical module temperature at the previous time t-1 can be inquired and obtained according to the data label to be 15 ℃, and then the dimension of the optical module at the current time is 17 ℃.

In another example embodiment, when the data type of the encoded data is an abnormal data type, the corresponding data content block is an abnormal reason identifier, and the second message parsing rule may be obtained by matching according to the abnormal data type. Further, the collector may determine, based on the second message parsing rule, the current collected data according to the anomaly cause identifier of the encoded data, where the current collected data is the cause of the anomaly data, and the data content corresponding to the data tag of the encoded data is the cause of the anomaly data.

Illustratively, identifying the data type of the encoded data as "7" indicates that the encoded data is an anomalous data type. The reason why the device cannot report the data can be determined according to the abnormal reason identification in the encoded data. For example, when the abnormality cause identifier is "1", it may be determined that the register abnormality cannot report data by querying a key value pair corresponding to the abnormality cause identifier, based on which timely maintenance of the register may be facilitated.

It should be noted that although the steps of the methods in the present disclosure are depicted in the accompanying drawings in a particular order, this does not require or imply that the steps must be performed in that particular order, or that all illustrated steps be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform, etc.

Further, in this example embodiment, a data processing apparatus is also provided. The device can be applied to a server or terminal equipment. Referring to fig. 9, the data processing apparatus 900 may include a data type determining module 910, a data encoding module 920, and a data transmitting module 930, wherein:

a data type determining module 910, configured to determine a data type of the collected data, where the data type includes at least one of a difference data type, an abnormal data type, a dynamic length field type, a fixed length field type, and a length separation type;

the data encoding module 920 is configured to encode the collected data according to a data type of the collected data to obtain encoded data;

And the data sending module 930 is configured to send the encoded data to a collector, so that the collector parses the encoded data.

In an alternative embodiment, data encoding module 920 is configured to encode the collected data according to a data type of the collected data, resulting in the encoded data;

In an alternative embodiment, the data type of the collected data is a difference data type; the data encoding module 920 includes:

the difference value determining unit is used for determining the data difference value of the current acquired data according to the data content of the last acquired data;

the first data coding unit is used for coding the data tag, the difference data type and the data difference value of the current acquired data to obtain the coded data.

In an alternative embodiment, the first data encoding unit comprises:

and the first data coding subunit is used for binary coding the data tag, the difference data type and the data difference value of the current acquired data to obtain the coded data.

In an alternative embodiment, the data type of the collected data is an abnormal data type; the data encoding module 920 includes:

the reason determining unit is used for determining an abnormal reason identifier of the current acquired data;

and the second data encoding unit is used for encoding the data tag of the current acquired data, the abnormal data type and the abnormal reason identifier to obtain the encoded data.

The specific details of each module in the above data processing apparatus have been described in detail in the corresponding data processing method, so that the details are not repeated here.

In this example embodiment, a data processing apparatus is also provided. The device can be applied to a server or terminal equipment. Referring to fig. 10, the data processing apparatus 1000 may include a data type determining module 1010, a data transmitting module 1020, a data receiving module 1030, and a data parsing module 1040, wherein:

a data type determining module 1010, configured to determine, by the network device, a data type of the collected data, where the data type includes at least one of a difference data type, an abnormal data type, a dynamic length field type, a fixed length field type, and a length separation type;

The data sending module 1020 is configured to encode the collected data according to a data type of the collected data by the network device, obtain encoded data, and send the encoded data to a collector;

the data receiving module 1030 is configured to receive the encoded data and match a corresponding message parsing rule according to a data type of the encoded data;

and the data analysis module 1040 is configured to analyze the encoded data according to the message analysis rule by using the collector, so as to obtain the collected data.

In an alternative embodiment, the data processing apparatus 1000 further includes:

and the rule presetting module is used for presetting a message analysis rule corresponding to the data type of the acquired data.

In an alternative embodiment, data receiving module 1030 is configured to receive the encoded data and extract a data type, a data tag, and a data content block of the encoded data; when the data type of the coded data is a difference data type, the corresponding data content block is a data difference value, and a first message analysis rule is obtained according to the difference data type; when the data type of the encoded data is an abnormal data type, the corresponding data content block is an abnormal reason identifier, and a second message analysis rule is obtained according to the abnormal data type matching.

In an alternative embodiment, the data parsing module 1030 includes:

the first data content determining unit is used for determining the data content of the last encoded data according to the data tag of the encoded data based on the first message parsing rule by the collector;

and the first acquired data determining unit is used for determining the current acquired data according to the data content of the last encoded data and the data difference value.

In an optional implementation manner, the data parsing module 1030 is configured to determine, based on the second message parsing rule, the current collected data according to the anomaly cause identifier of the encoded data, where the current collected data is a cause of the anomaly data corresponding to the data tag of the encoded data.

Exemplary embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon a program product capable of implementing the method described above in the present specification. In some possible implementations, aspects of the present disclosure may also be implemented in the form of a program product comprising program code for causing an electronic device to carry out the steps according to the various exemplary embodiments of the disclosure as described in the "exemplary methods" section of this specification, when the program product is run on an electronic device. The program product may employ a portable compact disc read-only memory (CD-ROM) and comprise program code and may be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The computer readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

The exemplary embodiment of the disclosure also provides an electronic device capable of implementing the method. An electronic device 1100 according to such an exemplary embodiment of the present disclosure is described below with reference to fig. 11. The electronic device 1100 shown in fig. 11 is merely an example and should not be construed as limiting the functionality and scope of use of the disclosed embodiments.

As shown in fig. 11, the electronic device 1100 may be embodied in the form of a general purpose computing device. Components of electronic device 1100 may include, but are not limited to: at least one processing unit 1110, at least one memory unit 1120, a bus 1130 connecting the different system components (including the memory unit 1120 and the processing unit 1110), and a display unit 1140.

The storage unit 1120 stores program codes that can be executed by the processing unit 1110, so that the processing unit 1110 performs the steps according to various exemplary embodiments of the present disclosure described in the above "exemplary method" section of the present specification. For example, processing unit 1110 may perform any one or more of the method steps of fig. 2, 5, 6, and 8.

The storage unit 1120 may include a readable medium in the form of a volatile storage unit, such as a Random Access Memory (RAM) 1121 and/or a cache memory 1122, and may further include a Read Only Memory (ROM) 1123.

Storage unit 1120 may also include a program/utility 1124 having a set (at least one) of program modules 1125, such program modules 1125 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.

The bus 1130 may be a local bus representing one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a bus using any of a variety of bus architectures.

The electronic device 1100 may also communicate with one or more external devices 1200 (e.g., keyboard, pointing device, bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 1100, and/or any devices (e.g., routers, modems, etc.) that enable the electronic device 1100 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 1150. Also, electronic device 1100 can communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 1160. As shown, network adapter 1160 communicates with other modules of electronic device 1100 via bus 1130. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 1100, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.

From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, including several instructions to cause a computing device (may be a personal computer, a server, a terminal device, or a network device, etc.) to perform the method according to the exemplary embodiments of the present disclosure.

Furthermore, the above-described figures are only schematic illustrations of processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily appreciated that the processes shown in the above figures do not indicate or limit the temporal order of these processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, for example, among a plurality of modules.

It should be noted that although in the above detailed description several modules or units of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit in accordance with embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.

It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method of data processing, comprising:

2. The data processing method according to claim 1, wherein the encoding the collected data according to the data type of the collected data to obtain encoded data includes:

3. The data processing method according to claim 2, wherein the data type of the collected data is a difference data type; the step of encoding the collected data according to the data type of the collected data to obtain encoded data comprises the following steps:

4. A data processing method according to claim 3, wherein said encoding the data tag of the current acquired data, the difference data type and the data difference value to obtain the encoded data comprises:

5. The data processing method according to claim 2, wherein the data type of the collected data is an abnormal data type; the step of encoding the collected data according to the data type of the collected data to obtain encoded data comprises the following steps:

Determining an abnormality reason identifier of the current acquired data;

6. A method of data processing, the method comprising:

7. The data processing method of claim 6, wherein before the collector receives the encoded data, the method further comprises:

8. The data processing method according to claim 6, wherein the collector receives the encoded data and matches a corresponding message parsing rule according to a data type of the encoded data, comprising:

9. The data processing method according to claim 8, wherein the collector analyzes the encoded data according to the message analysis rule to obtain the collected data, and the method comprises:

10. The data processing method according to claim 8, wherein the collector analyzes the encoded data according to the message analysis rule to obtain the collected data, and the method comprises:

11. A data processing apparatus, comprising:

12. A data processing apparatus, comprising:

13. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the method of any of claims 1-10.

14. An electronic device, comprising:

a processor; and

a memory for storing executable instructions of the processor;

wherein the processor is configured to perform the method of any of claims 1-10 via execution of the executable instructions.