CN116232710B - Log message sending and transmitting method and system for network flow acquisition equipment - Google Patents

Log message sending and transmitting method and system for network flow acquisition equipment Download PDF

Info

Publication number
CN116232710B
CN116232710B CN202310128799.1A CN202310128799A CN116232710B CN 116232710 B CN116232710 B CN 116232710B CN 202310128799 A CN202310128799 A CN 202310128799A CN 116232710 B CN116232710 B CN 116232710B
Authority
CN
China
Prior art keywords
message
mtp
stlv
data
log
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202310128799.1A
Other languages
Chinese (zh)
Other versions
CN116232710A (en
Inventor
顾欢欢
韩志前
姬汉贵
严伟
沈彬
陈国贺
刘杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NANJING SINOVATIO TECHNOLOGY CO LTD
Original Assignee
NANJING SINOVATIO TECHNOLOGY CO LTD
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NANJING SINOVATIO TECHNOLOGY CO LTD filed Critical NANJING SINOVATIO TECHNOLOGY CO LTD
Priority to CN202310128799.1A priority Critical patent/CN116232710B/en
Publication of CN116232710A publication Critical patent/CN116232710A/en
Application granted granted Critical
Publication of CN116232710B publication Critical patent/CN116232710B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00Network architectures or network communication protocols for network security
    • H04L63/14Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1408Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
    • H04L63/1425Traffic logging, e.g. anomaly detection
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D30/00Reducing energy consumption in communication networks
    • Y02D30/50Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate

Abstract

The invention discloses a method and a system for sending and transmitting log information of network flow acquisition equipment. The method for sending the log message of the network flow acquisition equipment comprises the following steps: acquiring log information of network traffic, and packaging the log information into an STLV structure message packet; and if the length of the message packet is smaller than the maximum data packet size, packaging the message packet into an MTP message block, and transmitting the MTP message block after adding the universal message header. The method supports the aggregation and transmission of the log messages, and improves the transmission efficiency of the log messages of the transmitting end.

Description

Log message sending and transmitting method and system for network flow acquisition equipment
Technical Field
The invention belongs to the technical field of network flow analysis, and particularly relates to a method and a system for sending and transmitting log messages of network flow acquisition equipment.
Background
With the advancement of digital transformation process, IT infrastructure, mobile internet and other technologies develop and change, and the network scale becomes more and more complex. The network traffic is deeply analyzed and is valued by all parties. For example, a network operator needs to deploy a flow acquisition device at a key node of the network to detect and analyze network flow, support the operator to develop service differentiation, charging diversity and marketing refinement work, and provide value-added service for clients; the IT department of the enterprise needs to deploy flow acquisition equipment at key nodes of the network, acquire key performance indexes of the network and the service, and provide high-efficiency support for network operation and maintenance; under the big background of network attack organization, network security faces more serious challenges, and related regulatory authorities need to deploy flow acquisition devices at network key nodes for protecting network space security and protecting people from illegal attack.
Under the current technical architecture, network traffic collection equipment (such as DPI equipment) is deployed at a network key node, traffic on a link is collected and identified, and log information (including metadata such as statistical data, traffic characteristics and key content fields) is sent to service analysis equipment, but the following problems exist: in the technical field of network traffic analysis, the sending of log messages lacks a unified standard, and the incompatibility of a message interface between acquisition equipment and service analysis equipment often occurs.
Disclosure of Invention
The invention aims to: aiming at the problems in the prior art, the invention provides a method for sending the log message of the network flow acquisition equipment, which supports the aggregation and the sending of the log message and improves the sending efficiency of the log message of a sending end.
The technical scheme is as follows: the invention discloses a method for sending log information of network flow acquisition equipment, which comprises the following steps:
acquiring log information of network traffic, and packaging the log information into an STLV structure message packet;
and if the length of the message packet is smaller than the maximum data packet size, packaging the message packet into an MTP message block, and transmitting the MTP message block after adding the universal message header.
Further, the STLV structure includes one STLV structure header 100 and at least one STLV structure data section 200;
the STLV structure header 100 includes a log type 101, a log version number 102;
the STLV structure data section 200 includes a structure identification code 201, a structure data end identification code 202, a structure type identification code 203, a structure data length 204, and structure data 205.
Further, the MTP message block includes an MTP message header 300 and MTP message data 400;
the MTP message header comprises an MTP message identification code (301), an MTP message block total length (302), an MTP message block serial number (303) and an MTP message block end identification code (304);
the MTP message data (400) is an STLV structured message packet.
Further, the maximum packet size is the difference between the value of the maximum transmission unit and the length of the MTP message header (300).
Further, the generic message header includes an MTP version number.
Further, the method further comprises the following steps:
if the length of the message packet is greater than the maximum data packet size, slicing the message packet, wherein the length of each sliced message sub-packet is smaller than the maximum data packet size;
the segmented message sub-packets are packaged into MTP message blocks, the MTP message identification codes (301) in each MTP message block are the same, the MTP message block serial numbers (303) are increased progressively according to the sequence of the message sub-packets in the message packet before segmentation, the MTP message block end identification codes (304) in the MTP message block corresponding to the last message sub-packet are 'end', and the MTP message block end identification codes (304) in the MTP message blocks corresponding to the rest message sub-packets are 'not end'.
On the other hand, the invention also discloses a system for realizing the method for sending the log message of the network flow acquisition equipment, which comprises the following steps:
the network flow log information acquisition module is used for acquiring log information of the network flow;
the STLV structure encapsulation module is used for encapsulating the acquired network traffic log information into an STLV structure message packet;
the MTP message encapsulation module is used for encapsulating the message packet formed by the STLV structure into an MTP message block;
and the message sending module is used for adding the general message header to the MTP message block and then sending the MTP message block.
On the other hand, the invention also discloses a method for transmitting the log message of the network flow acquisition equipment, wherein the log message is transmitted by adopting the method for transmitting the log message of the network flow acquisition equipment, and the method further comprises the following steps:
analyzing MTP message blocks from received data;
analyzing a message packet consisting of an STLV structure from the analyzed MTP message block;
and analyzing the network traffic log information from the message packet.
On the other hand, the invention also discloses a system for realizing the method for transmitting the log message of the network flow acquisition equipment, which comprises a sending system and a receiving system, wherein the sending system is the system for sending the log message of the network flow acquisition equipment, and the receiving system comprises:
the message receiving module is used for analyzing MTP message blocks from the received data;
the MPT message block analysis module is used for analyzing a message packet consisting of an STLV structure from the MTP message block;
and the STLV structure analysis module is used for analyzing the network flow log information from the message packet.
The beneficial effects are that: the method and the system for sending and transmitting the log message of the network flow acquisition equipment disclosed by the invention have the following beneficial effects:
1. the method for sending the log message of the network traffic acquisition equipment disclosed by the invention supports the aggregation and the sending of the log message, and improves the sending efficiency of the log message of the sending end; simultaneously supporting log message slicing, and sending the current acquired log message slicing to a receiving end when the resource of the sending end is insufficient;
2. supporting a log message to bear a multi-structure, and describing network traffic log messages from a multi-dimensional view (such as stream statistics information, network access behavior information and the like); support the fragmentation of the very long log message, for example, can extract and send in time to the very long log message sender, reorganize it by the receiver, lighten the buffer pressure of sender;
3. adopting a STLV (Strcut TLV) structure to package log information, and supporting the user to custom expand the structure and the structural members; has important significance for promoting the technical development of the network traffic analysis field.
Drawings
FIG. 1 is a flow chart of a method for sending log messages of a network traffic acquisition device disclosed by the invention;
fig. 2 is a schematic composition diagram of the STLV structure;
FIG. 3 is a schematic diagram of the composition of an MTP message block;
FIG. 4 is a schematic diagram of a transmit data format;
FIG. 5 is a schematic diagram of a data format transmitted after fragmentation;
fig. 6 is a schematic diagram of a log message sending system of a network traffic collection device according to the present invention;
FIG. 7 is a flow chart of a method for transmitting log messages of a network traffic acquisition device according to the present invention;
fig. 8 is a schematic diagram of a log message transmission system of a network traffic collection device according to the present invention.
Detailed Description
The invention is further elucidated below in connection with the drawings and the detailed description.
Example 1
The embodiment discloses a method for sending log information of network traffic acquisition equipment, as shown in fig. 1, comprising the following steps:
step 1, acquiring log information of network traffic, and packaging the log information into an STLV structure message packet;
in the present invention, the STLV structure includes one STLV structure header 100 and at least one STLV structure data section 200, as shown in fig. 2. The STLV structure header 100 includes a log type 101, a log version number 102; in this embodiment, the length of the log type 101 and the log version number 102 is 8bits, and in addition, in order to facilitate the late expansion, the present embodiment adds a 16bits reserved bit 103 to the STLV structure header 100. The STLV structure data section 200 includes a structure identification code 201, a structure data end identification code 202, a structure type identification code 203, a structure data length 204, and structure data 205. In this embodiment, the structure identification code 201 is 16bits, the structure data end identification code 202 is 1bit, the structure type identification code 203 is 15bits, the structure data length 204 is 32bits, and the structure data 205 is the log information of the user program.
Assuming that the network traffic of a certain user is to be counted, the acquired user log information comprises traffic statistics of video, pictures and texts, and the packaged STLV structure message packet comprises three STLV structure data parts, wherein the values of the structure identification codes 201 in the three STLV structure data parts are the same, and the log information of the same user is indicated; the structure type identification codes 203 are respectively type codes of videos, pictures and texts so as to distinguish different types of traffic; the structure data end identification code 202 in the first two STLV structure data sections are both "0", indicating that the data is not ended, and thereafter there is data, and the structure data end identification code 202 in the last STLV structure data section is "1", indicating that the data is ended; the value of the structure data length 204 is the length of the structure data 205 in the STLV structure data section, and information on video, picture, text traffic statistics in log information is stored in the structure data 205.
Step 2, if the length of the message packet is smaller than the maximum data packet size, the message packet is packaged into an MTP message block, and the MTP message block is sent after a universal message header is added;
in the present invention, the MTP message block includes an MTP message header 300 and MTP message data 400; as shown in fig. 3. The MTP message header includes an MTP message identification code 301, an MTP message block total length 302, an MTP message block sequence number 303, and an MTP message block end identification code 304. In this embodiment, the MTP message identifier 301 is 64bits, which is a global unique identifier; the total length 302 of the MTP message block and the length of the MTP message block sequence number 303 are both 16bits, and the MTP message block end identification code 304 is 1bit. In addition, to facilitate late expansion, the present embodiment adds a 31bits reserved bit 305 to the MTP message header 300, which also enables 4 byte alignment of the MTP message header 300.
The MTP message data 400 is the STLV structure message packet encapsulated in step 1.
In this embodiment, the maximum packet size is the difference between the value of the maximum transmission unit MTU and the length of the MTP message header.
The generic message header includes an MTP version number. The transmission data format after adding the Common Header is shown in fig. 4. In this embodiment, the MTP version number is 8bits data, and in addition, for later expansion, a reserved bit of 24bits is added, so that the universal message header can be aligned by 4 bytes.
Step 3, if the length of the message packet is greater than the maximum data packet size, slicing the message packet, wherein the length of each sliced message sub-packet is smaller than the maximum data packet size;
the segmented message sub-packets are packaged into MTP message blocks, the MTP message identification codes 301 in each MTP message block are the same, the MTP message block serial numbers 303 are increased according to the sequence of the message sub-packets in the message packet before segmentation, the MTP message block end identification codes 304 in the MTP message block corresponding to the last message sub-packet are 'end', and the MTP message block end identification codes 304 in the MTP message blocks corresponding to the rest message sub-packets are 'not end'; thereby realizing the transmission of the ultra-long STLV structure message.
The transmission data format after adding the common message header is shown in fig. 5.
The embodiment also discloses a system for implementing the method for sending the log message of the network traffic acquisition device, as shown in fig. 6, including:
the network flow log information acquisition module is used for acquiring log information of the network flow;
the STLV structure encapsulation module is used for encapsulating the acquired network flow log information into an STLV structure message packet according to the step 1;
the MTP message encapsulation module is used for encapsulating the message packet formed by the STLV structure into an MTP message block according to the step 2;
and the message sending module is used for adding the general message header to the MTP message block and then sending the MTP message block.
Example 2
The embodiment discloses a method for transmitting log messages of a network traffic acquisition device, wherein the log messages are sent according to the sending method in embodiment 1, as shown in fig. 7, and the method further comprises:
step 4, analyzing MTP message blocks from the received data;
step 5, analyzing a message packet consisting of the STLV structure from the analyzed MTP message block;
and 6, analyzing the network flow log information from the message packet.
The receiving side reads the received data, and skips the Common message header Common to read the MTP message header.
If the read MTP message block end identification code 304 in the MTP message header is 'end', only one MTP message block is read from the back of the MTP message header according to the value of the total length 302 of the MTP message block, and the data in the MTP message data 400 is read;
if the MTP message block end identification code 304 in the read MTP message header is "not ended," then there are multiple MTP message blocks. And reading the MTP message header of the next MTP message block according to the value of the total length 302 of the MTP message block, and so on, and acquiring all MTP message headers in the received data. Whether the same message packet is fragmented is determined based on whether the MTP message identification codes 301 in the MTP message headers are identical. If the MTP message blocks are segmented in the same message packet, the MTP message data in the MTP message blocks are sequentially analyzed according to the sequence numbers 303 of the MTP message blocks in the MTP message header, and the MTP message blocks are spliced together in sequence, namely the segmented ultra-long STLV structure message packets are recovered.
And skipping the STLV structure header from the MTP message data, reading the subsequent STLV structure data part, analyzing the structure data 205 according to the values of the structure identification code 201, the structure data end identification code 202, the structure type identification code 203 and the structure data length 204, and obtaining the original network flow log information sent by the sending end.
The embodiment also discloses a system for implementing the method for transmitting the log message of the network traffic acquisition device, as shown in fig. 8, including a sending system 500 and a receiving system 600, where the composition of the sending system 500 is shown in fig. 6, and the receiving system 600 includes:
the message receiving module is used for analyzing the MTP message block from the received data according to the step 4;
an MPT message block analysis module for analyzing the message packet composed of the STLV structure from the MTP message block according to the step 5;
and the STLV structure analysis module is used for analyzing the network flow log information from the message packet according to the step 6.

Claims (9)

1. The method for sending the log message of the network flow acquisition equipment is characterized by comprising the following steps:
acquiring log information of network traffic, and packaging the log information into a structured STLV structure message packet with a type length value;
if the length of the message packet is smaller than the maximum data packet size, the message packet is packaged into an MTP message block, and the MTP message block is sent after the general message header is added;
the acquired log information comprises flow statistics of videos, pictures and texts; the encapsulated STLV structure message packet comprises three STLV structure data parts, wherein the values of the structure identification codes (201) in the three STLV structure data parts are the same, and the three STLV structure data parts represent log information of the same user; the structure type identification codes (203) are respectively type codes of videos, pictures and texts so as to distinguish different types of traffic; the structure data end identification codes (202) in the first two STLV structure data sections are all "0", indicating that the data is not ended, and thereafter there is data, and the structure data end identification code (202) in the last STLV structure data section is "1", indicating that the data is ended; the value of the structure data length (204) is the length of the structure data (205) in the STLV structure data part, and the structure data (205) stores information about video, picture and text traffic statistics in log information.
2. A network traffic acquisition device log message sending method according to claim 1, characterized in that the STLV structure comprises one STLV structure header (100) and at least one STLV structure data section (200);
the STLV structure header (100) includes a log type (101), a log version number (102).
3. The network traffic collection device log message sending method of claim 1, wherein the MTP message block comprises an MTP message header (300) and MTP message data (400);
the MTP message header comprises an MTP message identification code (301), an MTP message block total length (302), an MTP message block serial number (303) and an MTP message block end identification code (304);
the MTP message data (400) is an STLV structured message packet.
4. A network traffic collecting device log message sending method according to claim 3, characterized in that the maximum packet size is the difference between the value of the maximum transmission unit and the length of the MTP message header (300).
5. The network traffic collection device log message sending method of claim 1, wherein the generic message header comprises an MTP version number.
6. The network traffic collection device log message sending method of claim 3, further comprising:
if the length of the message packet is greater than the maximum data packet size, slicing the message packet, wherein the length of each sliced message sub-packet is smaller than the maximum data packet size;
the segmented message sub-packets are packaged into MTP message blocks, the MTP message identification codes (301) in each MTP message block are the same, the MTP message block serial numbers (303) are increased progressively according to the sequence of the message sub-packets in the message packet before segmentation, the MTP message block end identification codes (304) in the MTP message block corresponding to the last message sub-packet are 'end', and the MTP message block end identification codes (304) in the MTP message blocks corresponding to the rest message sub-packets are 'not end'.
7. A network traffic collection device log message transmission system, comprising:
the network flow log information acquisition module is used for acquiring log information of the network flow; the acquired log information comprises flow statistics of videos, pictures and texts; the encapsulated STLV structure message packet comprises three STLV structure data parts, wherein the values of the structure identification codes (201) in the three STLV structure data parts are the same, and the three STLV structure data parts represent log information of the same user; the structure type identification codes (203) are respectively type codes of videos, pictures and texts so as to distinguish different types of traffic; the structure data end identification codes (202) in the first two STLV structure data sections are all "0", indicating that the data is not ended, and thereafter there is data, and the structure data end identification code (202) in the last STLV structure data section is "1", indicating that the data is ended; the value of the structure data length (204) is the length of the structure data (205) in the STLV structure data part, and the structure data (205) stores information about video, picture and text flow statistics in log information;
the STLV structure encapsulation module is used for encapsulating the acquired network traffic log information into an STLV structure message packet;
the MTP message encapsulation module is used for encapsulating the message packet formed by the STLV structure into an MTP message block;
and the message sending module is used for adding the general message header to the MTP message block and then sending the MTP message block.
8. A method for transmitting a log message of a network traffic collection device, where the log message is sent according to the sending method of any one of claims 1 to 6, and the method further includes:
analyzing MTP message blocks from received data;
analyzing a message packet consisting of an STLV structure from the analyzed MTP message block;
and analyzing the network traffic log information from the message packet.
9. A network traffic collecting device log message transmission system comprising a transmitting system and a receiving system, the transmitting system being the network traffic collecting device log message transmitting system of claim 7, the receiving system comprising:
the message receiving module is used for analyzing MTP message blocks from the received data;
the MPT message block analysis module is used for analyzing a message packet consisting of an STLV structure from the MTP message block; and the STLV structure analysis module is used for analyzing the network flow log information from the message packet.
CN202310128799.1A 2023-02-17 2023-02-17 Log message sending and transmitting method and system for network flow acquisition equipment Active CN116232710B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310128799.1A CN116232710B (en) 2023-02-17 2023-02-17 Log message sending and transmitting method and system for network flow acquisition equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310128799.1A CN116232710B (en) 2023-02-17 2023-02-17 Log message sending and transmitting method and system for network flow acquisition equipment

Publications (2)

Publication Number Publication Date
CN116232710A CN116232710A (en) 2023-06-06
CN116232710B true CN116232710B (en) 2023-12-29

Family

ID=86572612

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310128799.1A Active CN116232710B (en) 2023-02-17 2023-02-17 Log message sending and transmitting method and system for network flow acquisition equipment

Country Status (1)

Country Link
CN (1) CN116232710B (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110912943A (en) * 2019-12-30 2020-03-24 北京明朝万达科技股份有限公司 Cross-network traffic analysis system
CN114817190A (en) * 2022-04-29 2022-07-29 阿里云计算有限公司 Log synchronization method, device, system, equipment and storage medium

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI227612B (en) * 2003-06-25 2005-02-01 Hon Hai Prec Ind Co Ltd System and method for IP logging

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110912943A (en) * 2019-12-30 2020-03-24 北京明朝万达科技股份有限公司 Cross-network traffic analysis system
CN114817190A (en) * 2022-04-29 2022-07-29 阿里云计算有限公司 Log synchronization method, device, system, equipment and storage medium

Also Published As

Publication number Publication date
CN116232710A (en) 2023-06-06

Similar Documents

Publication Publication Date Title
CN101171806B (en) Method and apparatus for transmitting and receiving packet data using predefined length indicant in mobile communication system
KR100667739B1 (en) Apparatus for transmitting/receiving wireless data and method thereof
WO2022088779A1 (en) Deep packet processing method and apparatus, electronic device, and storage medium
CN109257143B (en) Method for fragmenting data packets for transmission in network transmission protocol with length limitation
KR101758882B1 (en) Method for transmitting broadcast signal, method for receiving broadcast signal, apparatus for transmitting broadcast signal, and apparatus for receiving broadcast signal
EP2479952A1 (en) Method for compressing and decompressing time stamp and equipment thereof
CN111083161A (en) Data transmission processing method and device and Internet of things equipment
CN111211980A (en) Transmission link management method, transmission link management device, electronic equipment and storage medium
CN101193284A (en) Validation method, system and device for transmission integrity of compressed file
CN102480335A (en) Method and system for transmitting business data
CN113810769A (en) Data transmission method and system
CN101453463A (en) Data packet packing, transmitting and receiving method for mobile multimedia broadcast system
CN115134138A (en) File synchronization method based on one-way optical gate
CN116405252A (en) Two-dimensional code data unidirectional transmission method
US6665292B1 (en) Transmission and reception of TCP/IP data over a wireless communication channel
CN101657981B (en) Method and apparatus for transmitting and receiving packet data unit in mobile communication system
CN112636880B (en) Side channel retransmission high-reliability ultra-high-definition video satellite transmission system
CN116232710B (en) Log message sending and transmitting method and system for network flow acquisition equipment
US6650636B1 (en) Transmission and reception of TCP/IP data over a wireless communication channel
WO2011113392A2 (en) Method for processing short messages, related device and system
CN101222516B (en) MAC layer data processing method, transmission device and receiving device
CN110611644A (en) Data transmission method and device
CN103020157A (en) High-reliability real-time file generation method spanning physical isolation
CN114979093B (en) RTP-based data transmission method, device, equipment and medium
CN112017049B (en) Security quotation forwarding system and method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant