CN111917777A - Network data analysis method and device and electronic equipment - Google Patents
Network data analysis method and device and electronic equipment Download PDFInfo
- Publication number
- CN111917777A CN111917777A CN202010767366.7A CN202010767366A CN111917777A CN 111917777 A CN111917777 A CN 111917777A CN 202010767366 A CN202010767366 A CN 202010767366A CN 111917777 A CN111917777 A CN 111917777A
- Authority
- CN
- China
- Prior art keywords
- analysis
- network
- data
- network data
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000007405 data analysis Methods 0.000 title claims abstract description 35
- 238000004458 analytical method Methods 0.000 claims abstract description 156
- 238000012545 processing Methods 0.000 claims abstract description 23
- 238000010586 diagram Methods 0.000 description 18
- 238000004891 communication Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 10
- 238000004590 computer program Methods 0.000 description 9
- 230000005540 biological transmission Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000010354 integration Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000003012 network analysis Methods 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000033772 system development Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/18—Protocol analysers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application discloses a network data analysis method and device and electronic equipment. The method comprises the following steps: acquiring network data; classifying and sniffing the network data according to the network data acquisition mode to obtain a sniffing result, wherein the sniffing result comprises the identified network protocol type; classifying, integrating and storing the network data in a database as a data source to be analyzed according to the sniffing result; and analyzing the data source to be analyzed in different modes and different degrees according to the analysis requirement of the user to obtain the analysis result of the network data. According to the network data analysis method, traditional automatic analysis can be achieved, layered analysis of any layer network data message and nested network protocol analysis can be achieved, independent addition of a private protocol is supported, and flexibility and expandability of network data processing are greatly improved.
Description
Technical Field
The present application relates to the field of network data processing technologies, and in particular, to a network data parsing method and apparatus, and an electronic device.
Background
In a network data processing scenario, due to a layered design of a network, different layered Protocol data exists on the network, such as UDP (User Datagram Protocol) Protocol data, which is a transmission layer often used in unreliable transmission, RTP (Real-time Transport Protocol) Protocol data in IP (Internet Protocol) voice communication, and the like. How to acquire and effectively analyze target protocol data has very important significance in network analysis and network security application.
At present, a conventional sniffing tool includes Wireshark (a network packet analysis software, which has no Chinese translation name temporarily), Sniffer (Sniffer, a network packet capturing tool), etc., Wireshark can analyze a network protocol layer by layer in a tree structure mode, and can meet certain practical application requirements, but protocol data in a practical network is very complex in form, and protocol repackaging conditions often occur, the network protocol is not strictly packaged layer by layer according to a sequence from high to low, and sometimes, a load of an upper layer protocol can nest some lower layer protocol data, which causes great difficulty for protocol analysis. In addition, the traditional sniffing tools generally only support analysis of protocols with public standards, and do not support private protocol analysis, and the protocols are often presented to users in a form of UI (User Interface), so that specific details of bottom layer development are hidden, great difficulty is caused to system development and maintenance, and application scenarios are greatly limited.
Disclosure of Invention
In view of the above technical problems, the present application is proposed to provide a network data parsing method, apparatus and electronic device that overcome the above technical problems or at least partially solve the above technical problems.
According to a first aspect of the present application, there is provided a network data parsing method, including:
acquiring network data;
classifying and sniffing the network data according to the network data acquisition mode to obtain a sniffing result, wherein the sniffing result comprises the identified network protocol type;
classifying, integrating and storing the network data in a database as a data source to be analyzed according to the sniffing result;
and analyzing the data source to be analyzed in different modes and different degrees according to the analysis requirement of the user to obtain the analysis result of the network data.
According to a second aspect of the present application, there is provided a network data parsing apparatus, including:
an acquisition unit configured to acquire network data;
the sniffing unit is used for classifying and sniffing the network data according to the network data acquisition mode to obtain a sniffing result, and the sniffing result comprises the identified network protocol type;
the database unit is used for classifying, integrating and storing the network data in a database as a data source to be analyzed according to the sniffing result;
and the analysis unit is used for analyzing the data source to be analyzed in different modes and different degrees according to the analysis requirement of the user to obtain the analysis result of the network data.
In accordance with a third aspect of the present application, there is provided an electronic device comprising:
a memory storing computer executable instructions and a processor,
the executable instructions, when executed by the processor, implement any of the foregoing network data parsing methods.
According to a fourth aspect of the present application, there is provided a computer readable storage medium storing one or more programs which, when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the network data parsing method as in any one of the preceding.
According to the technical scheme, the network data can be acquired through different data acquisition modes, and different network data analysis scenes are met; classifying and sniffing the network data according to different network data acquisition modes to obtain sniffing results such as network protocol types and other information; classifying, integrating and storing the network data in a database according to the sniffing result so as to provide data support for subsequent data analysis; and finally, analyzing the data source to be analyzed in different modes and different degrees in the database according to the analysis requirement of the user to obtain the analysis result of the network data. According to the network data analysis method, traditional automatic analysis can be achieved, layered analysis of any layer network data message and nested network protocol analysis can be achieved, independent addition of a private protocol is supported, and flexibility and expandability of network data processing are greatly improved.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the application. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
fig. 1 is a flowchart of a network data parsing method according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a database classification method according to an embodiment of the present application;
fig. 3 is a schematic diagram of data transmission of a digital communication system according to an embodiment of the present application;
FIG. 4 is a diagram illustrating a nested parsing process based on a PPP data frame structure according to an embodiment of the present application;
fig. 5 is a schematic diagram illustrating a network data parsing process according to an embodiment of the present application;
fig. 6 is a block diagram of a network data analysis device according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of an electronic device in an embodiment of the present application.
Detailed Description
Exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Fig. 1 is a schematic flowchart illustrating a network data parsing method according to an embodiment of the present application, and referring to fig. 1, the network data parsing method according to the embodiment of the present application includes the following steps S110 to S140:
step S110, network data is acquired.
Before network data analysis is performed, network data can be acquired by using a network data acquisition mode in the prior art, the source of the network data has various channels, and corresponding data acquisition modes are different, so that the network data can be acquired by directly intercepting and capturing messages in a target network, and physical layer data or link layer data such as network data in wireless communication signals can be acquired by processing and the like.
For the message data packet directly intercepted through the network, the message data packet can be directionally captured through a capture filter of a Tschark network protocol analysis tool. Tsharp is an open source network sniffing tool based on a command line and can be used independently of Wireshark, and the data acquisition efficiency is high. The acquisition of network packets in data of a physical layer or a link layer generally requires complex data processing, such as highly customized development. The two different network data acquisition modes are respectively suitable for different application scenes, the message interception is suitable for the situation that the target network data can be directly acquired, and the network message acquisition of the physical layer data or the link layer data needs to be customized according to the acquired network data. In a specific application, a person skilled in the art may flexibly select an acquisition manner of the network data according to an implementation requirement of a user, which is not specifically limited herein.
Step S120, classifying and sniffing the network data according to the network data acquisition mode to obtain a sniffing result, wherein the sniffing result comprises the identified network protocol type.
After the network data is obtained, a preliminary classification sniffing may be performed on the network data. Due to different network data acquisition modes, corresponding classification sniffing methods are also different, that is, the sniffing mode is closely related to the network data acquisition mode. Therefore, in specific implementation, the network data can be classified and sniffed correspondingly according to the network data acquisition mode, so as to obtain a sniffing result, where the sniffing result mainly includes the identified protocol type of the network data, and certainly may include other data, and how to configure the network data can be flexibly set by a person skilled in the art according to the actual needs of the user, which is not listed here.
The network protocols generally comprise physical layer protocols, link layer protocols, network layer protocols, transport layer protocols, application layer protocols and the like, most of the network protocols are realized by software and are flexible, so that more consideration is given to the convenience in realization when the protocols are specifically applied, some personalized protocol processing often appears, and great trouble is brought to protocol identification.
And step S130, classifying, integrating and storing the network data in a database as a data source to be analyzed according to the sniffing result.
In order to provide powerful data support for the subsequent network data parsing process, the embodiments of the present application may perform processing such as classification, integration, and storage on the network data according to the obtained sniffing result. Classification rules for network data, such as classification dimensions, etc., may be set according to the actual needs of the user. As shown in fig. 2, a schematic diagram of a database classification method is provided, in which network data related to a data link layer protocol (such as Ethernet protocol, ppp (Point to Point protocol), etc.) is classified into one type, network data related to a transport layer protocol (such as tcp (transmission Control protocol) transmission Control protocol, udp (user Datagram protocol), etc.) is classified into one type, and network data related to an application layer protocol (such as ftp (file Transfer protocol) file Transfer protocol, http (hypertext Transfer protocol), etc.) is classified into one type. The data integration and storage mode can also be configured according to the actual requirement of the user.
According to different application requirements, customized application development can be carried out based on the construction process of the database, so that protocol analysis personnel can know the transmission state, the network state and the like of the data message more intuitively, and support is provided for subsequent network analysis and protocol processing.
Step S140, according to the analysis requirement of the user, the data source to be analyzed is analyzed in different modes and different degrees, and the analysis result of the network data is obtained.
After the data source to be analyzed is obtained, deep analysis may be performed on the data packet in the data source to be analyzed according to different user requirements, which may specifically include performing analysis in different manners and at different degrees on the data packet.
According to the network data analysis method, the network data can be analyzed in a personalized mode according to the requirements of the user, and flexibility and expandability of network data processing are greatly improved.
In an embodiment of the present application, the classifying and sniffing the network data according to the network data acquisition manner to obtain a sniffing result includes: under the condition of intercepting a target network message to obtain the network data, firstly, utilizing a Tschark network protocol analysis tool to identify the network protocol of the network data layer by layer from a bottom layer to a high layer; and for network data which cannot be identified by the Tschark network protocol analysis tool, carrying out network protocol identification by utilizing a network data analysis rule which is customized and developed.
In specific implementation, as described above, due to different network data acquisition modes, the corresponding classification sniffing methods are also different. For a network data acquisition mode of directly intercepting a target network message, firstly, Tschark is adopted to identify an underlying network protocol, and on the basis of identifying the underlying network protocol, layer-by-layer identification of a high-level network protocol is further carried out, so that the protocol identification efficiency of Tschark is fully exerted. For network protocols which cannot be identified by Tsharp, usually some proprietary network protocols, customized and developed proprietary network protocol analysis rules can be adopted for individual identification, and manual intervention can be performed if necessary, so that the identification efficiency and accuracy of network data are improved.
In an embodiment of the present application, the classifying and sniffing the network data according to the network data acquisition manner to obtain a sniffing result includes: under the condition of acquiring and processing physical layer data or link layer data to obtain the network data, firstly, utilizing a customized and developed network data analysis rule to identify an underlying network protocol of the network data; then, the identified type and data of the underlying network protocol are transmitted to a Tschark network protocol analysis tool to identify the network protocol layer by layer; and for the network data which cannot be identified by the Tschark network protocol analysis tool, identifying the network protocol by using the customized and developed network data analysis rule again.
For the way of processing physical layer data or link layer data to acquire network data, a customized and developed network data analysis rule can be called to identify an underlying network protocol, the identified type and data of the underlying network protocol are transmitted to Tsharp to be identified layer by layer, network protocols which cannot be identified by Tsharp are identified, usually some private network protocols, can be individually identified by adopting a customized and developed privatized network protocol analysis rule, and can be manually intervened when necessary, so that the identification efficiency and accuracy of the network data are improved.
In this embodiment of the present application, the acquisition of the network packet of the physical layer data or the link layer data may specifically take the digital communication system shown in fig. 3 as an example, and this embodiment does not pay attention to specific implementation details of the digital communication system, and when the network analyzer clarifies various communication protocols (such as modulation, coding, frame structure, and the like) of the digital communication system, the data stream that is streamed to the sink after being decoded by the source may be transmitted by inter-process communication.
In an embodiment of the present application, the classifying, integrating, and storing the network data in a database according to the sniffing result, and the serving as a data source to be analyzed includes: classifying the network data according to the identified network protocol type to obtain corresponding network data of each layer; respectively acquiring sensitive protocol domains from network data of each layer, wherein the sensitive protocol domains refer to key characteristic information in the network data; integrating the network data according to the key characteristic information, and determining the association degree between the network data; forming network data with the relevance exceeding a preset threshold into a data set to obtain a plurality of data sets, and selecting a constant in each data set to determine as an index word corresponding to each data set; and storing the plurality of data sets in a database to serve as the data source to be analyzed, wherein a new data set needs to be compared with a historical data set according to index words and corresponding database operation is carried out before being stored in the database.
In specific implementation, the network data may be classified according to the identified network protocol type to obtain corresponding network data of each layer, such as a data link layer, a network layer, a transmission layer, an application layer, and the like. And then, respectively acquiring a sensitive protocol domain from each layer of network data, where the sensitive protocol domain may be understood as key feature information in the network data, such as an address domain of an ethernet or PPP data frame acquired from a data link layer, an IP address acquired from an IP packet of a network layer, or a network packet of a fixed port, which may be regarded as the sensitive protocol domain.
And then integrating the network data according to the key characteristic information, wherein the integration refers to forming the closely-associated data into a data set according to the association between the network data, and selecting a constant in the data set, such as IP fragmentation, a Media Access Control (MAC) address of mutual communication, and the like, as an index word of the data set, so that the subsequent application is facilitated. Before a new data set enters a database, the historical data set needs to be compared according to index words, and corresponding database operations such as addition, deletion, modification and the like are performed, and specific database operation types can be flexibly set by a person skilled in the art according to user requirements without specific limitation.
In an embodiment of the present application, the analyzing the data source to be analyzed in different manners includes: performing at least one of automatic analysis, nested analysis, any layer analysis and privatization analysis on the data source to be analyzed; the analyzing the data source to be analyzed to different degrees comprises: and acquiring partial analysis of the analysis result of the network protocol or acquiring full analysis of all aspects of the content of the network protocol as required.
In the embodiment of the present application, the parsing in different manners may include: automatic analysis, nested analysis, arbitrary layer analysis, private analysis, and the like. The automatic analysis adopts a Tschark traditional mode to directly carry out layer-by-layer analysis, provides a visual analysis result from bottom to top according to a network system structure, mainly supports the analysis of a public standard protocol, and can play an important role in application scenes such as network flow statistics and monitoring. The nested parsing refers to parsing a certain data packet load as required, and as shown in fig. 4, a PPP data frame structure is provided, where the data frame includes an IP packet and a UDP packet, and the UDP packet includes a lower protocol type, and for the complex protocol application mode, Tshark needs to be called according to the nested protocol type to perform parsing as required. For any layer of analysis, because the existing automatic analysis is a layer-by-layer analysis process, if only one high-layer protocol is adopted, the traditional layer-by-layer analysis method from the bottom layer to the high layer has no way to analyze the process, so that the any layer of analysis in the embodiment of the application can call the analysis rule of the corresponding layer of the Tschark network protocol analysis tool to analyze according to any protocol layer to be analyzed. The privatization analysis means that aiming at the analyzed and clarified privatization protocol, the purpose of analyzing the privatization protocol is achieved by solidifying the well-defined protocol rule into a separate protocol analysis module and embedding the protocol analysis module into Tschark to be used as a component of the protocol analysis module.
In the embodiment of the present application, the parsing to different degrees includes: partial resolution and full resolution. Partial analysis does not pay attention to all specific details of the protocol, and according to the specific application condition of a user, the network protocol analysis result is acquired as required, for example, an IP message only acquires an address domain and the like. And the full analysis acquires the grammar, the semantics, the time sequence and other contents of the network protocol to obtain a comprehensive analysis result.
As shown in fig. 5, a schematic diagram of a network data parsing flow is provided. Firstly, network data is acquired in a certain way, for example, a target network message is intercepted, or data of a physical layer or a link layer is processed. And then, according to different data acquisition modes, respectively adopting corresponding classification sniffing methods to identify the network data. And finally, according to the analysis requirements of users, carrying out analysis on the network data stored in the database according to requirements, such as automatic analysis, nested analysis, any layer analysis, privatization analysis and the like, so as to obtain a final network data analysis result.
The network data analysis method and the network data analysis device belong to the same technical concept, and the embodiment of the application also provides a network data analysis device. Fig. 6 is a block diagram of a network data parsing apparatus according to an embodiment of the present application, and referring to fig. 6, the network data parsing apparatus 600 includes: an acquisition unit 610, a sniffing unit 620, a database unit 630 and a parsing unit 640.
The obtaining unit 610 of the embodiment of the present application is configured to obtain network data.
The sniffing unit 620 in the embodiment of the application is configured to perform classified sniffing on the network data according to an acquisition mode of the network data to obtain a sniffing result, where the sniffing result includes an identified network protocol type.
The database unit 630 of the embodiment of the present application is configured to classify, integrate, and store the network data in a database as a data source to be analyzed according to the sniffing result.
The parsing unit 640 according to the embodiment of the application is configured to parse the data source to be parsed in different manners and at different degrees according to parsing requirements of a user, so as to obtain a parsing result of the network data.
In an embodiment of the present application, the obtaining unit 610 is further configured to: and intercepting a target network message, or processing data on a physical layer or a link layer to obtain the network data.
In an embodiment of the present application, the sniffing unit 620 is further configured to: under the condition of intercepting a target network message to obtain the network data, firstly, utilizing a Tschark network protocol analysis tool to identify the network protocol of the network data layer by layer from a bottom layer to a high layer; and for network data which cannot be identified by the Tschark network protocol analysis tool, carrying out network protocol identification by utilizing a network data analysis rule which is customized and developed.
In an embodiment of the present application, the sniffing unit 620 is further configured to: under the condition of acquiring and processing physical layer data or link layer data to obtain the network data, firstly, utilizing a customized and developed network data analysis rule to identify an underlying network protocol of the network data; then, the identified type and data of the underlying network protocol are transmitted to a Tschark network protocol analysis tool to identify the network protocol layer by layer; and for the network data which cannot be identified by the Tschark network protocol analysis tool, identifying the network protocol by using the customized and developed network data analysis rule again.
In an embodiment of the present application, the database unit 630 is further configured to: classifying the network data according to the identified network protocol type to obtain corresponding network data of each layer; respectively acquiring sensitive protocol domains from network data of each layer, wherein the sensitive protocol domains refer to key characteristic information in the network data; integrating the network data according to the key characteristic information, and determining the association degree between the network data; forming network data with the relevance exceeding a preset threshold into a data set to obtain a plurality of data sets, and selecting a constant in each data set to determine as an index word corresponding to each data set; and storing the plurality of data sets in a database to serve as the data source to be analyzed, wherein a new data set needs to be compared with a historical data set according to index words and corresponding database operation is carried out before being stored in the database.
In an embodiment of the present application, the parsing unit 640 is further configured to: performing at least one of automatic analysis, nested analysis, any layer analysis and privatization analysis on the data source to be analyzed; the automatic analysis refers to the step of calling a Tschark network protocol analysis tool to carry out layer-by-layer analysis from the bottom layer to the high layer on the data source to be analyzed; the nested analysis refers to calling the Tschark network protocol analysis tool to carry out analysis as required according to the protocol type nested in the data source to be analyzed; the arbitrary layer analysis refers to that the analysis rule of the corresponding layer of the Tschark network protocol analysis tool is called according to the arbitrary protocol layer to be analyzed to analyze the data source to be analyzed; the private analysis means that a clear private protocol analysis rule is solidified into an independent analysis module which is embedded into the Tsharp network protocol analysis tool, and the Tsharp network protocol analysis tool is called to carry out on-demand analysis on the data source to be analyzed; and acquiring partial analysis of the analysis result of the network protocol or acquiring full analysis of all aspects of the content of the network protocol as required.
It should be noted that, the network data analysis device can implement the steps of the network data analysis method executed by the electronic device provided in the foregoing embodiment, and the related explanations about the network data analysis method are applicable to the network data analysis device, and are not described herein again.
In summary, according to the technical scheme of the embodiment of the application, the network data can be acquired through different data acquisition modes, and different network data analysis scenarios are met. And classifying and sniffing the network data according to different network data acquisition modes to obtain sniffing results such as network protocol types and other information. And then classifying, integrating and storing the network data in a database according to the sniffing result so as to provide data support for subsequent data analysis. And finally, analyzing the data source to be analyzed in different modes and different degrees in the database according to the analysis requirement of the user to obtain the analysis result of the network data. According to the network data analysis method, traditional automatic analysis can be achieved, layered analysis of any layer network data message and nested network protocol analysis can be achieved, independent addition of a private protocol is supported, and flexibility and expandability of network data processing are greatly improved.
It should be noted that:
fig. 7 illustrates a schematic structural diagram of an electronic device. Referring to fig. 7, at a hardware level, the electronic device includes a memory and a processor, and optionally further includes an interface module, a communication module, and the like. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may also include a non-volatile Memory, such as at least one disk Memory. Of course, the electronic device may also include hardware required for other services.
The processor, the interface module, the communication module, and the memory may be connected to each other via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 7, but this does not indicate only one bus or one type of bus.
A memory for storing computer executable instructions. The memory provides computer executable instructions to the processor through the internal bus.
A processor executing computer executable instructions stored in the memory and specifically configured to perform the following operations:
acquiring network data;
classifying and sniffing the network data according to the network data acquisition mode to obtain a sniffing result, wherein the sniffing result comprises the identified network protocol type;
classifying, integrating and storing the network data in a database as a data source to be analyzed according to the sniffing result;
and analyzing the data source to be analyzed in different modes and different degrees according to the analysis requirement of the user to obtain the analysis result of the network data.
The functions performed by the network data analysis device according to the embodiment shown in fig. 6 of the present application may be implemented in a processor or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor.
The electronic device may further perform the steps performed by the network data analysis method in fig. 1, and implement the functions of the network data analysis method in the embodiment shown in fig. 1, which are not described herein again in this embodiment of the present application.
An embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores one or more programs, where the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to perform the network data parsing method in the embodiment shown in fig. 1, and are specifically configured to perform:
acquiring network data;
classifying and sniffing the network data according to the network data acquisition mode to obtain a sniffing result, wherein the sniffing result comprises the identified network protocol type;
classifying, integrating and storing the network data in a database as a data source to be analyzed according to the sniffing result;
and analyzing the data source to be analyzed in different modes and different degrees according to the analysis requirement of the user to obtain the analysis result of the network data.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) characterized by computer-usable program code.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) characterized by computer-usable program code.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A network data parsing method is characterized by comprising the following steps:
acquiring network data;
classifying and sniffing the network data according to the network data acquisition mode to obtain a sniffing result, wherein the sniffing result comprises the identified network protocol type;
classifying, integrating and storing the network data in a database as a data source to be analyzed according to the sniffing result;
and analyzing the data source to be analyzed in different modes and different degrees according to the analysis requirement of the user to obtain the analysis result of the network data.
2. The method of claim 1, wherein the obtaining network data comprises:
and intercepting a target network message, or processing data on a physical layer or a link layer to obtain the network data.
3. The method according to claim 2, wherein the classifying and sniffing the network data according to the network data acquisition manner to obtain the sniffing result comprises:
under the condition of intercepting a target network message to obtain the network data, firstly, utilizing a Tschark network protocol analysis tool to identify the network protocol of the network data layer by layer from a bottom layer to a high layer;
and for network data which cannot be identified by the Tschark network protocol analysis tool, carrying out network protocol identification by utilizing a network data analysis rule which is customized and developed.
4. The method according to claim 2, wherein the classifying and sniffing the network data according to the network data acquisition manner to obtain the sniffing result comprises:
under the condition of acquiring and processing physical layer data or link layer data to obtain the network data, firstly, utilizing a customized and developed network data analysis rule to identify an underlying network protocol of the network data; then, the identified type and data of the underlying network protocol are transmitted to a Tschark network protocol analysis tool to identify the network protocol layer by layer;
and for the network data which cannot be identified by the Tschark network protocol analysis tool, identifying the network protocol by using the customized and developed network data analysis rule again.
5. The method according to claim 1, wherein the classifying, integrating and storing the network data in a database according to the sniff result comprises:
classifying the network data according to the identified network protocol type to obtain corresponding network data of each layer;
respectively acquiring sensitive protocol domains from network data of each layer, wherein the sensitive protocol domains refer to key characteristic information in the network data;
integrating the network data according to the key characteristic information, and determining the association degree between the network data;
forming network data with the relevance exceeding a preset threshold into a data set to obtain a plurality of data sets, and selecting a constant in each data set to determine as an index word corresponding to each data set;
and storing the plurality of data sets in a database to serve as the data source to be analyzed, wherein a new data set needs to be compared with a historical data set according to index words and corresponding database operation is carried out before being stored in the database.
6. The method of claim 1, wherein the parsing the data source to be parsed in different manners comprises:
performing at least one of automatic analysis, nested analysis, any layer analysis and privatization analysis on the data source to be analyzed; wherein,
the automatic analysis refers to the step of calling a Tschark network protocol analysis tool to carry out layer-by-layer analysis from the bottom layer to the high layer on the data source to be analyzed;
the nested analysis refers to calling the Tschark network protocol analysis tool to carry out analysis as required according to the protocol type nested in the data source to be analyzed;
the arbitrary layer analysis refers to that the analysis rule of the corresponding layer of the Tschark network protocol analysis tool is called according to the arbitrary protocol layer to be analyzed to analyze the data source to be analyzed;
the private analysis means that a clear private protocol analysis rule is solidified into an independent analysis module which is embedded into the Tsharp network protocol analysis tool, and the Tsharp network protocol analysis tool is called to carry out on-demand analysis on the data source to be analyzed;
the analyzing the data source to be analyzed to different degrees comprises:
and acquiring partial analysis of the analysis result of the network protocol or acquiring full analysis of all aspects of the content of the network protocol as required.
7. A network data parsing apparatus, comprising:
an acquisition unit configured to acquire network data;
the sniffing unit is used for classifying and sniffing the network data according to the network data acquisition mode to obtain a sniffing result, and the sniffing result comprises the identified network protocol type;
the database unit is used for classifying, integrating and storing the network data in a database as a data source to be analyzed according to the sniffing result;
and the analysis unit is used for analyzing the data source to be analyzed in different modes and different degrees according to the analysis requirement of the user to obtain the analysis result of the network data.
8. The apparatus of claim 7, wherein the sniffing unit is further configured to:
under the condition of intercepting a target network message to obtain the network data, firstly, utilizing a Tschark network protocol analysis tool to identify the network protocol of the network data layer by layer from a bottom layer to a high layer;
and for network data which cannot be identified by the Tschark network protocol analysis tool, carrying out network protocol identification by utilizing a network data analysis rule which is customized and developed.
9. The apparatus of claim 7, wherein the parsing unit is further configured to:
performing at least one of automatic analysis, nested analysis, any layer analysis and privatization analysis on the data source to be analyzed; wherein,
the automatic analysis refers to the step of calling a Tschark network protocol analysis tool to carry out layer-by-layer analysis from the bottom layer to the high layer on the data source to be analyzed;
the nested analysis refers to calling the Tschark network protocol analysis tool to carry out analysis as required according to the protocol type nested in the data source to be analyzed;
the arbitrary layer analysis refers to that the analysis rule of the corresponding layer of the Tschark network protocol analysis tool is called according to the arbitrary protocol layer to be analyzed to analyze the data source to be analyzed;
the private analysis means that a clear private protocol analysis rule is solidified into an independent analysis module which is embedded into the Tsharp network protocol analysis tool, and the Tsharp network protocol analysis tool is called to carry out on-demand analysis on the data source to be analyzed;
the analyzing the data source to be analyzed to different degrees comprises:
and acquiring partial analysis of the analysis result of the network protocol or acquiring full analysis of all aspects of the content of the network protocol as required.
10. An electronic device, comprising: a memory storing computer executable instructions and a processor,
the executable instructions, when executed by the processor, implement the network data parsing method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010767366.7A CN111917777B (en) | 2020-08-03 | 2020-08-03 | Network data analysis method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010767366.7A CN111917777B (en) | 2020-08-03 | 2020-08-03 | Network data analysis method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111917777A true CN111917777A (en) | 2020-11-10 |
| CN111917777B CN111917777B (en) | 2023-04-18 |
Family
ID=73287012
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010767366.7A Active CN111917777B (en) | 2020-08-03 | 2020-08-03 | Network data analysis method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111917777B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112688924A (en) * | 2020-12-15 | 2021-04-20 | 中国海洋大学 | Network protocol analysis system |
| CN112714044A (en) * | 2020-12-28 | 2021-04-27 | 北京恒光信息技术股份有限公司 | Network data analysis method and device based on formal language protocol |
| CN112751845A (en) * | 2020-12-28 | 2021-05-04 | 北京恒光信息技术股份有限公司 | Network protocol analysis method, system and device |
| CN113242205A (en) * | 2021-03-19 | 2021-08-10 | 武汉绿色网络信息服务有限责任公司 | Network traffic classification control method, device, server and storage medium |
| CN113708990A (en) * | 2021-08-06 | 2021-11-26 | 上海龙旗科技股份有限公司 | Method and equipment for packet grabbing and unpacking of data packet |
| CN115208798A (en) * | 2022-09-16 | 2022-10-18 | 中国电子科技集团公司第三十研究所 | Automatic detection method, system, equipment and medium for Ethernet private line mode |
| CN115514570A (en) * | 2022-09-26 | 2022-12-23 | 于霄宇 | Network diagnosis processing method and system and cloud platform |
| CN116016702A (en) * | 2022-12-26 | 2023-04-25 | 浪潮云信息技术股份公司 | Application observable data acquisition processing method, device and medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090157904A1 (en) * | 2007-12-14 | 2009-06-18 | Smith Thomas M | Analysis tool for intra-node application messaging |
| CN104270392A (en) * | 2014-10-24 | 2015-01-07 | 中国科学院信息工程研究所 | Method and system for network protocol recognition based on tri-classifier cooperative training learning |
| CN104506484A (en) * | 2014-11-11 | 2015-04-08 | 中国电子科技集团公司第三十研究所 | Proprietary protocol analysis and identification method |
| CN105245407A (en) * | 2015-10-30 | 2016-01-13 | 盐城工学院 | Network sniffer based on socket and method thereof |
| CN106850338A (en) * | 2016-12-30 | 2017-06-13 | 西可通信技术设备（河源）有限公司 | A kind of R+1 classes application protocol recognition method and device based on semantic analysis |
| CN107395639A (en) * | 2017-08-29 | 2017-11-24 | 天津艾科仪科技有限公司 | Intelligence obtains the method and system of video data in network |
| KR101860645B1 (en) * | 2016-11-16 | 2018-05-23 | 국방과학연구소 | Apparatus and Method for implementing communication jamming scheme using packet spoofing in wireless network |
| CN111371651A (en) * | 2020-03-12 | 2020-07-03 | 杭州木链物联网科技有限公司 | Industrial communication protocol reverse analysis method |
-
2020
- 2020-08-03 CN CN202010767366.7A patent/CN111917777B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090157904A1 (en) * | 2007-12-14 | 2009-06-18 | Smith Thomas M | Analysis tool for intra-node application messaging |
| CN104270392A (en) * | 2014-10-24 | 2015-01-07 | 中国科学院信息工程研究所 | Method and system for network protocol recognition based on tri-classifier cooperative training learning |
| CN104506484A (en) * | 2014-11-11 | 2015-04-08 | 中国电子科技集团公司第三十研究所 | Proprietary protocol analysis and identification method |
| CN105245407A (en) * | 2015-10-30 | 2016-01-13 | 盐城工学院 | Network sniffer based on socket and method thereof |
| KR101860645B1 (en) * | 2016-11-16 | 2018-05-23 | 국방과학연구소 | Apparatus and Method for implementing communication jamming scheme using packet spoofing in wireless network |
| CN106850338A (en) * | 2016-12-30 | 2017-06-13 | 西可通信技术设备（河源）有限公司 | A kind of R+1 classes application protocol recognition method and device based on semantic analysis |
| CN107395639A (en) * | 2017-08-29 | 2017-11-24 | 天津艾科仪科技有限公司 | Intelligence obtains the method and system of video data in network |
| CN111371651A (en) * | 2020-03-12 | 2020-07-03 | 杭州木链物联网科技有限公司 | Industrial communication protocol reverse analysis method |
Non-Patent Citations (5)
| Title |
|---|
| 张楠等: "面向数据链路层的网络嗅探器的开发与实现", 《计算机应用》 * |
| 李莹等: "谈网络嗅探技术的实现", 《中国科技信息》 * |
| 王维颀: "局域网数据包抓取与分析器的设计", 《科技资讯》 * |
| 苏建美等: "网络嗅探器中Lua嵌入脚本的设计", 《软件》 * |
| 陈烽华等: "基于Raw Socket技术的改进Sniffer", 《计算机时代》 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112688924A (en) * | 2020-12-15 | 2021-04-20 | 中国海洋大学 | Network protocol analysis system |
| CN112714044A (en) * | 2020-12-28 | 2021-04-27 | 北京恒光信息技术股份有限公司 | Network data analysis method and device based on formal language protocol |
| CN112751845A (en) * | 2020-12-28 | 2021-05-04 | 北京恒光信息技术股份有限公司 | Network protocol analysis method, system and device |
| CN112714044B (en) * | 2020-12-28 | 2022-06-07 | 北京恒光信息技术股份有限公司 | Network data analysis method, device and storage medium based on formal language protocol |
| CN112751845B (en) * | 2020-12-28 | 2022-12-02 | 北京恒光信息技术股份有限公司 | Network protocol analysis method, system and device |
| CN113242205A (en) * | 2021-03-19 | 2021-08-10 | 武汉绿色网络信息服务有限责任公司 | Network traffic classification control method, device, server and storage medium |
| CN113708990A (en) * | 2021-08-06 | 2021-11-26 | 上海龙旗科技股份有限公司 | Method and equipment for packet grabbing and unpacking of data packet |
| CN113708990B (en) * | 2021-08-06 | 2022-12-27 | 上海龙旗科技股份有限公司 | Method and equipment for packet grabbing and unpacking of data packet |
| CN115208798A (en) * | 2022-09-16 | 2022-10-18 | 中国电子科技集团公司第三十研究所 | Automatic detection method, system, equipment and medium for Ethernet private line mode |
| CN115208798B (en) * | 2022-09-16 | 2023-01-31 | 中国电子科技集团公司第三十研究所 | Automatic detection method, system, equipment and medium for Ethernet private line mode |
| CN115514570A (en) * | 2022-09-26 | 2022-12-23 | 于霄宇 | Network diagnosis processing method and system and cloud platform |
| CN116016702A (en) * | 2022-12-26 | 2023-04-25 | 浪潮云信息技术股份公司 | Application observable data acquisition processing method, device and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111917777B (en) | 2023-04-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111917777B (en) | Network data analysis method and device and electronic equipment | |
| CN112468520B (en) | Data detection method, device and equipment and readable storage medium | |
| CN107085549B (en) | Method and device for generating fault information | |
| CN111130883A (en) | Method and device for determining topological graph of industrial control equipment and electronic equipment | |
| CN111818035B (en) | Permission verification method and device based on API gateway | |
| CN110784486A (en) | Industrial vulnerability scanning method and system | |
| CN111277569B (en) | Network message decoding method and device and electronic equipment | |
| CN113596078A (en) | Service problem positioning method and device | |
| CN115733894A (en) | Multi-protocol data access management method, device and equipment for power system | |
| CN102271331B (en) | Method and system for detecting reliability of service provider (SP) site | |
| CN112565232B (en) | Log analysis method and system based on template and flow state | |
| CN112688924A (en) | Network protocol analysis system | |
| WO2024174447A1 (en) | Data processing method and apparatus, storage medium and electronic device | |
| CN114285769B (en) | Shared internet surfing detection method, device, equipment and storage medium | |
| CN109462496B (en) | Data processing method and device for video network terminal | |
| CN105635225A (en) | Method and system of using mobile terminal to access mobile internet-based server and mobile terminal | |
| CN114039741A (en) | Sniffing method, system and device for internet surfing behavior and readable storage medium | |
| CN113965408A (en) | Method, device, medium and equipment for extracting HTTP (hyper text transport protocol) message | |
| CN113852551A (en) | Message processing method and device | |
| CN103634164B (en) | A kind of method and system for obtaining flow information | |
| CN111597198A (en) | Internet of things data query method for heterogeneous resource access and related equipment | |
| KR102001814B1 (en) | A method and apparatus for detecting malicious scripts based on mobile device | |
| CN105743992B (en) | Information processing method and device | |
| CN113542203B (en) | Video service DPI identification method and server | |
| CN117648204A (en) | Method, device, equipment and medium for capturing package of application program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |