CN108989301A - A kind of network flow data index method, equipment and storage medium indexed more - Google Patents
A kind of network flow data index method, equipment and storage medium indexed more Download PDFInfo
- Publication number
- CN108989301A CN108989301A CN201810716858.6A CN201810716858A CN108989301A CN 108989301 A CN108989301 A CN 108989301A CN 201810716858 A CN201810716858 A CN 201810716858A CN 108989301 A CN108989301 A CN 108989301A
- Authority
- CN
- China
- Prior art keywords
- data packet
- network flow
- session
- index
- flow data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/146—Markers for unambiguous identification of a particular session, e.g. session cookie or URL-encoding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L49/00—Packet switching elements
- H04L49/90—Buffering arrangements
- H04L49/9057—Arrangements for supporting packet reassembly or resequencing
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present invention relates to network flow data analysis technical fields, in view of the problems of the existing technology, provide network flow data index method, equipment and storage medium that one kind indexes more.The present invention first parses network flow data, and extracts data packet index information；Then, conversate recombination, and session recombination meaning is, can merge a large amount of data packet redundant information, reduces session and indexes volume；Finally, to all session establishment session index informations.Session index is made of the public head of session and data packet index information, and all data packet index informations for quickly navigating to session can be indexed by session.Data packet index be made of information such as data packet length, hereof deviation post and reference numbers of a document, can be indexed by data packet in reference number of a document, deviation post, data packet length, accurately read original network traffic data information.
Description
Technical field
The present invention relates to network flow data analysis technical field, especially a kind of network flow data index indexed more
Method, equipment and storage medium.
Background technique
As what the development of network technology and various network services were applied popularizes, it is raw that network has become people's routine work
Indispensable information carrying tool in work.By the analysis to network flow data, network technician can be helped more preferable
Awareness network operating condition, network traffic content, can preferably safeguard, optimize network, promote communication performance and safety,
Improve the efficiency solved the problems, such as.The explosive increase of network flow data is analyzed data for network technician and is brought greatly
Challenge, there are many disadvantages for existing mainstream network data on flows analysis tool, such as: load document speed is slow, hardware resource
It occupies high, each inquiry to be required to scan for entire file, response speed is slow, is only capable of one file of analysis every time, can not
Multifile analysis etc. simultaneously.
Summary of the invention
The technical problems to be solved by the present invention are: in view of the above problems, providing the network flow that one kind indexes more
Measure data index method, equipment and storage medium.Two-stage index can be quickly generated --- session index information and data packet index
Information.Firstly, parsing to network flow data, and extract data packet index information；Then, conversate recombination, session
Recombination meaning is, can merge a large amount of data packet redundant information, reduces session and indexes volume；Finally, being built to all sessions
Vertical session index information.Session index is made of the public head of session and data packet index information, can be indexed by session quick
Navigate to all data packet index informations of session.Data packet is indexed by data packet length, hereof deviation post and text
Part number etc. information constitute, can be indexed by data packet in reference number of a document, deviation post, data packet length, it is accurate
Read original network traffic data information.
The technical solution adopted by the invention is as follows:
The network flow data index method of index includes: more than a kind of
First class paper number is carried out to the file consolidation where network flow data packet to be processed；
The network flow data packet is parsed, network flow data packet index information is obtained；Wherein data packet index information
Number, network flow data packet including file where network flow data packet document misregistration position hereof and network
Data on flows package informatin；
It conversates recombination to network flow data packet, and carries out termination condition setting；
It based on the network flow data package informatin, extracts and generates session index information, then pass through the meeting of the session
Words index information network flow data packet index information corresponding with the session is associated；
For the session of end, file is written into the associated data packet index information of current sessions, for this document setting the
Second grade file number, while the number value and data packet index offset for the data packet index file number being arranged in session index
Then session index is written out to above-mentioned file by the deviation post value of position.
Further, when session is not finished, then add up session data packet quantity.
Further, the data that the session index information includes at least network flow data package informatin, initial value is arranged
Packet index file number, the data packet index offset position of setting initial value and session information and seven tuple informations.
Further, when not setting up the corresponding session index information of a certain network flow data packet, then it is based on institute again
Network flow data package informatin is stated, extract and generates session index information.
Further, by session index in data packet index file number and data packet index offset position find
Corresponding data packet index file information.
Further, pass through the number of file, network where the network flow data packet in data packet index file information
Find the network flow data packet in the document misregistration position of data on flows packet hereof.
Further, it conversates recombination to network flow data packet, and carries out termination condition setting and refer to judging net
The transport layer protocol flag bit or session timeout of network data on flows packet are foundation.
A kind of storage medium, wherein being stored with a plurality of instruction, described instruction is suitable for being loaded by processor and being executed described
More indexes network flow data index method the step of.
The network flow data indexing apparatus indexed a kind of includes processor more, is adapted for carrying out each instruction；And storage is set
It is standby, it is suitable for storing a plurality of instruction, described instruction is suitable for being loaded by processor and executing 1 to 7 any one institute of the claims
The network flow data index method for the more indexes stated.
In conclusion by adopting the above-described technical solution, the beneficial effects of the present invention are:
Detailed description of the invention
Examples of the present invention will be described by way of reference to the accompanying drawings, in which:
Structure chart Fig. 1 of the invention.
Specific embodiment
All features disclosed in this specification or disclosed all methods or in the process the step of, in addition to mutually exclusive
Feature and/or step other than, can combine in any way.
Any feature disclosed in this specification unless specifically stated can be equivalent or with similar purpose by other
Alternative features are replaced.That is, unless specifically stated, each feature is an example in a series of equivalent or similar characteristics
?.
Related description of the present invention:
1, network flow data package informatin includes data packet length, payload length, packet time.
Scheme 1: the network flow data index methods of index include: more than a kind of
Step 1: first class paper number is carried out to the file consolidation where network flow data packet to be processed；
Step 2: parsing the network flow data packet (original document of example as shown in figure 1), obtain network flow data packet
Index information；Wherein data packet index information includes reference number of a document (i.e. file where network flow data packet where the first order
Number, example as shown in figure 1 data packet index in original document ID), (network flow data packet is in file for first order deviation post
In document misregistration position, the deviation post in the data packet index of example as shown in figure 1) and network flow data package informatin (example
The data packet length etc. in data packet index as shown in figure 1)；
Step 3: conversating recombination to network flow data packet, and carry out termination condition setting；
Such as: it conversates recombination to network flow data packet, and carries out termination condition setting and refer to judging network flow
The transport layer protocol flag bit or session timeout for measuring data packet are foundation；
1) Transmission Control Protocol is then conversated recombination using five-tuple, and using FIN flag, RST mark or time-out as session
Termination condition；
2) it if it is udp protocol, is equally conversated recombination using five-tuple, and only using overtime as conversation end item
Part；
Wherein, five-tuple information includes source IP address, purpose IP address, source port, destination port, transport layer protocol), meeting
Words time-out refers between same endpoints within the scope of certain time without any data communication, endpoint refer to IP address it is identical and
Port is identical, ignores communication direction.
Step 4: being based on the network flow data package informatin, extract and generate session index information, then pass through the meeting
The session index information of words network flow data packet index information corresponding with the session is associated；
Wherein, the second level volume that the session index information includes at least network flow data package informatin, initial value is arranged
Number (data packet index file number, example as shown in figure 1 data packet index file number in session index information), setting initial value
Second level deviation post (data packet index offset position, the example data packet index offset position in session index information as shown in figure 1
Set), session information and seven tuple informations.
Session information includes session start time, conversation end time and session data packet quantity, session start time
Refer to that first network flow data packet acquisition time, conversation end time refer to that the last one network flow data packet obtains
Time, session data packet quantity is taken to refer to the quantity of network flow data packet；
Seven tuple informations include source IP, destination IP, source port, destination port, source MAC, target MAC (Media Access Control) address, transmission
Layer protocol.
Step 5: for the session of end, file is written into the associated data packet index information of current sessions, is this document
(second level number refers to data packet rope to the second level number that second level reference number of a document is arranged, while being arranged in session index
Draw reference number of a document) number value and second level deviation post (second level deviation post refers to data packet index offset position)
Then session index is written out to above-mentioned file by deviation post value.
Scheme 2: on the basis of scheme 1, when session is not finished, then add up session data packet quantity.
Scheme 3: on the basis of scheme 1 or 2, when not setting up the corresponding session index information of a certain network flow data packet
When, then it is based on the network flow data package informatin again, extracts and generates session index information.
Scheme 4, on the basis of scheme 1,2 or 3, by session index in data packet index file number and data packet
Find corresponding data packet index file information in index offset position；Then pass through the network flow in data packet index file information
The network flow data is found in the number of file, the document misregistration position of network flow data packet hereof where amount data packet
Packet.
The invention is not limited to specific embodiments above-mentioned.The present invention, which expands to, any in the present specification to be disclosed
New feature or any new combination, and disclose any new method or process the step of or any new combination.
Claims (9)
1. a kind of network flow data index method indexed more, characterized by comprising:
First class paper number is carried out to the file consolidation where network flow data packet to be processed；
The network flow data packet is parsed, network flow data packet index information is obtained；Wherein data packet index information includes
The document misregistration position hereof of the number of file, network flow data packet where network flow data packet and network flow
Packet information；
It conversates recombination to network flow data packet, and carries out termination condition setting；
It based on the network flow data package informatin, extracts and generates session index information, then pass through the session rope of the session
Fuse ceases network flow data packet index information corresponding with the session and is associated；
For the session of end, file is written into the associated data packet index information of current sessions, the second level is set for this document
Reference number of a document, while the number value for the data packet index file number being arranged in session index and data packet index offset position
Deviation post value, then by session index be written out to above-mentioned file.
2. according to the method described in claim 1, it is characterized in that then adding up session data packet quantity when session is not finished.
3. method according to claim 1 or 2, it is characterised in that the session index information includes at least network flow number
According to package informatin, the data packet index file number of initial value, the data packet index offset position of setting initial value, session letter are set
Breath and seven tuple informations.
4. according to the method described in claim 3, it is characterized in that when not setting up the corresponding session of a certain network flow data packet
When index information, then it is based on the network flow data package informatin again, extracts and generates session index information.
5. method according to claim 1,2 or 4, it is characterised in that by session index in data packet index file compile
Number and data packet index offset position find corresponding data packet index file information.
6. according to the method described in claim 5, it is characterized in that passing through the network flow number in data packet index file information
The network flow data packet is found according to the number of file, the document misregistration position of network flow data packet hereof where packet.
7. according to claim 1, method described in 2,4 or 6, it is characterised in that it conversates recombination to network flow data packet,
And it carries out termination condition setting and refers to the transport layer protocol flag bit for judging network flow data packet or session timeout for foundation.
8. a kind of storage medium, wherein being stored with a plurality of instruction, described instruction is suitable for being loaded by processor and being executed such as right
It is required that described in one of 1 to 7 the step of the network flow data index method of more indexes.
9. a kind of network flow data indexing apparatus indexed, it is characterised in that including processor, be adapted for carrying out each instruction more；With
And storage equipment, it is suitable for storing a plurality of instruction, described instruction is suitable for being loaded by processor and executing the claims 1 to 7
The network flow data index method of more indexes described in meaning one.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810716858.6A CN108989301A (en) | 2018-07-03 | 2018-07-03 | A kind of network flow data index method, equipment and storage medium indexed more |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810716858.6A CN108989301A (en) | 2018-07-03 | 2018-07-03 | A kind of network flow data index method, equipment and storage medium indexed more |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108989301A true CN108989301A (en) | 2018-12-11 |
Family
ID=64536484
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810716858.6A Withdrawn CN108989301A (en) | 2018-07-03 | 2018-07-03 | A kind of network flow data index method, equipment and storage medium indexed more |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108989301A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110704714A (en) * | 2019-09-27 | 2020-01-17 | 杭州九略智能科技有限公司 | Method and device for quickly indexing data of pcap file |
| CN111737222A (en) * | 2020-06-24 | 2020-10-02 | 四川长虹电器股份有限公司 | Message queue data packet storage and retrieval method based on one-to-many request response model |
| CN115604207A (en) * | 2022-12-12 | 2023-01-13 | 成都数默科技有限公司（Cn） | Session-oriented network flow storage and indexing method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101193273A (en) * | 2006-11-20 | 2008-06-04 | 中兴通讯股份有限公司 | A storage and playing method for real time multimedia image information |
| CN101540885A (en) * | 2009-04-30 | 2009-09-23 | 中兴通讯股份有限公司 | IPTV on-demand system and method |
| CN102231860A (en) * | 2011-06-03 | 2011-11-02 | 南京远古科技有限公司 | Live time shift data storage method |
| CN103281213A (en) * | 2013-04-18 | 2013-09-04 | 西安交通大学 | Method for extracting, analyzing and searching network flow and content |
| CN105324768A (en) * | 2013-06-13 | 2016-02-10 | 微软技术许可有限责任公司 | Dynamic query resolution using accuracy profiles |
| WO2017189080A1 (en) * | 2016-04-29 | 2017-11-02 | Qualcomm Incorporated | Narrow band synchronization signal |
-
2018
- 2018-07-03 CN CN201810716858.6A patent/CN108989301A/en not_active Withdrawn
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101193273A (en) * | 2006-11-20 | 2008-06-04 | 中兴通讯股份有限公司 | A storage and playing method for real time multimedia image information |
| CN101540885A (en) * | 2009-04-30 | 2009-09-23 | 中兴通讯股份有限公司 | IPTV on-demand system and method |
| CN102231860A (en) * | 2011-06-03 | 2011-11-02 | 南京远古科技有限公司 | Live time shift data storage method |
| CN103281213A (en) * | 2013-04-18 | 2013-09-04 | 西安交通大学 | Method for extracting, analyzing and searching network flow and content |
| CN105324768A (en) * | 2013-06-13 | 2016-02-10 | 微软技术许可有限责任公司 | Dynamic query resolution using accuracy profiles |
| WO2017189080A1 (en) * | 2016-04-29 | 2017-11-02 | Qualcomm Incorporated | Narrow band synchronization signal |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110704714A (en) * | 2019-09-27 | 2020-01-17 | 杭州九略智能科技有限公司 | Method and device for quickly indexing data of pcap file |
| CN111737222A (en) * | 2020-06-24 | 2020-10-02 | 四川长虹电器股份有限公司 | Message queue data packet storage and retrieval method based on one-to-many request response model |
| CN115604207A (en) * | 2022-12-12 | 2023-01-13 | 成都数默科技有限公司（Cn） | Session-oriented network flow storage and indexing method |
| CN115604207B (en) * | 2022-12-12 | 2023-03-10 | 成都数默科技有限公司 | Session-oriented network flow storage and indexing method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106209490B (en) | Select and monitor the method and system of multiple service key performance indicators | |
| US7570661B2 (en) | Script-based parser | |
| JP5889445B2 (en) | Method and apparatus for identifying an application associated with an IP flow using DNS data | |
| EP2434689B1 (en) | Method and apparatus for detecting message | |
| CN104320304B (en) | A kind of core network user flow application recognition methods of the multimode fusion easily extended | |
| CN108989301A (en) | A kind of network flow data index method, equipment and storage medium indexed more | |
| CN101557329B (en) | Application layer-based data segmenting method and device thereof | |
| US20130191890A1 (en) | Method and system for user identity recognition based on specific information | |
| CN105828310B (en) | Charging method, device and system for data service | |
| CN110401581A (en) | Industry control agreement fuzz testing case generation method based on flow retrospect | |
| CN103428261A (en) | Method to process HTTP header with hardware assistance | |
| US9894074B2 (en) | Method and system for extracting access control list | |
| CN103873356A (en) | Household gateway based application identification method and system, and household gateway | |
| CN109151880A (en) | Mobile application flow identification method based on multilayer classifier | |
| US9569421B2 (en) | Method and system for improved language identification using language tags | |
| CN104333483A (en) | Identification method, system and identification device for internet application flow | |
| US11032184B2 (en) | Method and device for collecting traffic flow value of BGP community attribute or BGP extended community attribute | |
| CN106452954B (en) | HTTP data characteristics analysis method and system | |
| CN116055448A (en) | Identification data management platform for electric power operation | |
| CN101771697B (en) | Network data stream identification method based on pattern matching method | |
| CN105939304A (en) | Tunnel message analysis method and device | |
| CN107517237A (en) | A kind of video frequency identifying method and device | |
| WO2021128936A1 (en) | Message processing method and apparatus | |
| CN114189572B (en) | Packet detection rule matching method, device, network element and storage medium | |
| Li et al. | MP-ROOM: Optimal matching on multiple PDUs for fine-grained traffic identification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20181211 |