CN101588275A - Method for information monitoring of network application layer - Google Patents
Method for information monitoring of network application layer Download PDFInfo
- Publication number
- CN101588275A CN101588275A CNA2008102414947A CN200810241494A CN101588275A CN 101588275 A CN101588275 A CN 101588275A CN A2008102414947 A CNA2008102414947 A CN A2008102414947A CN 200810241494 A CN200810241494 A CN 200810241494A CN 101588275 A CN101588275 A CN 101588275A
- Authority
- CN
- China
- Prior art keywords
- data
- user
- network
- packet
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a method for information monitoring of a network application layer, which aims to monitor, record, display and count various network data passing through service terminal equipment such as emails, web access and chatting records, provide efficient, convenient and quick network behavior analysis service for network management personnel, and prevent network users from abusing the internet, consuming resources, wasting working time, and even leaking company secrets. The method takes root in an opening source system platform, namely Linux, adopts a modular design concept, and can conveniently perform function extension. Main parts of the invention comprise: (a) a kernel packet receiving module with extremely high efficiency; (b) a classified analysis module with a user mode; and (c) a storage statistics module.
Description
Technical field
The present invention relates to the information monitoring of network application layer technology, this technology is one of core of broadband services terminal, its purport is the variety of network data of monitoring, record, demonstration, statistics process service terminal equipment, as mail, web page access, chat record, for the network management personnel provides high efficient and convenient network behavior Analysis Service.Prevent network user abuse the Internet, consumption of natural resource and waste operating time even leakage company secret.
Background technology
Along with the development of broadband network, network security and problem of management also highlight day by day.Network is also be scattered here and there when the facilitating attentiveness of enterprise staff of work.Some employees that lack self-discipline see news in the operating time, play games, chat, and What is more does the private job done during work hours instead of in one's spare time, company profile divulges a secret.These ways gently then consume corporate resource, and are heavy then have influence on operating efficiency, destroy the office atmosphere, become the serious hidden danger of a lot of corporate operations development.Manage the inevitable choice that Control Network also just becomes the corporate manager and introduce the network monitoring soft hardware equipment.
For the network monitoring function, mainly contain two kinds of implementations: employ full-time webmaster design and realize, perhaps buy network monitoring software and hardware product.The former is very high to webmaster personnel's technical merit requirement, and the latter is often expensive.In addition, both have a common limitation: the monitoring function on the monitoring host computer realizes that by user's attitude software program efficiency is not high fully.Consequently or monitoring function has influence on the unobstructed of network, office is caused negative effect; Monitoring is not sufficiently complete, and information is lost in a large number; Pay high hardware cost for reaching requirement.
Summary of the invention
The objective of the invention is to utilize a kind of much efficient monitoring technique of comparing with the legacy network monitor mode, realize the analysis of network application layer data monitoring, for the enterprise customer provides convenient network monitoring cheaply, storage, analysis, statistical integration solution.
The present invention is rooted in this system platform of increasing income of Linux, adopts modular design concept, can carry out the function expansion easily.And do not rely on the characteristics of particular hardware, make it can both operational excellence on various 32/64 main frame, now be transplanted to homemade Godson 64 bit platforms smoothly, can fully excavate it after tested and carry out potentiality, can satisfy the instructions for use of perfect in shape and function, low energy consumption high-efficiency rate fully.
Major part of the present invention comprises:
(a) the high kernel packet receiving module of efficient
(b) the classification parsing module of user's attitude
(c) storage statistical module
Below describe in detail with regard to each several part:
(a) the high kernel packet receiving module of efficient
This module is a core of the present invention, compares with user's attitude program packet receiving mode that other monitoring technique is adopted, and the kernel packet receiving can obtain the improved efficiency more than 10 times after tested.Its main process is:
(a1) connect with user's attitude program, create high-speed buffer, this step is only carried out when program start, and resource overhead is disposable.
(a2) operating system nucleus is accepted packet by network protocol stack, deposits it in buffering area.Because mainly operating in the kernel of packet receiving realized, do not need user's attitude program frequently to switch to kernel state and obtain data, therefore can obtain performance boost significantly.
(a3) by data packet delivery being arrived user's space, continued for second step after finishing with being connected of user's attitude program foundation.Because adopt kernel state and user's attitude communication mechanism of bottom, and adopted pass through mechanism in batches, the overhead in this step is few.
(b) the classification parsing module of user's attitude
After user's attitude program obtains the prompting of kernel, can obtain network packet by what set up with the kernel state program easy to connectly.But the packet of this moment is the original packet that contains the IP head, obtain the needed result to the user, also needs to do following processing:
(b1) recombination data bag obtains complete data flow.This step realizes according to the TCP/IP standard fully, guarantees compatibility and autgmentability to greatest extent.Also introduce buffering area mechanism in addition, incomplete data be saved in the buffering area of opening up when starting, etc. data obtain complete after unified again the processing.
(b2) data stream is analyzed the data that intercepting needs.This step also is one of key of improved efficiency, and this module is utilized senior data structure, and critical code is realized by compilation, reached designing requirement well.
(b3) need not the recombination data bag for individual other network application, as long as the keyword in each packet of analysis can obtain the data of needs, this also helps this module and even a whole set of supervisory control system can further promote efficient.
At present, the network application that user's morphotype piece can be analyzed mainly contains Email, web page browsing, chat communication, and except encrypting individually or relating to beyond the network application of copyright, whole monitoring discrimination is more than 95%.User's morphotype piece can also customize very easily, opens a certain, several monitoring functions separately according to user's demand, more supports to be specified by the user conditions such as the main frame monitored, period, provides flexibility in the use for the user to greatest extent.
(c) storage statistical module
After obtaining needed data, need deposit data in database.The design focal point of this part is to solve because the excessive database that brings of data volume is too huge the problem of storing queries utmost point consumes resources.Solution is:
(c1) data that will preserve are compressed, obtained optimal balance point at memory space with between the processing time
(c2) optimal Storage structure, appropriate design data tableau format is done independent preservation for the data that size is huge such as the mail of tens MB, only preserves its query path in the database.
(c3) adopt the principle of " resolving during inquiry ", the analysis of partial data is consuming time relatively, not as direct preservation complete content, by the time retrieve and resolve during the submit queries request.And the user often only can inquire about the sub-fraction in all records.Like this, increase to some extent although taken the expense of more hard drive space and single inquiry, as a whole, system resource can have very big saving.
Above functional module can be passed through running background, also can cooperate operation with web server by specific interface.The latter has hidden all realization details, for the user provides convenient understandable operating environment, can reduce the technical threshold of use well, and certain functions expanding is provided.Other relevant functional module also comprises log record, DB Backup, and power loss recovery or the like, these functions match with 3 main modular, can provide further help for the user uses equipment, supervising the network.
Total the above, the present invention by packet receiving, parsing, store these 3 modules and intactly realized the information monitoring of network application layer function, in each step serves as preferential to promote execution efficient all, take into account system compatibility and stability, provide high efficiency, manageability, integrated service terminal platform cheaply for the enterprise customer as far as possible.
Description of drawings
Fig. 1 is a general frame structure of the present invention;
Fig. 2 is the workflow of kernel packet receiving module of the present invention;
The workflow of Fig. 3 user's attitude classification parsing module;
Fig. 4 stores the workflow of statistical module
Embodiment
As shown in Figure 1, be general frame structure chart of the present invention.Shown the processing procedure of the present invention to packet 11, kernel packet receiving module 20 deposits high-speed buffer in being about to packet 11 after kernel protocol stack 12 obtains packet.The parsing module 30 of user's attitude can obtain complete packet from high-speed buffer then, according to transport layer, application layer protocol standard it is carried out recombination analysis, obtaining required information, and it is delivered to storage statistical module 40.The latter deposits database after with data qualification optimization in for user inquiring 14.Embodiment is as follows:
1, after linux system starts, loads corresponding networking rule, dispose the protocol type that to monitor by port numbers.After networking rule was loaded, kernel packet receiving module 20 parts of the present invention were written into the kernel operation, the following (see figure 2) of the operational process of this module with the form of kernel module:
1.1. starting, initialization 21, registration withdraw from routine 22;
1.2. initialization and user's attitude program are set up 23 the Link mechanism of communicating by letter;
1.3. create high-speed buffer 24;
1.4. the core network protocol stack is read in circulation, obtains packet by reading kernel protocol stack, the packet that obtains is placed into high-speed buffer 25;
1.5. according to port numbers coupling with the affiliated connection of determining this packet needs monitored 26 whether;
If 1.6. the match is successful, packet is delivered to user's space 27 by Link mechanism;
2, after the networking rule loading of previous step is finished, the classification parsing module of user's attitude begins on the system backstage to carry out with the form of independent finger daemon.The following (see figure 3) of the operational process of this module:
2.1. Data Structures initialization 31, reading database or file obtain start-up parameter;
2.2. the submodule that is used for packet reorganization and generation partial data stream is carried out initialization 32;
2.3. initialization is set up the Link mechanism 33 of communicating by letter with the kernel state program;
After 2.4. connect, begin circulation and obtain network packet 34 with the kernel state program;
2.5. the network packet that gets access to is carried out type matching 35, abandon the packet that need not to monitor, all the other bags are sent into packet recon module form complete data flow, these information will be kept in internal memory or the disk 36 according to the magnitude classification of size before obtaining fully;
2.6. when data flow obtain complete after, according to concrete application layer protocol it is resolved, obtain the needed information 37 of user;
2.7. need not the recombination data bag for individual other network application, data needn't be sent into bag recon module, directly it are resolved 38, obtain required information;
2.8. call the storage statistical module the resulting information in last two steps was handled 39.
3, the storage statistical module is logically independent, in still being included in the independent finger daemon of user's attitude, and the following (see figure 4) of its main flow process:
3.1. accept storage request 41;
3.2. the information that previous step obtains is compressed adjustment 42;
3.3. connect database 43;
3.4. preservation information is to database 44;
3.5. the turn-off data storehouse connects 45;
3.6. according to configuration, the result that will operate writes daily record 46 selectively.
Claims (4)
1, a kind of method of network application layer monitoring.Be to be rooted in this system platform of increasing income of Linux, adopt modular design concept, can carry out the function expansion easily.And do not rely on the characteristics of particular hardware, make it can both operational excellence on various 32/64 main frame, now be transplanted to homemade Godson 64 bit platforms smoothly, can fully excavate it after tested and carry out potentiality, can satisfy the instructions for use of perfect in shape and function, low energy consumption high-efficiency rate fully.It is characterized in that, comprise following major part:
(a) the high kernel packet receiving module of efficient
(b) the classification parsing module of user's attitude
(c) storage statistical module
2, the method for network layer monitoring according to claim 1 is characterized in that (a) is further comprising the steps of for content:
This module is a core of the present invention, compares with user's attitude program packet receiving mode that other monitoring technique is adopted, and the kernel packet receiving can obtain the improved efficiency more than 10 times after tested.Its main process is:
(a1) connect with user's attitude program, create high-speed buffer, this step is only carried out when program start, and resource overhead is disposable.
(a2) operating system nucleus is accepted packet by network protocol stack, deposits it in buffering area.Because mainly operating in the kernel of packet receiving realized, do not need user's attitude program frequently to switch to kernel state and obtain data, therefore can obtain performance boost significantly.
(a3) by data packet delivery being arrived user's space, continued for second step after finishing with being connected of user's attitude program foundation.Because adopt kernel state and user's attitude communication mechanism of bottom, and adopted pass through mechanism in batches, the overhead in this step is few.
3, the method for network layer monitoring according to claim 1 is characterized in that (b) is further comprising the steps of for content:
After user's attitude program obtains the prompting of kernel, can obtain network packet by what set up with the kernel state program easy to connectly.But the packet of this moment is the original packet that contains the IP head, obtain the needed result to the user, also needs to do following processing:
(b1) recombination data bag obtains complete data flow.This step realizes according to the TCP/IP standard fully, guarantees compatibility and autgmentability to greatest extent.Also introduce buffering area mechanism in addition, incomplete data be saved in the buffering area of opening up when starting, etc. data obtain complete after unified again the processing.
(b2) data stream is analyzed the data that intercepting needs.This step also is one of key of improved efficiency, and this module is utilized senior data structure, and critical code is realized by compilation, reached designing requirement well.
(b3) need not the recombination data bag for individual other network application, as long as the keyword in each packet of analysis can obtain the data of needs, this also helps this module and even a whole set of supervisory control system can further promote efficient.
At present, the network application that user's morphotype piece can be analyzed mainly contains Email, web page browsing, chat communication, and except encrypting individually or relating to beyond the network application of copyright, whole monitoring discrimination is more than 95%.User's morphotype piece can also customize very easily, opens a certain, several monitoring functions separately according to user's demand, more supports to be specified by the user conditions such as the main frame monitored, period, provides flexibility in the use for the user to greatest extent.
4, the method for network layer monitoring according to claim 1 is characterized in that (c) is further comprising the steps of for content:
After obtaining needed data, need deposit data in database.The design focal point of this part is to solve because the excessive database that brings of data volume is too huge the problem of storing queries utmost point consumes resources.Solution is:
(c1) data that will preserve are compressed, obtained optimal balance point at memory space with between the processing time
(c2) optimal Storage structure, appropriate design data tableau format is done independent preservation for the data that size is huge such as the mail of tens MB, only preserves its query path in the database.
(c3) adopt the principle of " resolving during inquiry ", the analysis of partial data is consuming time relatively, not as direct preservation complete content, by the time retrieve and resolve during the submit queries request.And the user often only can inquire about the sub-fraction in all records.Like this, increase to some extent although taken the expense of more hard drive space and single inquiry, as a whole, system resource can have very big saving.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNA2008102414947A CN101588275A (en) | 2008-12-25 | 2008-12-25 | Method for information monitoring of network application layer |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNA2008102414947A CN101588275A (en) | 2008-12-25 | 2008-12-25 | Method for information monitoring of network application layer |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN101588275A true CN101588275A (en) | 2009-11-25 |
Family
ID=41372349
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNA2008102414947A Pending CN101588275A (en) | 2008-12-25 | 2008-12-25 | Method for information monitoring of network application layer |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101588275A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103259695A (en) * | 2013-04-16 | 2013-08-21 | 李军 | Portable internet terminal monitor and monitoring method thereof |
| CN103780410A (en) * | 2012-10-19 | 2014-05-07 | 任子行网络技术股份有限公司 | Content obtaining system and method for encryption application |
| CN108075942A (en) * | 2016-11-15 | 2018-05-25 | 天津慧康溢德科技发展有限公司 | A kind of Networking Education Technology service trade application monitoring system |
| CN108123933A (en) * | 2017-12-05 | 2018-06-05 | 南京南邮信息产业技术研究院有限公司 | Information leakage automatic monitoring method and system based on internet big data |
| CN108256320A (en) * | 2017-12-27 | 2018-07-06 | 北京梆梆安全科技有限公司 | Differential domain dynamic testing method and device, equipment and storage medium |
| CN108446304A (en) * | 2018-01-30 | 2018-08-24 | 上海天旦网络科技发展有限公司 | Data block retrieval system and method |
| CN109274774A (en) * | 2018-11-16 | 2019-01-25 | 郑州云海信息技术有限公司 | A kind of date storage method, device and computer readable storage medium |
| CN113835856A (en) * | 2021-09-17 | 2021-12-24 | 苏州浪潮智能科技有限公司 | Storage statistical method, device and equipment for AI platform |
-
2008
- 2008-12-25 CN CNA2008102414947A patent/CN101588275A/en active Pending
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103780410A (en) * | 2012-10-19 | 2014-05-07 | 任子行网络技术股份有限公司 | Content obtaining system and method for encryption application |
| CN103780410B (en) * | 2012-10-19 | 2017-06-06 | 任子行网络技术股份有限公司 | A kind of content obtaining system and method for encrypting application |
| CN103259695A (en) * | 2013-04-16 | 2013-08-21 | 李军 | Portable internet terminal monitor and monitoring method thereof |
| CN103259695B (en) * | 2013-04-16 | 2018-11-16 | 广东恒电信息科技股份有限公司 | Portable internet terminal monitor and its monitoring method |
| CN108075942A (en) * | 2016-11-15 | 2018-05-25 | 天津慧康溢德科技发展有限公司 | A kind of Networking Education Technology service trade application monitoring system |
| CN108123933A (en) * | 2017-12-05 | 2018-06-05 | 南京南邮信息产业技术研究院有限公司 | Information leakage automatic monitoring method and system based on internet big data |
| CN108123933B (en) * | 2017-12-05 | 2020-12-08 | 南京南邮信息产业技术研究院有限公司 | Information leakage automatic monitoring method and system based on internet big data |
| CN108256320A (en) * | 2017-12-27 | 2018-07-06 | 北京梆梆安全科技有限公司 | Differential domain dynamic testing method and device, equipment and storage medium |
| CN108446304A (en) * | 2018-01-30 | 2018-08-24 | 上海天旦网络科技发展有限公司 | Data block retrieval system and method |
| CN109274774A (en) * | 2018-11-16 | 2019-01-25 | 郑州云海信息技术有限公司 | A kind of date storage method, device and computer readable storage medium |
| CN113835856A (en) * | 2021-09-17 | 2021-12-24 | 苏州浪潮智能科技有限公司 | Storage statistical method, device and equipment for AI platform |
| CN113835856B (en) * | 2021-09-17 | 2023-07-14 | 苏州浪潮智能科技有限公司 | Storage statistics method, device and equipment of AI platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101588275A (en) | Method for information monitoring of network application layer | |
| CN101853287B (en) | Data compression quick retrieval file system and method thereof | |
| CN100521623C (en) | High-performance Syslog processing and storage method | |
| WO2019134226A1 (en) | Log collection method, device, terminal apparatus, and storage medium | |
| US9037960B2 (en) | Monitoring and tracking application usage | |
| CN104778188B (en) | A kind of distributed apparatus log collection method | |
| JP4403068B2 (en) | High-performance change capture for data warehousing | |
| CN109739818B (en) | Portable high-throughput big data acquisition method and system | |
| CN106897411A (en) | ETL system and its method based on Spark technologies | |
| CN103546343B (en) | The network traffics methods of exhibiting of network traffic analysis system and system | |
| CN102648464A (en) | System and method for generating vocabulary from network data | |
| CA2343288A1 (en) | Rules-based multimedia customer/enterprise interaction-network operating-system | |
| CN102571420A (en) | Method and system for network element data management | |
| US20030126109A1 (en) | Method and system for converting message data into relational table format | |
| CN101447064B (en) | Auditing management system and auditing management method | |
| CN102497427A (en) | Method and device for realizing data acquisition services of renewable energy source monitoring system | |
| CN103124226A (en) | Household broadband net-system play monitoring system and method | |
| CN104618410B (en) | Resource supplying method and apparatus | |
| CN101998687A (en) | Method and device for collecting network parameters of WLAN (Wireless Local Area Network) | |
| CN103870560A (en) | Log managing method for cluster | |
| CN107426017A (en) | A kind of method for carrying out data analysis by gathering switch network flow | |
| US8819135B2 (en) | Method of performing data mediation, and an associated computer program product, data mediation device and information system | |
| CN101420773A (en) | Self-protection system for sensor network operating system | |
| CN102306148A (en) | Share memory database access method | |
| CN105991789A (en) | Method for realizing virtual machine port mapping, servers and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Open date: 20091125 |