CN102316074A - HTTP (hyper text transfer protocol) multithreading restoration method based on libnids - Google Patents
HTTP (hyper text transfer protocol) multithreading restoration method based on libnids Download PDFInfo
- Publication number
- CN102316074A CN102316074A CN2010102150683A CN201010215068A CN102316074A CN 102316074 A CN102316074 A CN 102316074A CN 2010102150683 A CN2010102150683 A CN 2010102150683A CN 201010215068 A CN201010215068 A CN 201010215068A CN 102316074 A CN102316074 A CN 102316074A
- Authority
- CN
- China
- Prior art keywords
- multithreading
- libnids
- packet
- data
- content
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention designs an HTTP (hyper text transfer protocol) multithreading restoration method based on libnids. The system operates in a Linux operating system, and the computer functions for running the system are required to be normal and run stably; and additionally, the computer which is provided with software in the invention is in a local area network. The method comprises the following steps of: firstly setting a network card of a host computer at a promiscuous mode; then monitoring all data packets passing through the network segment; judging whether the data packets are HTTP data packets or not; naming the data packets through a URL (uniform resource locator) if yes, recording a source IP (Internet protocol), a source port and a target IP to be stored in a data structure as a tetrad; judging whether the data packets are data packets downloaded through the multithreading continuously, and recording two variables such as content_range and content_length if yes; calculating the positions of the data packets in the original file based on the content_range and content_length; and writing the contents in the data packets in the corresponding positions. The method provided by the invention has the following beneficial effects that: (1) integrality: most of the data packets in the local area network can be captured, and files comprising the files downloaded through the multithreading can be restored; (2) continuity: the system can run stably in a stable environment; and (3) high efficiency: a multithreading buffer area technology is adopted to process the protocols of different application layers individually, the processing efficiency is improved, and most of the data can be still captured under the condition of great network flow.
Description
Technical field
The present invention relates to malicious code.
Background technology
Present based on network attack is more and more general and changes various; And these based on network intrusion behaviors are difficult to attacked the main frame discovery; Therefore the place that based on network in recent years monitoring instruction system is different from Host Based intruding detection system is that main frame type intruding detection system can directly obtain each item information of main frame; Like system's running log etc.; And Network Intrusion Detection System can only obtain the packet of network communication; Can't directly obtain the particular content of communication, can only be through packet be carried out the particular content that protocol assembly obtains network service, most of network intrusions is to have only according to the content of these reduction just to carry out the intrusion behavior detection.Because the extensive use of TCP/IP network, present most intruding detection system all are to be directed against the TCP/IP network.As far as the TCP/IP network, the reduction of ICP/IP protocol becomes one of key technology of Network Intrusion Detection System.
Some relevant patents are also arranged at present, do a little introductions below.
Patent 200610125451.3 is used for the method for HTTP reduction of data.A kind of method that is used for the HTTP reduction of data; It is characterized in that: intercept and capture request msg that client sends server and server end response data client-requested; Filter then, parsing, buffer memory; Form the overall data of html form, if during transfer of data through transfer encoding then the overall data that forms is separated transfer encoding; If data are compressed before transmission the overall data that forms is decompressed, so just form the data that can use browser direct display, the reduction of http data is also just accomplished.HTTP bag data to these requests and response are intercepted and captured and are handled, and just can the reduction of data of HTTP bag be come out, and user's internet behavior also can reproduce out.But along with the development of internet, with a sudden peal of thunder, Flashget, download whirlwind etc. spread unchecked for the multithreading download tool of representative day by day.In the HTTP reduction of data, the situation that many IP address and multiport are downloaded identical file gets more and more.And 200610125451.3 of patents have realized under the situation one to one; Promptly a client is downloaded a file a service end; Protocol assembly; Do not consider one-to-many (client is downloaded same file in a plurality of service ends), and the reduction situation of multi-to-multi (a plurality of clients are downloaded same file in a plurality of service ends), further research do not done.In comparison; This patent can not only be complete catch most packet in the local area network (LAN); Can complete reduction comprise the multithreading downloaded files, can also adopt the multithreading buffer technology, improve treatment effeciency to different application layer protocol individual processing; Guaranteed under the very big situation of network traffics, still can capture most data.
Summary of the invention
The invention provides a kind of method of reducing of the http agreement multithreading based on libnids, it has integrality, continuity, high efficiency.
At first host network card is made as promiscuous mode, monitors all then and each packet is judged through the packet of this network segment.At first the judgment data bag is from client or service end.If client data has the HTTP characteristic value when then searching in the get field, if whether service end then has the HTTP characteristic value in the judgment data packet header.If confirming as the HTTP packet then also passing through URL is analyzed through the URL in the extraction packet is this packet name, note source IP simultaneously, source port, purpose IP deposits a data structure in as a five-tuple.All five-tuple data structures are together in series with the packet name as unique keyword of confirming all packets of a file with chained list.Continue whether the judgment data bag is a multithreading data downloaded bag; Add that data name and entity content form second chained list if then note content_range and these two variablees of content_length, its effect is to be that entity writes file content and locatees when writing file., identical file also need set up the 3rd data structure again when writing data comprising filename, content_range, and long file_position is used to locate concrete writing position.In the multithreading downloading process; The mode that packet arrives the customer side is with content_range file to be divided into a plurality of sections transmission simultaneously; So the order of packet arrival customer side is out of order in each section; Will be with its section that belongs to of these packet peaces and orderly the combining with regard to needs in position that in the section of belonging to, belongs to through three data structure co.Content in the packet is write correspondence position.If data structure and chained list are the untimely deletions of dynamic growth then internal memory is exhausted.Program can't steady in a long-term be moved, can cause file successfully to reduce again but when all data of a file also all do not arrive the customer side, just in chained list, delete its data structure.Based on we adopt Libnids that nids_state is provided in the delete procedure of above consideration data structure in chained list.Our packet that just is regarded as a file all arrives if the state of nids_state is NIDS_CLOSE and NIDS_RESET; Has the data structure of identical filename through the filename keyword search in the chained list, so the data structure of same filename is deleted.Program is divided into two threads.Main thread is responsible for catching packet, analyzes agreement, the assembling data.Inferior thread is responsible for creating file, writes data, content control etc.
This patent system comprises following five functional modules:
Libnids: major function comprises catches network packet, ip fragmentation reorganization, the reorganization of tcp data stream and Port Scan Attacks test and the test of abnormal data bag etc.
Filtering data bag module: with the packet that libnids captures, analyze, filter the packet that drops out beyond the http agreement.
Protocol-analysis model: extract in the http agreement the information that will use, mainly comprise: filename, purpose ip, port etc.
The data writing module: the packet that will belong to same four-tuple writes identical file.
The file delete module: whether the packet of judging identical file end of transmission, if end of transmission its four-tuple of deletion in internal memory just.
The Libnids module mainly was responsible for wrapping catching when this patent was implemented; Filtering data bag module is responsible for packet filtering; Protocol-analysis model is responsible for from bag, extracting critical data; The data writing module is responsible for writing the data packet file, and the file delete module is responsible for judging whether end of transmission of file, has deleted expired data structure.
Native system moves in (SuSE) Linux OS, requires the computer function of operation native system normal, stable.In addition, the computer that requires this patent software is installed is in the local area network (LAN).And the network interface card of this computer is set to promiscuous mode.
Description of drawings
Fig. 1 is the protocal analysis flow chart;
Fig. 2 is the data stream filtering module;
Fig. 3 is the data writing module;
Fig. 4 is the file delete module
Fig. 5 is the particular flow sheet of this patent;
Embodiment
Below in conjunction with accompanying drawing technical scheme of the present invention is elaborated.
Fig. 1 has shown the process of protocal analysis, and the major function of protocal analysis comprises that branch writes the http agreement, extracts critical data.Each step of refinement Fig. 1 is following:
Step S101: judge it is that packet belongs to customer side or service end through analyzing packet packet header first row.
Step S102: if which kind of state is service end data extract conditional code belong in order to judge to connect.
Step S103: if the URL that provides in the service end data extract packet.
Step S104: analyze URL and extract the also filename of original through the regular expression coupling.
Step S105: extract purpose IP, order ground port, source IP, source port are used for the chained list location.
Step S106: locating bias amount when extraction content_length and content_range field are used to write file.
Step S107: extract the entity content, be used for the file reduction.
Fig. 2 representes the filtering data bag, analyzes the data packet message head, has judged whether the http characteristic value.If do not filter, otherwise give protocol-analysis model.It has comprised following steps:
Step S201: judge it is that packet belongs to customer side or service end through analyzing packet packet header first row.
Step S202: the customer side request header, judge the get field, if do not have then discarded packets
Step S203: the customer side is judged does not have characteristic value HTTP.If do not have then discarded packets
Step S204:, judge whether the beginning part is HTTP if be service end data then extract the packet statusline.If have the HTTP characteristic value then be regarded as the HTTP packet.
Fig. 3 representes the data writing module, and the entity content of extracting is write file by its position hereof.Detailed process is following:
Step S301: so because filename is to obtain having uniqueness through the protocol-analysis model analysis at first to judge through filename whether file is created.
Step S302: the conditional code through protocol-analysis model extracts judges that whether file belongs to is the multithreading file in download
Step S303: if the multithreading file in download is extracted content_range, content_length attribute.Be used for document alignment.
Step S304: the cheap amount of entity content peace file is write the file correspondence position.
Fig. 4 representes the data removing module.The judgment data connection status, corresponding data structure in the deletion chained list.
Step S401: monitoring nids_state variable state.
Step S402: judge whether the nids_state state is NIDS_CLOSE and NIDS_RESET.This two states representation file end of transmission.
Step S403: through ip order ground, order ground port, source ip source port finds chained list corresponding data structure.This four-tuple can be confirmed all packets of identical file in chained list
Step S404: extract the filename in the data structure.Filename not only can be confirmed all bags of identical file in the chained list, and it is all packets of unique definite identical file in three chained lists simultaneously.
Step S405: find all data structures in three chained lists according to filename.
Step S406: deleted data structure.The releasing memory space.
Though this specification has only been described the details of said method; And do not refer to application of the present invention more; But owing to the important value of http agreement multithreading method of reducing in malicious code research based on libnids; Its application surface is very widely, so the spirit and scope of the present invention should not be confined to embodiment described herein.
Claims (10)
1. the http agreement multithreading method of reducing based on libnids is characterized in that and will host network card be made as promiscuous mode, monitors all packets through this network segment then.Judge whether it is the packet of http agreement.If then be this packet name, and note source IP through URL, source port, purpose IP deposits a data structure in as a four-tuple.Continue whether the judgment data bag is a multithreading data downloaded bag, if then note content_range and these two variablees of content_length.Calculate the position of packet in original document through content_range and content_length.Content in the packet is write correspondence position.
2. like right 1 described http agreement multithreading method of reducing, it is characterized in that native system moves in (SuSE) Linux OS, require the computer function of operation native system normal, stable based on libnids.In addition, the computer that requires this patent software is installed is in the local area network (LAN).And the network interface card of this computer is set to promiscuous mode.
3. like right 1 described http agreement multithreading method of reducing based on libnids; It is characterized in that, adopt the multithreading buffer technology, improve treatment effeciency to different application layer protocol individual processing; Can guarantee under the very big situation of network traffics, still can capture most data.
4. like right 1 described http agreement multithreading method of reducing, it is characterized in that libnids will be responsible for catching network packet, ip fragmentation reorganization, the reorganization of tcp data stream and Port Scan Attacks test and the test of abnormal data bag etc. based on libnids.
5. like right 1 described http agreement multithreading method of reducing, it is characterized in that the packet that filtering data bag module will capture libnids is analyzed, and filters the packet that drops out beyond the http agreement based on libnids.
6. like right 1 described http agreement multithreading method of reducing, it is characterized in that based on libnids, protocol-analysis model must extract in the http agreement the information that will use, mainly comprise: filename, purpose ip, port etc.
7. like right 1 described http agreement multithreading method of reducing, it is characterized in that the data writing module is responsible for the packet that belongs to same four-tuple is write identical file based on libnids.
8. like right 1 described http agreement multithreading method of reducing, it is characterized in that the packet that the file delete module will be judged identical file is end of transmission whether, if end of transmission its four-tuple of deletion in internal memory just based on libnids.
9. like right 2 described http agreement multithreading method of reducing based on libnids, it is characterized in that having continuity, system can stable operation in a stable environment.
10. like right 3 described http agreement multithreading method of reducing, it is characterized in that based on libnids, can be complete catch most packet in the local area network (LAN), can complete reduction comprise the multithreading downloaded files.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010102150683A CN102316074A (en) | 2010-07-01 | 2010-07-01 | HTTP (hyper text transfer protocol) multithreading restoration method based on libnids |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010102150683A CN102316074A (en) | 2010-07-01 | 2010-07-01 | HTTP (hyper text transfer protocol) multithreading restoration method based on libnids |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102316074A true CN102316074A (en) | 2012-01-11 |
Family
ID=45428900
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2010102150683A Pending CN102316074A (en) | 2010-07-01 | 2010-07-01 | HTTP (hyper text transfer protocol) multithreading restoration method based on libnids |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102316074A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103618720A (en) * | 2013-11-29 | 2014-03-05 | 华中科技大学 | Method and system for Trojan network communication detecting and evidence obtaining |
| CN104394211A (en) * | 2014-11-21 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Design and implementation method for user behavior analysis system based on Hadoop |
| CN105491158A (en) * | 2016-01-15 | 2016-04-13 | 成都科来软件有限公司 | HTTP content reduction method and HTTP content reduction system based on network data flow |
| CN106911637A (en) * | 2015-12-23 | 2017-06-30 | 北京奇虎科技有限公司 | Cyberthreat treating method and apparatus |
| CN106911640A (en) * | 2015-12-23 | 2017-06-30 | 北京奇虎科技有限公司 | Cyberthreat treating method and apparatus |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1997030A (en) * | 2006-12-13 | 2007-07-11 | 武汉虹旭信息技术有限责任公司 | Method for HTTP data recovery |
| US20080033905A1 (en) * | 2006-08-05 | 2008-02-07 | Terry Lee Stokes | System and Method for the Capture and Archival of Electronic Communications |
-
2010
- 2010-07-01 CN CN2010102150683A patent/CN102316074A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080033905A1 (en) * | 2006-08-05 | 2008-02-07 | Terry Lee Stokes | System and Method for the Capture and Archival of Electronic Communications |
| CN1997030A (en) * | 2006-12-13 | 2007-07-11 | 武汉虹旭信息技术有限责任公司 | Method for HTTP data recovery |
Non-Patent Citations (1)
| Title |
|---|
| 曾铖，韩桂华，: "基于网络的入侵检测系统分析与设计", 《成都信息工程学院学报》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103618720A (en) * | 2013-11-29 | 2014-03-05 | 华中科技大学 | Method and system for Trojan network communication detecting and evidence obtaining |
| CN104394211A (en) * | 2014-11-21 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Design and implementation method for user behavior analysis system based on Hadoop |
| CN106911637A (en) * | 2015-12-23 | 2017-06-30 | 北京奇虎科技有限公司 | Cyberthreat treating method and apparatus |
| CN106911640A (en) * | 2015-12-23 | 2017-06-30 | 北京奇虎科技有限公司 | Cyberthreat treating method and apparatus |
| CN105491158A (en) * | 2016-01-15 | 2016-04-13 | 成都科来软件有限公司 | HTTP content reduction method and HTTP content reduction system based on network data flow |
| CN105491158B (en) * | 2016-01-15 | 2018-12-25 | 成都科来软件有限公司 | A kind of HTTP content reduction method and system based on network data flow |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220046052A1 (en) | Automatic creation and updating of event group summaries | |
| US7548848B1 (en) | Method and apparatus for semantic processing engine | |
| US8751787B2 (en) | Method and device for integrating multiple threat security services | |
| CN112039904A (en) | Network traffic analysis and file extraction system and method | |
| KR20080037909A (en) | A method and a device for network-based internet worm detection with the vulnerability analysis and attack modeling | |
| CN105103496A (en) | System and method for extracting and preserving metadata for analyzing network communications | |
| CN102316074A (en) | HTTP (hyper text transfer protocol) multithreading restoration method based on libnids | |
| CN105743732B (en) | Method and system for recording transmission path and distribution condition of local area network files | |
| EP3340097B1 (en) | Analysis device, analysis method, and analysis program | |
| JP5752642B2 (en) | Monitoring device and monitoring method | |
| WO2023241202A1 (en) | Supervision engine for network assets | |
| CN102004877B (en) | Method for monitoring source of computer virus | |
| CN111641589A (en) | Advanced sustainable threat detection method, system, computer and storage medium | |
| Tazaki et al. | MATATABI: multi-layer threat analysis platform with Hadoop | |
| Hurley et al. | ITACA: Flexible, scalable network analysis | |
| US8149841B2 (en) | System and method for end-user custom parsing definitions | |
| CN111131180B (en) | Distributed deployed HTTP POST (hyper text transport protocol) interception method in large-scale cloud environment | |
| CN112527772A (en) | Graph database auditing method and auditing equipment | |
| CN109981529B (en) | Message acquisition method, device, system and computer storage medium | |
| CN114244610A (en) | File transmission method and device, network security equipment and storage medium | |
| CN114553546A (en) | Message capturing method and device based on network application | |
| JP5925287B1 (en) | Information processing apparatus, method, and program | |
| CN108055276B (en) | Intrusion detection real-time analysis system for big data application platform | |
| KR102640648B1 (en) | Corporate asset management system through specialized database construction | |
| Patel et al. | Analyzing network traffic data using Hive queries |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20120111 |