KR102125966B1 - System for collecting traffic and feature of TOR network using private network and virtual machine - Google Patents

Abstract

The Tor network traffic collection system according to the present invention includes: a computing device installed on a private network, functioning as multiple Tor clients, and including virtual machines each having a private IP address and functioning as a Tor client; and a traffic collection server for traffic collection between the virtual machines and a Tor network. Each of the virtual machines includes: a service access unit accessing a hidden service server on the Tor network by executing and controlling a Tor browser and generating traffic through the Tor network by downloading a webpage file provided by the hidden service server; a pcap file generation unit generating a pcap file including the generated traffic; a filtering unit having a list of the IP addresses of the entry nodes of the Tor network, comparing the entry node IP addresses with the destination IP address of the traffic in the pcap file, extracting the traffic in which the destination IP address includes the entry node IP address, and generating a pcap file including only the extracted traffic; and a file transmission unit transmitting the pcap file generated by the filtering unit to the traffic collection server. The traffic collection server controls the virtual machines, converts the pcap file from the virtual machines into a csv file, and stores the csv file in a database.

Description

System for collecting traffic and feature of TOR network using private network and virtual machine

The present invention relates to a system for collecting traffic and feature points of a Tor network using a private network and a virtual machine.

Anonymous networks are used for illegal transactions such as drugs and weapons, and their utilization rate is rapidly increasing because the identity of users or service providers of the network that uses the service is hidden and unknown. Channels are a problem.

A typical example of an anonymous network is the TOR network. TOR stands for'The Oinion Routing'. The Tor network is a network that guarantees anonymity between the sender and the receiver by making it impossible to analyze traffic or track IP addresses online. It is composed, and the more users (relay computers), like BitTorrent's P2P program, the faster and faster anonymity is.

1 shows a schematic structure of a Tor network. In the Tor network, the client does not communicate directly with the destination, but communicates through multiple nodes running the Tor router in the middle, where the packets are encrypted through the keys of each node. That is, the client communicates with a hidden service server that provides a hidden web service through three steps of an entry node, a middle node, and an exit node. The entry node is the first node connected to the client, and the middle node is a node that relays information between the entry node and the exit node, and serves to maintain the anonymity of the information. The exit node is the last node, the exit of the network, and the hidden service server recognizes the exit node as a source.

Traffic in the Tor network is not delivered directly to the final destination, and because each Tor's relay server receives encrypted data, it cannot decrypt in the middle, and the origin and final destination of the packet are unknown, and the next relay to move Only the address of the server is known. In addition, packets are encrypted and transmitted as many as the number of relay servers to be moved, and each time they pass through the relay server, they are decrypted one by one and then moved to the next relay server, and information about the path of the packet is periodically deleted. Is taking. As the encrypted data is decrypted at the exit relay computer immediately preceding the final destination, protocol contents such as HTTP and FTP are displayed and delivered to the final destination. That is, packing interception can be performed between the exit relay computer and the final destination website. Through the above process, the Tor network ensures encryption of contents (packets), hiding of the header (final destination), and hiding of the user's current location, so that it cannot easily track or analyze the entire communication through any node.

In recent years, the possibility of copyright infringement has increased as the number of illegal copy distribution sites using streaming services, etc. in anonymous networks, such as the Tor network, and the increase in illegal content transactions rapidly increases. Accordingly, there is an increasing demand for monitoring and responding to infringement of illegal content distribution. In order to effectively respond to this, a technology capable of monitoring and realizing users or service providers in the Tor network is required. To this end, it is necessary to analyze the traffic of the Tor network, and for traffic analysis, it is necessary to effectively collect a lot of traffic from the Tor network. do.

Technical problem to be achieved by the present invention is to provide a traffic collection system that can effectively collect a lot of traffic in the Tor network.

In addition, a technical problem to be achieved by the present invention is to provide a traffic collection system capable of collecting only traffic related to the Tor network.

In addition, a technical problem to be achieved by the present invention is to provide a traffic collection system capable of collecting the URL of the hidden web service provided in the Tor network and the feature point information of the corresponding web page.

The problems to be solved of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

The traffic collection system of the Tor network according to the present invention for solving the above technical problem is a computing device installed in a private network and serving as a plurality of Tor clients, each having a private IP address and serving as a Tor client. A computing device including virtual machines; And a traffic collection server that serves to collect traffic between the plurality of virtual machines and the Tor network, each of the virtual machines running and controlling a Tor browser to access a hidden service server on the Tor network, and A service connection unit that downloads a web page file provided by the hidden service server and generates traffic through the Tor network; A pcap file generation unit generating a pcap file including the generated traffic; A list of IP addresses of the entry nodes of the Tor network is maintained, and the destination IP address of the traffic included in the pcap file is compared with the IP address of the entry node, so that the destination IP address is the address of the entry node. A filtering unit for extracting traffic including an IP address and generating a pcap file including only the extracted traffic; And a file transmission unit that transmits the pcap file generated by the filtering unit to the traffic collection server, and the traffic collection server controls the virtual machines and converts the pcap files received from the virtual machines into csv files. Characterized in that it is stored in the database.

The service connection unit may download the web page file using a virtual display.

The service connection unit, the pcap file generation unit, the filtering unit and the file transmission unit are included in the user space of the virtual machine, and are included in the kernel space of the virtual machine as a packet capture unit, which captures packets of the generated traffic. And a packet capture unit for temporarily storing the kernel space in the kernel buffer and copying the temporarily stored packet to the user space dumpcap, and the pcap file generating unit is a packet copied to the dumpcap. You can create a pcap file from.

The service connection unit automatically sets the visitor history to be stored when the Tor browser is executed by storing a file in an xpi format that executes a function of storing the visit history in the Tor browser profile directory. The Tor browser can store the records of the sites you have accessed in a file in the profile directory.

Each of the virtual machines retrieves the URL of the site accessed by the Tor browser from a file in the profile directory, analyzes the web page file corresponding to the URL, and the number of all domains as a feature point for the web page file, Same number of domains, number of third party links, number of tag paths, number of tags, number of unique tag paths, minimum tag path, maximum tag path, total tag path, maximum number of tag paths, minimum number of tag paths, middle value tag Further comprising a feature point extracting unit for extracting a value of a feature point including the number of paths, the number of all characters, the number of files, and the html document size, and generating a feature point file including the URL and the value of the feature point, and transmitting the file The unit may transmit the feature point file together with the pcap file generated by the filtering unit to the traffic collection server, and the traffic collection server may store the feature point file in the database.

The feature point extracting unit analyzes the webpage file corresponding to the URL only when the filtering unit compares the destination IP address and the IP address of the entry node, and the destination IP address includes the IP address of the entry node. A feature point value may be extracted, and a feature point file including the URL and the value of the feature point may be generated.

According to the present invention described above, it is possible to provide a traffic collection system capable of effectively collecting a large amount of traffic in a Tor network.

In addition, according to the present invention, it is possible to provide a traffic collection system capable of collecting only traffic related to the Tor network.

Further, according to the present invention, it is possible to provide a traffic collection system that can be used for machine learning by collecting the URL of the hidden web service provided in the Tor network and the feature point information of the corresponding web page.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 shows a schematic structure of a Tor network.
2 shows a configuration of a traffic collection system of a Tor network according to an embodiment of the present invention.
3 shows a configuration of the virtual machine 100 of the computing device 10 according to an embodiment of the present invention.
4 shows a table structure for storing traffic collection data in the database 30.
5 shows feature points extracted from the feature point extracting unit 127 and a description of each feature point.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. In the following description and the accompanying drawings, elements that are substantially the same are denoted by the same reference numerals, and redundant description will be omitted. In addition, in the description of the present invention, when it is determined that detailed descriptions of related known functions or configurations may unnecessarily obscure the subject matter of the present invention, detailed descriptions thereof will be omitted.

2 shows a configuration of a traffic collection system of a Tor network according to an embodiment of the present invention.

The traffic collection system according to this embodiment is installed in a private network, and includes a computing device 10, a traffic collection server 20, a database 30, a switching hub 40, a gateway 50, a PC 60, and the like. It can contain. It is desirable to build a separate network for collecting Tor network traffic in a private network so that traffic is not blocked by a firewall, IDS (Intrusion Detection System), or IPS (Intrusion Prevention System).

The computing device 10 serves as a number of Tor clients. To this end, the computing device 10 includes a plurality of (for example, about 20) virtual machines 100 each serving as a Tor client, and each of the virtual machines 100 is assigned and has a fixed private IP address.

The PC 60 also functions as a Thor client, has a private IP address, and has the same operating system, configuration, and operation as the virtual machine 100 to be described later. Having the PC 60 serving as a Thor client separately from the virtual machine 100 is intended to supplement a problem that may occur when using the virtual machine. Several PCs 60 may also be installed.

The traffic collection server 20 serves to collect traffic between the computing device 10 and the Tor network, that is, traffic between the virtual machines 100 and the Tor network, through the virtual machines 100 and the PC 60 The collected traffic data and web page feature point data are stored in the database 30. In addition, the traffic collection server 20 controls the operations of the virtual machines 100 of the computing device 10 and the PC 60. For example, the traffic collection server 20 may issue commands such as starting and ending traffic collection to the virtual machines 100 and the PC 60.

The switching hub 40 is connected to the computing device 10, the PC 60, the traffic collection server 20, and the gateway 50 of the private network to serve as a traffic switching. Accordingly, the virtual machines 100 of the computing device 10 and the PC 60 are connected to the Tor network through the switching hub 40 and the gateway 50.

3 shows a configuration of the virtual machine 100 of the computing device 10 according to an embodiment of the present invention.

The virtual machine 100 includes a service connection unit 110, a pcap file generation unit 120, a filtering unit 125, a feature point extraction unit 127, a file transmission unit 130, a packet capture unit 140, and a kernel buffer ( 150), a protocol stack 160, a dump cap 170, a virtual machine network driver 180, and the like.

The service connection unit 110, the pcap file generation unit 120, the filtering unit 125, the feature point extraction unit 127, the file transmission unit 130, and the dump cap 170 are included in the user space of the virtual machine 100. The packet capture unit 140, the kernel buffer 150, and the protocol stack 160 may be included in the kernel space of the virtual machine 100.

The service connection unit 110 accesses the Thor network through the protocol stack 160 and the virtual machine network driver 180. The service connection unit 110 accesses a hidden service server 200 that provides a hidden web service through an entry node, and downloads a web page file provided by the hidden service server 200 to generate traffic through the Tor network. . The service connection unit 110 may execute and control a Tor Browser to access the Hidden Service Server 200 on the Tor network and download a web page file provided by the Hidden Service Server 200. In addition, the service connection unit 110 may download a webpage file using a virtual display without actually opening a browser window when executing the Tor browser.

The Tor Browser is a type of browser such as Chrome, Firefox, and Explorer that uses the Tor Network to access general websites as well as hidden web services provided by the Tor Network. The Tor Browser is based on Firefox.

Selenium is primarily a testing framework for web applications, supporting a variety of features for automated testing. Selenium provides various web drivers to execute and control browsers such as Firefox, Chrome, and IE based on script. Up to version 2.53 of Selenium, you could run and operate the Firefox browser without installing a separate driver. However, Selenium 2.53 or later can install the geckodriver separately and enter the path of the geckodriver to launch the Firefox browser. The geckodriver is a web browser engine developed by the Mozilla Foundation. Typical examples of this are Firefox and Thunderbird. Selenium-based code sends requests to geckodriver using W3C Webdriver protocol, and geckodriver converts to Marionette protocol and sends to Firefox, Firerox browser understands commands and performs actions.

Since there is no separate web driver for executing and operating the Tor browser in Selenium, the service connection unit 110 uses a method of running Firefox. When you run Firefox with Selenium, you change Firefox's settings to Tor Browser's settings to run Firefox, so Selenium manipulates Firefox, but you can actually operate Tor Browser. A Tor browser is installed in each virtual machine 100, and a geckodriver for operating Firefox with Selenium is installed in the Tor browser directory.

Meanwhile, Tor Browser basically does not leave any traces such as visit history. When the Tor browser is launched, the default settings can be set in the profile directory just like Firefox. Therefore, the service connection unit 110 uses an extension of Firefox to store a file of xpi format that executes a function of storing a visit record in a profile directory of the Tor browser, thereby executing the Tor browser. It is automatically configured to always save your visit history, so the Tor Browser saves the history of the site you accessed in the sqlite file in your profile directory.

When traffic collection starts, the virtual machine network driver 180 transmits a copy of the packet to the packet capture unit 140 in the kernel space whenever a packet of network traffic is received. The packet capture unit 140 captures packets from the virtual machine network driver 180 and temporarily stores them in the kernel buffer 150. Then, when the packet capture unit 140 copies the temporarily stored packet in the kernel buffer 150 to the dump cap 170 in the user space through context switching, the pcap file generation unit 120 copies the dump cap 170 Creates a pcap file (libpcap) from the packet.

Meanwhile, the pcap file generated as described above includes not only traffic through the Tor network, but also traffic not related to the Tor network, such as traffic by a general web service or communication with the traffic collection server 20. Therefore, it is necessary to collect only traffic related to the Tor network. To this end, the filtering unit 125 holds a list of IP addresses of the entry nodes of the Tor network, and holds the destination IP addresses of the traffic included in the pcap file generated by the pcap file generating unit 120 The IP address of the entry node being compared is compared to extract the traffic whose destination IP address is the IP address of the entry node, and a pcap file including only the extracted traffic is generated. Information about the entry nodes can be found by crawling, for example, from the Tor Network Status website ().

The file transfer unit 130 transmits the pcap file generated by the filtering unit 125 to the traffic collection server 20 through the protocol stack 160 and the virtual machine network drive 180. To this end, the file transfer unit 130 may use a Common Internet File System (CIFS) protocol. The traffic collection server 20 converts the pcap file received from the virtual machines 100 into a csv file and stores it in the database 30.

Referring again to FIG. 2, the traffic collection server 20 may be connected to the mirroring port of the switching hub 40. Then, the traffic collection server 20 collects pcap files received from the virtual machines 100 and the PC 60 of the computing device 10, and through the mirroring port of the switching hub 40, the switching hub 40 ) To collect all traffic passing through it. The traffic collection server 20 may compare the traffic received from the virtual machines 100 and the PC 60 with traffic collected through the mirroring port of the switching hub 40 to check whether or not a packet is lost.

4 shows a table structure for storing traffic collection data in the database 30. The table of the database 30 includes a traffic data table representing traffic information based on a virtual machine and a hidden web service, a packet table representing traffic time, MAC, IP address, port, and packet length based on the hidden web service, and a Tor A hidden service table indicating the address of a hidden web service that distributes illegal content on the network, a category table indicating the type of hidden web service, a client table indicating information of virtual machines, an entry node table indicating information of entry nodes in the Tor network, And an exit node table indicating information on exit nodes of the Tor network.

Referring back to FIG. 3, the feature point extraction unit 127 queries the URL of the site accessed by the Tor browser from the sqlite file in the Tor browser's profile directory, and analyzes the web page file (html file) corresponding to the viewed URL. By extracting the values of the feature points for the web page file.

5 shows feature points extracted from the feature point extracting unit 127 and a description of each feature point. As shown in the figure, the feature point of the web page file is the number of all domains, the same number of domains, the number of third party links, the number of tag paths, the number of tags, the number of unique tag paths, the minimum tag path, the maximum tag path, and the tag. It can include a total of 15 items, such as the sum of paths, the maximum number of tag paths, the minimum number of tag paths, the number of median tag paths, the number of all characters, the number of files, and the html document size.

The feature point extracting unit 127 generates a feature point file including a client IP, a feature point extraction target URL, a feature point collection time, a feature point collection end time, and values of feature points. The value for each feature point may be expressed as a number, and may be included in the feature point file in the form of a list according to the order of the items.

The feature point extracting unit 127 extracts the feature points from the web page to generate the feature point files, as a result of comparing the destination IP address of the traffic included in the pcap file with the IP address of the entry node in the above-described filtering unit 125 When the destination IP address includes the IP address of the entry node, that is, it is performed only when traffic through the Tor network occurs, it can be extracted only for the URL accessed through the Tor network.

The file transfer unit 130 transmits the feature point file together with the pcap file generated by the filtering unit 125 to the traffic collection server 20, and the traffic collection server 20 stores the feature point file in the database 30.

According to the present invention described above, it is possible to effectively collect a lot of traffic in the Tor network using a plurality of virtual machines 100, and only traffic related to the Tor network.

In addition, URLs of hidden web services provided by the Tor network and feature point information of corresponding web pages may be collected. Big data of the URL of the hidden web service and the feature point information of the corresponding web page is provided as learning data for machine learning, and is used to find and classify unknown hidden web services using machine learning Can be.

Meanwhile, the above-described embodiments of the present invention can be written in a program executable on a computer and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium. The computer-readable recording medium includes a storage medium such as a magnetic storage medium (eg, ROM, floppy disk, hard disk, etc.), an optical reading medium (eg, CD-ROM, DVD, etc.).

Embodiments of the present invention may be represented by functional block configurations and various processing steps. These functional blocks can be implemented with various numbers of hardware or/and software configurations that perform specific functions. For example, an embodiment may be implemented in an integrated circuit configuration such as memory, processing, logic, look-up table, etc., capable of executing various functions by control of one or more microprocessors or other control devices. You can hire them. Similar to the components of the present invention that can be executed with software programming or software components, embodiments include C, C++, including various algorithms implemented with a combination of data structures, processes, routines or other programming components. , Can be implemented in programming or scripting languages such as Java, assembler, etc. Functional aspects can be implemented with algorithms running on one or more processors. In addition, embodiments may employ conventional techniques for electronic environment setting, signal processing, and/or data processing. Terms such as "mechanism", "element", "means", and "configuration" can be used widely and are not limited to mechanical and physical configurations. The term may include the meaning of a series of routines of software in connection with a processor or the like.

The specific implementations described in the embodiments are examples, and do not limit the scope of the embodiments in any way. For brevity of the specification, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection or connection members of the lines between the components shown in the drawings are illustrative examples of functional connections and/or physical or circuit connections. In the actual device, alternative or additional various functional connections, physical It can be represented as a connection, or circuit connections. In addition, unless specifically mentioned, such as "essential", "important", etc., it may not be a necessary component for the application of the present invention.

So far, the present invention has been focused on the preferred embodiments. Those skilled in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in terms of explanation, not limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent range should be interpreted as being included in the present invention.

Claims (6)

In the traffic collection system of the Tor network, the traffic collection system is installed in a private network, and the traffic collection system comprises:
A computing device serving as a plurality of Tor clients, comprising: a computing device each having a private IP address and including a plurality of virtual machines acting as a Tor client; And
And a traffic collection server that serves to collect traffic between the plurality of virtual machines and the Tor network,
Each of the virtual machines,
A service connection unit that executes and controls a Tor browser to access a hidden service server on the Tor network and downloads a web page file provided by the Hidden service server to generate traffic through the Tor network;
A pcap file generation unit generating a pcap file including the generated traffic;
A list of IP addresses of the entry nodes of the Tor network is maintained, and the destination IP address of the traffic included in the pcap file is compared with the IP address of the entry node, so that the destination IP address is the address of the entry node. A filtering unit for extracting traffic including an IP address and generating a pcap file including only the extracted traffic;
And a file transmission unit transmitting the pcap file generated by the filtering unit to the traffic collection server,
The traffic collection server controls the virtual machines, converts the pcap file received from the virtual machines into a csv file and stores it in a database,
The service access unit stores an xpi format file for executing a function of storing a visit record in a profile directory of the tor browser and automatically sets to store a visit record when the tor browser is executed, so that the tor browser is accessed. The site records are stored in a file in the profile directory.
Each of the virtual machines,
The URL of the site accessed by the Tor browser is retrieved from the file in the profile directory, and the web page file corresponding to the URL is analyzed, and as a characteristic point for the web page file, the number of all domains, the number of third party links, and tags Number of paths, number of tags, number of unique tag paths, minimum value of tag paths, maximum value of tag paths, sum of tag paths, maximum number of tag paths, minimum number of tag paths, median number of tag paths, number of all characters, number of files, and Further comprising a feature point extraction unit for extracting the value of the feature point including the html document size, and generating a feature point file containing the URL and the value of the feature point,
The file transmission unit transmits the feature point file together with the pcap file generated by the filtering unit to the traffic collection server,
And the traffic collection server stores the feature point file in the database. According to claim 1,
The service connection unit, the traffic collection system of the Tor network, characterized in that for downloading the web page file using a virtual display. According to claim 1,
The service connection unit, the pcap file generation unit, the filtering unit and the file transmission unit is included in the user space of the virtual machine,
As a packet capture part included in the kernel space of the virtual machine, the generated traffic packet is captured and temporarily stored in a kernel buffer of the kernel space, and a packet temporarily stored in the kernel buffer is dump cap of the user space Further comprising a packet capture unit to copy,
The pcap file generating unit generates a pcap file from packets copied to the dump cap, the traffic collection system of the Tor network. delete delete According to claim 1,
The feature point extracting unit analyzes the webpage file corresponding to the URL only when the filtering unit compares the destination IP address and the IP address of the entry node, and the destination IP address includes the IP address of the entry node. The traffic collection system of the Tor network, characterized by extracting a value of a feature point and generating a feature point file including the URL and the value of the feature point.

Images