KR102043508B1 - Apparatus, method for analysing tor service based on distributed processing, and computer readable recording medium - Google Patents

Abstract

A control device is provided for communicating with a tor service providing device through a communication network to analyze a tor service. Tor service analysis control device according to the present invention comprises a communication unit configured to communicate with the tor service providing device through the communication network; A storage unit configured to receive and store, from the plurality of external collection devices, information that the plurality of external collection devices crawled the tor service providing device through the communication unit; And a controller configured to search for a domain of the tor service, generate address information of the tor service providing device, and transmit at least a portion of the address information to the plurality of external collection devices.

Description

Methods, devices, and computer-readable recording media for Thor service analysis using distributed processing {APPARATUS, METHOD FOR ANALYSING TOR SERVICE BASED ON DISTRIBUTED PROCESSING, AND COMPUTER READABLE RECORDING MEDIUM}

The present invention relates to a method of searching for hidden services, and more particularly, to a method and apparatus for searching for Thor hidden services using distributed processing.

Hidden services are websites and servers that are configured to receive inbound connections through Tor using Tor programs that support anonymous communication. Hidden services do not reveal the server's IP address, but instead use a special address called onion address, which consists of 16 chaotic strings of alphanumeric characters. Hidden services do not reveal the server's IP address, and have high anonymity by using the Tor network, which uses anonymous communication. Thanks to this anonymity, hidden services have been used as Internet spaces for free movement by state-run human rights activists and journalists seeking freedom.

Recently, however, hidden services have been used as a space for trading goods that are legally prohibited in the country, such as firearms, drugs, and child pornography, by exploiting their anonymity. In the previous study, the contents of hidden services that received a lot of requests were analyzed. As a result, adult contents, drugs, and weapons transactions were found as popular contents, and many studies were conducted to analyze and analyze the illegal services of Tor. come.

Tor's analysis requires special methods, such as using the Tor browser or using socket communication with a Proxy program. Tor's special structure for anonymity causes technical problems in analyzing services through common web crawlers, such as behaving differently at the TCP level and using socket connections. In the early days of Tor's emergence, the analysis of hidden services was conducted by searching in the Tor network band by analyzing Tor's packets and protocols. In-band search in this manner consumes a lot of resources and time to obtain information about Tor.

Since 2008, the tor2web project has allowed some of Tor's hidden services to be accessed from the public network, and tor2web has increased the number of hidden services accessible from the public network. Based on these results, a new out-of-band search is proposed for the Tor network. Out-of-band discovery attempts to discover Tor in a normal network outside of the Tor network band. The out-of-band discovery method discovers and analyzes Tor's hidden services by collecting the addresses of hidden services that are exposed through search in a general network. As access to the Tor network is also possible in the general network, out-of-band discovery can provide information about the Tor network with less resources and time than previous studies.

However, out-of-band browsing using these web page collectors (Scrapy, Heritrix, Apache Nutch, etc.) requires access to hidden service pages using Tor, through the configuration of separate proxy programs such as Privoxy and Polipo, and through Tor programs. There is a difficulty. Depending on the type of proxy program and system used, the crawling method is different and a lot of resources are required to configure the environment for using the Collector.

Due to these technical limitations, the Tor network takes a lot of time to crawl a page by accessing a single hidden service as it communicates by bypassing the client several times for anonymity, and it is possible to configure a separate proxy program such as Privoxy, Polipo, and Tor. There is a problem that crawling through distributed processing is difficult because it must be accessed through a program.

Republic of Korea Patent Registration No. 10-1469010 Republic of Korea Patent Registration No. 10-1620957

Therefore, there is a need for a method and apparatus for directly accessing a hidden service page using a Tor browser without using a proxy program.

In addition, there is a need for a method and apparatus for dividing a set of hidden service addresses into several bundles and crawling them through multiple computers in a distributed manner.

According to an aspect of the present invention, there is provided a control device for communicating with a tor service providing device through a communication network, to analyze the tor service. The apparatus includes a communication unit configured to communicate with the tor service providing apparatus through the communication network; A storage unit configured to receive and store, from the plurality of external collection devices, information that the plurality of external collection devices crawled the tor service providing device through the communication unit; And a controller configured to search for a domain of the tor service, generate address information of the tor service providing device, and transmit at least a portion of the address information to the plurality of external collection devices.

In one embodiment, the control unit is configured to search the top-level domain of the domain of the Tor service through an external search engine, collect a string of the top-level domain from the search results to search the address set, the control unit is the top It can be configured to restore the original hidden service address set by replacing the domain part with .onion.

In one embodiment, the control unit may be configured to obtain a unique hidden service address set by removing a path value for a URL regarding a lower page of the same address and removing duplicate information from the set of addresses.

In one embodiment, when the control unit transmits at least a portion of the address information to the plurality of external collection devices, the control unit is configured to send more address information to the collection device having a large amount of information crawled and transmitted among the plurality of external collection devices. Can be configured to transmit.

In one embodiment, when the control unit transmits at least some of the address information to the plurality of external collection devices, the control unit transmits the address information to each of the plurality of external collection devices so as not to duplicate the number of the address information May be configured to distribute the address information so that one or more differences do not exist.

According to another aspect of the present invention, there is provided a method for communicating with a tor service providing apparatus through a communication network, and analyzing the tor service in a control apparatus. The method includes collecting an address of the tor service providing device to create a list of top-level domains of the collected addresses, replacing the top-level domain list with a .onion except for a common value, and replacing the original hidden service address set. Restoring to a URL, removing a path value for a URL of a lower page of the same address, and removing duplicated information from the address set to obtain a unique hidden service address set, wherein the obtained unique hidden service address set − The number of addresses in the address set is N-Distributing N / k by k to a plurality of external plurality of collecting devices-Distributing the address information to each of the plurality of external collecting devices without overlapping-, The The external collection device directly accesses the hidden service page, collects the information of the page, The information collected may include the step, and analyzing the state of a unique hidden service by analyzing the collected information to be transmitted to the control device.

According to another aspect of the present invention, there is provided a computer readable recording medium containing one or more instructions, wherein the one or more instructions cause the computer to perform the above-described method when executed by the computer.

According to the present disclosure, it is possible to provide a simplified collection method and apparatus for Tor web pages that directly access hidden service pages using Python modules tbSelenium and Tor browser without using a proxy program.

According to the present disclosure, it is also possible to provide a method and apparatus for dividing a set of hidden service addresses into several bundles and crawling them through a plurality of computers in a distributed manner to improve the speed of discovery and collection.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.
FIG. 2 is a diagram schematically showing an exemplary configuration of the control device of FIG. 1.
3 is a schematic operational flow diagram illustrating operations performed on a control device for dividing a set of hidden service addresses into several bundles and crawling them in a distributed manner through several computers according to an embodiment of the present invention.
FIG. 4 is a diagram illustrating an exemplary method of crawling a hidden service page by collecting an address of a hidden service according to the operation described with reference to FIG. 3.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following, when it is determined that there is a risk of unnecessarily obscuring the gist of the present invention, a detailed description of the known functions and configurations will be omitted. In addition, it should be understood that the following description only relates to one embodiment of the present invention, but the present invention is not limited thereto.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. For example, a component expressed in the singular should be understood as a concept including a plurality of components unless the context clearly indicates the singular. In addition, in the specification of the present invention, terms such as 'comprise' or 'have' are merely intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist. The use of the term is not intended to exclude the presence or addition of one or more other features or numbers, steps, actions, components, parts or combinations thereof.

In the embodiments described herein, 'block' or 'unit' refers to a functional part that performs at least one function or operation, and may be implemented in hardware or software or a combination of hardware and software. In addition, a plurality of 'blocks' or 'units' may be integrated into at least one software module and implemented as at least one processor, except for 'blocks' or 'units' that need to be implemented with specific hardware. .

In addition, all terms used herein, including technical or scientific terms, unless otherwise defined, have the same meaning as commonly understood by one of ordinary skill in the art. It is to be understood that generally defined terms used should be construed as having a meaning consistent with the contextual meaning of the related art, and shall not be construed as being excessively limited or extended unless expressly defined otherwise in the specification of the present invention. You should know

Hereinafter, with reference to the accompanying drawings, it will be described in detail an embodiment of the present invention.

1 is a diagram schematically illustrating a configuration of a network environment 100 according to an embodiment of the present invention. As shown, the network environment 100 includes a control device 110, a communication network (not shown), and one or more collection devices 120a-120n. 1 is an example for describing the invention and the number of control devices or the number of collection devices is not limited as shown in FIG. 1.

According to an embodiment of the present invention, the control device 110 communicates with the plurality of collecting devices 120a-120n through a communication network according to a predetermined purpose, and provides commands, codes, files, contents, services, and the like. A computer device or a plurality of computer devices.

For example, the control device 110 may provide a file for installing the program to the collection device (120a-120n) connected through a communication network. In this case, the collection apparatuses 120a-120n may install an application using a file provided from the control apparatus 110. In addition, the control apparatus 110 may be connected to the control apparatus 110 under the control of an operating system (OS) and the at least one program included in the collection apparatuses 120a-120n to receive a service or content provided by the control apparatus 110. Can be. For example, the collection devices 120a-120n transmit a service request message to the control device 110 through a communication network under the control of an application, and the control device 110 collects a code corresponding to the service request message. 120a-120n, the collection apparatuses 120a-120n may perform a command according to a code according to the control of an application, and provide a result of execution to the controller 110.

According to an embodiment of the present invention, the communication network may be any wired or wireless communication network, and may include, for example, a WiFi network, a LAN network, a WAN network, an Internet network, and the like, but the present invention is not limited thereto.

According to an embodiment of the present invention, the plurality of collection apparatuses 120a-120n communicate with a web page providing server (not shown) through a communication network according to a predetermined purpose, and include commands, codes, files, contents, It may be implemented as a computer device or a plurality of computer devices that provide services. According to an embodiment of the present invention, the plurality of collecting devices 120a-120n may communicate with the control device 110 through a communication network and transmit and receive necessary information. According to an embodiment of the present invention, each of the plurality of collection apparatuses 120a-120n may access an external web page through a communication network to retrieve or collect information of the web page. According to an embodiment of the present invention, each of the plurality of collection apparatuses 120a-120n collects Tor web pages by directly executing Tor's dedicated browser through a web test automation program such as Selenium, without a proxy program and a system setting process. can do. Dedicated browsers for Selenium and Tor can be installed and run on any OS, allowing the same Collector environment to be configured on both device and system environments.

2 is a view schematically showing an exemplary configuration of the control device of FIG. 1 in accordance with an embodiment of the present invention.

As shown, the control device 110 may include a control unit 202, a storage unit 204, and a communication unit 206.

According to one embodiment of the invention, the control unit 202 may be configured to be connected to the storage unit 204 and the communication unit 206 to execute various computer programs and control the operation thereof to control them. The controller 202 may be configured to receive data from one component and format the data in a format suitable for use by the other component. The control unit 202 may be connected to the storage unit 204 in a communicative manner. In various embodiments, the control unit 202 may be integrally integrated with other components such as, for example, the communication unit 206.

The control unit 202 may receive the information of the web page through the communication unit 206, and analyze the state of the web page by analyzing the received information. According to an embodiment of the present invention, the control unit 202 may detect the web page state by using the web page information stored in the storage unit 204, and classify and analyze the web page according to the purpose.

The storage unit 204 can store and update web page information and the like transmitted from the plurality of collection apparatuses 120a-120n through the communication network and the communication unit 206 of FIG. 1.

The storage unit 204 may also store and update the contents of the web page transmitted from an external web page providing server (not shown) through the communication network and the communication unit 206 of FIG. 1.

The storage unit 204 may include a flash memory type, a hard disk type, a multimedia card (MMC), a memory of a card type (eg, a Secure Digital (SD) card or an XD). (eXtream Digital) cards, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM) The magnetic memory device may be implemented as a storage medium of any one of a magnetic memory, a magnetic disk, and an optical disk. However, one of ordinary skill in the art will recognize that the implementation of the storage unit 204 is not limited thereto.

According to an embodiment of the present invention, the communication unit 206 enables the control device 110 to communicate with the plurality of collecting devices 120a-120n through a communication network. According to an embodiment of the present invention, the communication unit 206 may transmit the information processed by the control unit 104 to the plurality of collecting devices 120a-120n through the communication network 104 according to a predetermined protocol. According to an embodiment of the present invention, the communication unit 206 may appropriately encode or decode the transmitted and received information according to the protocol of the communication network of FIG. 1. According to one embodiment of the present invention, the communication unit 206 can receive various types of information from the plurality of collection apparatuses 120a-120n and perform appropriate processing.

The embodiments described above are programmable in terms of hardware, application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), and programmable. At least one of Logic Devices (PLDs), Field-Programmable Gate Arrays (FPGAs), processors, controllers, micro-controllers, and microprocessors It can be implemented using one.

Embodiments that include a procedure, step, or function may be implemented as firmware / software modules executable on a hardware platform to perform at least one function or operation. In this case, the firmware / software module may be implemented by a software application written in a suitable program language.

FIG. 3 is a schematic operation flow diagram illustrating operations performed on a control device for dividing a set of hidden service addresses into several bundles and crawling them in a distributed manner through several computers according to an embodiment of the present invention.

Starting from collecting the address list of the Tor web page (S310). In one embodiment, the control device 110 may collect the address of the hidden service using the out-of-band search method as an address collector using the out-of-band search (S312). For example, the top-level domains of the hidden services issued by tor2web are searched by a search engine of a general network, and items corresponding to the hidden service address string are collected in the result page (S314). For example, a search is performed on the 12 top-level domains identified in Table 1, and the search engine may use three of Google, Bing, and DuckDuckGo. Meanwhile, the controller 110 may optionally receive a list of addresses of Tor web pages from an external server.

Table 1: List of top-level domains issued through tor2web

In one embodiment, the full set of addresses obtained through the address collector is tor2web, which is the set of addresses whose top level domain has been changed to one of the values in Table 1. The address sets have 16 random strings in common with the original hidden service address. To crawl these addresses using Tor, replace the top-level domain part except for common values with .onion and restore the original hidden service address set (S316). In this process, URLs related to lower pages of the same address are removed to treat the directory as the same page as the top page (S318). Subsequent removal of duplicate addresses from the address set yields a unique set of hidden service addresses.

Next, the control device 110 divides the collected address list and provides the collected device 120 to the collecting device 120 and instructs the collecting device 120 to crawl the hidden service page (S320). In one embodiment, the control unit 110 provides a plurality of collection lists of addresses collected in the plurality of collecting devices 120 and by causing the plurality of collecting devices (120a-120n) to crawl in a distributed manner The collection time can be reduced (S322). In general, the Tor network takes a lot of time to crawl a page by accessing a single hidden service as it communicates by bypassing it several times for client anonymity. In the experiment of the present invention, access time of up to 25 seconds was required for a normally hidden live service. Because of this, investigating Tor's hidden services through a single computer has more overhead than normal web page crawls, which reduces the collection time by allowing crawling in a distributed manner.

The collection device 120 crawls a hidden service page crawler directly accesses the hidden service using the onion address of the collected hidden service address set and crawls the corresponding page (S324). In general, a method of accessing a hidden service page using Tor should be accessed through the setting of a separate proxy program such as Privoxy and Polipo, and through the Tor program. However, in one embodiment of the present invention, the collection device 120 may use the proxy. You can access hidden service pages directly using the Python modules tbSelenium and Tor browser without using a program. A crawler implemented with tbSelenium can directly access Tor service and save the page by executing Tor browser directly.

The collecting device 120 requests the Http status code for the service address to be crawled and classifies the result according to the received status code value (S326). Since the entire address set collected by the address collector using the out-of-band search also includes the address of a service that has already been deleted or cannot be accessed, the collecting device 120 accesses through the Tor browser only for addresses that return Http status code 200. To crawl and save the page (S328).

Finally, the control device 110 may receive the crawled information from the plurality of collection devices 120a-120n and analyze the web page.

FIG. 4 is a diagram illustrating an exemplary method of crawling a hidden service page by collecting an address of a hidden service according to the operation described with reference to FIG. 3. As shown in FIG. 4, the controller can collect the hidden service address and analyze the information of the crawled hidden service page.

In one embodiment of the invention, the hidden service address collector may be composed of two parts as shown in FIG. The first is an address collector using out-of-band search, which collects Tor hidden service addresses through out-of-band search. The second is the hidden service page crawler, which accesses the hidden service page directly. During this process, it checks whether the hidden service with the address survives and collects the page of the live service.

1. Address Collector Using Out-of-Band Search

In one embodiment, the address collector using out-of-band search collects the addresses of hidden services using the out-of-band search method. For example, the top-level domains of hidden services issued by tor2web are searched by a search engine of a general network, and items corresponding to the hidden service address strings are collected on the result page. In one embodiment, the search is performed on the 12 top-level domains identified in Table 1, and the search engine may use three of Google, Bing, and DuckDuckGo.

In one embodiment, the full set of addresses obtained through the address collector is tor2web, which is the set of addresses whose top-level domain has been changed to one of the values in Table 1 above. The address sets have 16 random strings in common with the original hidden service address. To crawl these addresses using Tor, we replace the top-level domain part, except for the common value, with .onion to restore the original set of hidden service addresses. In this process, the URLs for subpages of the same address are stripped of directory path values to be treated as the same page as the top page. Subsequent removal of duplicate addresses from the address set yields a unique set of hidden service addresses.

2. Hidden service page crawler

The hidden service page crawler uses the onion address of the collected hidden service address set to directly access and crawl the hidden service. In general, to access the hidden service page using Tor, you need to access it through the configuration of a separate proxy program such as Privoxy and Polipo, and the Tor program. In one embodiment, instead of using this proxy program, the Python modules tbSelenium and Tor browsers are used to directly access hidden service pages. The crawler implemented by tbSelenium executes Tor browser directly to access hidden services and save the page. In the entire address set collected by the address collector using out-of-band search, there are also addresses of services that are already deleted or not accessible. To classify a hidden service in this state, the crawler requests the Http status code for that address and classifies the result according to the received status code value. The hidden service page crawler attempts to access the Tor browser only for addresses that return an Http status code of 200, which crawls and stores the page.

The Tor network takes as much time to crawl a page by accessing a single hidden service as it communicates by bypassing it several times for client anonymity. In the experiments of this study, access time of up to 25 seconds was required for a normally hidden live service. Because of this, investigating Tor's hidden services on a single computer is more expensive than crawling a typical web page. To reduce overhead, the Tor Hidden Service Address Collector can divide a set of hidden service addresses into several bundles and distribute them crawled across multiple computers.

In the embodiments disclosed herein, the arrangement of the components shown may vary depending on the environment or requirements on which the invention is implemented. For example, some components may be omitted or several components may be integrated and implemented as one. In addition, the arrangement order and connection of some components may be changed.

While the invention and its various functional components have been described in particular embodiments, the invention may be implemented in hardware, software, firmware, middleware, or a combination thereof, and the system, subsystem, components, or sub-configurations thereof. It should be understood that the elements can be utilized. If implemented in software, the elements of the present invention may be instructions / code segments for performing the necessary tasks. The program or code segments may be stored in a machine readable medium, a computer program product, such as a processor readable medium. Machine readable media or processor readable media may include any medium that can store or transmit information in a form readable and executable by a machine (eg, processor, computer, etc.).

Although various embodiments of the present invention have been illustrated and described above, the present invention is not limited to the above-described specific embodiments, and the above-described embodiments deviate from the gist of the present invention as claimed in the appended claims. Without departing from the scope of the present invention pertains to those skilled in the art, various modifications may be made to those skilled in the art, and such modified embodiments should not be understood separately from the technical spirit or scope of the present invention. Accordingly, the technical scope of the present invention should be defined only by the appended claims.

Claims (7)

A control device that analyzes a Thor service by communicating with a Thor service providing device through a communication network,
A communication unit configured to communicate with the tor service providing device through the communication network;
A storage unit configured to receive and store, from the plurality of external collection devices, information that the plurality of external collection devices crawled the tor service providing device through the communication unit; And
And a controller configured to search for a domain of the tor service, generate address information of the tor service providing device, and transmit at least a portion of the address information to the plurality of external collection devices. The method of claim 1,
The control unit is configured to search the top-level domain of the domain of the tor service through an external search engine, collect a string of the top-level domain from a search result, and search an address set.
And the control unit is configured to restore the original hidden service address set by replacing the top-level domain portion with .onion. The method of claim 2,
And the control unit is configured to remove a path value for a URL regarding a lower page of the same address, and to remove a duplicated information from the address set to obtain a unique hidden service address set. The method of claim 1,
When the control unit transmits at least some of the address information to the plurality of external collection devices, the tor service configured to transmit more address information to a collection device having a large amount of information crawled and transmitted among the plurality of external collection devices. Analytical Control Unit. The method of claim 1,
When the control unit transmits at least some of the address information to the plurality of external collection devices, the control unit transmits the address information to each of the plurality of external collection devices so as not to overlap, and the number of the address information is one or more different. And to distribute the address information. As a method of communicating with a service provider for a Thor service through a communication network, the controller analyzes the service to the service,
Collecting an address of the tor service providing device and creating a list of top-level domains of the collected addresses;
Restoring the list of top-level domains by replacing .onion with a top-level domain portion excluding common values, and restoring the original hidden service address set;
Removing a path value for a URL of a subpage of the same address, and removing duplicated information from the set of addresses to obtain a unique set of hidden service addresses;
Distributing the obtained unique hidden service address set, wherein the number of addresses of the address set is N pieces, to N / k pieces of the plurality of external collecting devices, wherein the address information is allocated to each of the plurality of external collecting devices. Distribute without overlapping-,
Causing the external collection device to directly access the hidden service page to collect information of the page and transmit the collected information to the control device; and
Analyzing the state of the unique hidden service by analyzing the collected information
Tor service analysis method comprising a. A computer readable recording medium carrying one or more instructions, wherein the one or more instructions cause the computer to perform the method according to claim 6 when executed by a computer.

Images