WO2013162264A1

WO2013162264A1 - Method and system for collecting objects by using packet mirroring

Info

Publication number: WO2013162264A1
Application number: PCT/KR2013/003477
Authority: WO
Inventors: 송진영
Original assignee: 줌인터넷 주식회사
Priority date: 2012-04-23
Filing date: 2013-04-23
Publication date: 2013-10-31

Abstract

The present invention relates to a method and a system for collecting objects by using packet mirroring. The method for a search system collecting content, which is included in an object that is a search target, comprises: a mirroring step for mirroring a packet which is transreceived between a user terminal and a web server; an identification information extraction step for extracting identification information from the packet which is mirrored in the mirroring step; a content collection step for collecting the content of the object, which is the search target, from the packet that is mirrored in the mirroring step; and a saving step for matching the content that is collected in the content collection step with the content, associating same with the identification information that is extracted in the identification information extraction step, and saving same. According to the present invention, content corresponding to the search target can be collected by using minimal hardware resource.

Description

Object collection method and system using packet mirroring

The present invention relates to a method and a system for collecting objects, and more particularly, to a method and a system for collecting content included in an object to be searched.

Recently, Internet communication technology has been developed, and accordingly, the method of producing and consuming content is changing with the focus on Internet communication technology. Compared to the production and consumption of contents through off-line media, information distribution in the Internet age is not comparable in speed and impact. Search technology is the most important and pivotal technology in the information distribution in the Internet age.

The search technology refers to a technology for searching all the search target objects interspersed in the Internet network as needed to find information included in the search target object.

For example, web page search technology may include link collection technology to obtain a web page link (or uniform resource locator) that is the target of a search, crawling technology to obtain web page contents of collected links, and crawled web page content. It is composed of indexing technology that processes the data into a form suitable for search, and search engine technology that provides web page results related to a search word required by a user.

According to this, the conventional search technology secures a link of a web page to be searched, and then acquires the content of the web page based on the link. At this time, in order to obtain the contents of the web page, the crawler (Crawler) continuously collects and processes the content of the search target web page, there is a problem that not only the hardware resources required for this, but also a huge traffic occurs.

That is, in the existing search technology, not only has to invest a lot of money in the equipment to collect the content of the search target, but also unnecessarily generated a lot of traffic in the process of requesting and receiving the content of the search target. Therefore, a method for securing content to be searched without generating traffic is required.

Furthermore, in the conventional search technology, in order to maintain the freshness of the already-searched content, the content collection has to be repeated periodically, and the spam is filtered after collecting the content of the search target. Therefore, there is a problem that unnecessary resources are wasted.

An object of the present invention is to provide an object collection method and system that can collect the content of the object corresponding to the search target without generating new traffic.

Another object of the present invention is to provide a method and system for collecting objects that can minimize unnecessary hardware resources.

Another object of the present invention is to provide an object collection method and system that can optimally maintain the freshness of an object to be searched.

Furthermore, another object of the present invention is to provide an object collection method and system that can reduce the number of search targets to collect content by filtering spam in advance.

As a technical means for achieving the above-described technical problem, the object collection method using the packet mirroring according to the first aspect of the present invention, the user terminal in the method for collecting the content contained in the object to be searched by the search system, A mirroring step of mirroring packets transmitted and received with the web server; An identification information extraction step of extracting identification information of an object to be searched from the packet mirrored in the mirroring step; A content collection step of collecting content of an object to be searched from the packet mirrored in the mirroring step; And a storage step of storing the content collected in the content collection step in association with the content and in association with the identification information extracted in the identification information extraction step.

An object collection system using packet mirroring according to a second aspect of the present invention includes an identification information database for storing identification information of an object that is a target of a search service and content of an object corresponding to the identification information; A mirroring unit for mirroring packets transmitted and received by a user terminal with a web server; A link processing unit which extracts identification information from the packet mirrored by the mirroring unit, and adds the extracted identification information to the identification information database; And a content collecting unit for extracting content of an object corresponding to identification information added to the identification information database by the link processing unit from the packet mirrored by the mirroring unit, and storing the extracted content in the identification information database. Can be.

According to one embodiment of the present invention having the above configuration, it is possible to provide an object collection method and system that can collect the content of the object corresponding to the search target without generating new traffic.

In addition, according to an embodiment of the present invention, unnecessary hardware resources can be minimized.

According to an embodiment of the present invention, the freshness of the object to be searched can be optimally maintained.

Furthermore, according to an embodiment of the present invention, it is possible to reduce the number of search targets for collecting content by filtering out spam in advance.

1 is a network diagram including an object collection system using packet mirroring according to an embodiment of the present invention.

2 is a functional block diagram illustrating an object collection system using packet mirroring according to an embodiment of the present invention.

3 is a flowchart illustrating an object collection method using packet mirroring according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is "connected" to another part, this includes not only "directly connected" but also "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, which means that it may further include other components, except to exclude other components unless otherwise stated.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an object collection system using packet mirroring according to an embodiment of the present invention.

The network N may include a local area network (LAN), a wide area network (WAN), a value added network (VAN), a personal local area network (PAN), and a mobile communication network (PAN). It can be implemented in all kinds of wired and wireless networks such as mobile radio communication network or satellite communication network.

The user terminal 100 is a communication terminal for connecting to a remote web server 300 through a network N, and may be implemented as a computer, a portable terminal, a television, etc. that can be connected to other terminals and servers. Here, the computer includes, for example, a laptop, desktop, laptop, etc., which is equipped with a web browser, and the portable terminal is, for example, a wireless communication device that ensures portability and mobility. , Personal Communication System (PCS), Global System for Mobile communications (GSM), Personal Digital Cellular (PDC), Personal Handyphone System (PHS), Personal Digital Assistant (PDA), International Mobile Telecommunication (IMT) -2000, Code CDMA Division Multiple Access (2000), including all kinds of handheld based wireless communication devices such as W-Code Division Multiple Access (W-CDMA), Wireless Broadband Internet (WBRO) terminals, smart phones, etc. can do. In addition, the television may include an Internet Protocol Television (IPTV), an Internet Television (Internet Television), a terrestrial TV, a cable TV, or the like.

Web server 300 is a computer system that can be connected to the user terminal 100 or another server in the remote place through the network (N), mainly serves to provide a data service to the communication partner.

The user terminal 100 transmits a service request to the web server 300 through a web browser or other application program. For example, a service request is made in which an HTTP request packet through Hyper Text Transfer Protocol (HTTP) is transmitted from the user terminal 100 to the web server 300, and the web server 300 corresponds to the HTTP request packet. Service is provided by transmitting an HTTP Response packet to the user terminal 100.

At this time, the object collection system 200 according to the present invention is also a computer system that can be connected to the remote user terminal 100 or another server through the network (N), the HTTP request packet received from the user terminal 100 The service may be provided by transmitting a corresponding HTTP response packet back to the user terminal 100. In this case, the object collection system 200 may include a mirroring unit 201 connected to the network backbone network or may communicate with the mirroring unit 201 to receive data acquired by the mirroring unit 201.

Hereinafter, the present invention will be described as the object collection system 200 includes a mirroring unit 201, but the object collection system 200 must include a mirroring unit 201 connected to a network (N) backbone network. It is not necessary to merely receive data acquired by the mirroring unit 201.

Hereinafter, a more detailed configuration of the object collection system 200 will be described. 2 is a functional block diagram illustrating an object collection system using packet mirroring according to an embodiment of the present invention.

As shown in FIG. 2, the object collection system 200 according to the exemplary embodiment of the present invention includes a mirroring unit 201 connected to a wired or wireless communication line connecting the user terminal 100 and the network N. FIG. .

The mirroring unit 201 may mirror or sniff packet data transmitted and received between the web server 300 connected to the network N and the user terminal 100. In practice, the mirroring unit 201 of the object collection system 200 may be implemented as a tap (TAP) device or a switch mirror installed in a network backbone network, or the network of the user terminal 100. It may also be a module of hardware or software installed in an access point serving as a contact point. In this case, the packet mirrored or sniffed by the mirroring unit 201 may be an HTTP request packet transmitted from the user terminal 100 to the web server 300 or may be an HTTP response packet. In particular, in order to collect the content in the content collecting unit 204 to be described later, the mirroring unit 201 may mirror the HTTP response packet and may mirror both the HTTP request packet and the HTTP response packet.

And the object collection system 200 according to an embodiment of the present invention may include a link processing unit 202. The link processing unit 202 processes the packets mirrored by the mirroring unit 201 and adds search target identification information to the identification information database 208 to be described later.

The search object identification information obtained by the link processing unit 202 from the mirrored packet is information for uniquely identifying the object to be searched. For example, an address (URL; Uniform Resource Locator) of the search object is Can be. In addition, the object to be searched may be any type of data existing on the Internet such as a web page, an image, a video, and a document.

At this time, the link processing unit 202 stores the identification information of the search target in the identification information database 208 in a structural form. Here, the identification information database 208 stores and manages search object identification information indicating each object to be searched, which is obtained from the mirrored packet. In addition, the identification information database 208 stores and manages additional information on each search target identification information, wherein the additional information includes the identification information, a packet including the identification information, or the identification information. Information or content included in at least one of another packet corresponding to the packet or an object corresponding to the identification information may be included. The identification database 208 may be configured as a physical or software module including a conventional database system for structurally storing data.

In particular, the link processing unit 202 extracts the identification information included in the packet mirrored by the mirroring unit 201 and writes the HTTP request packet mirrored by the mirroring unit 201 to record the identification information database 208. It is available. In this case, the link processing unit 202 may include not only the identification information included in the HTTP request packet, but also information included in the header of the layered protocol of the packet, including time information on which the packet is mirrored (for example, source address information (Source IP address)). Alternatively, destination address information (Destination IP address) may be stored together with the identification information. In this case, the information included in the header of the layer protocol may be used by the pairing unit 205 to be described later.

In addition, the object collection system 200 may include a spam processing unit 203. The spam processing unit 203 may determine at least some of the identification information registered in the identification information database 208 by the link processing unit 202 as spam. For example, when the status information (HTTP status code) included in the HTTP response packet corresponding to the identification information registered in the identification information database 208 is "200", the spam processing unit 203 performs normal processing. 400 ”can be classified as spam. Alternatively, the spam processing unit 203 may refer to the referrer included in the HTTP request packet including the identification information, and may classify the identification information as spam if the referrer is not a pre-registered host.

The object collection system 200 may include a content collector 204. The content collection unit 204 collects content included in each search target object classified by the identification information. The content collecting unit 204 may collect content by mirroring an HTTP response packet transmitted from the web server 300 to the user terminal 100 at the request of the user terminal 100. That is, the web server 300 may collect the content included in the body of the HTTP response packet transmitted to the user terminal 100.

In this case, the content collecting unit 204 may not collect the content corresponding to the identification information determined by the spam processing unit 203 as spam. In particular, the content collector 204 mirrors a plurality of packets corresponding to one piece of identification information in order to collect the contents of a specific object that is divided into a plurality of packets and then collects the contents included in the body of the packet. have.

In addition, the content collection unit 204 may not collect the content within a certain period of time, if the content has already been collected for one identification information. Alternatively, if a predetermined time has elapsed since the last time content was collected for one piece of identification information, but a packet including the content corresponding to the identification information is not mirrored, an HTTP request for an object corresponding to the identification information is directly performed. By sending a packet, the content contained in the object may be crawled, and the content stored in association with the identification information may be updated with the crawled content.

In particular, the content collection unit 204 may collect only the content of the host allowed to access according to a pre-stored access permission policy (for example, robots.txt). The content collection unit 204 discards the content of the object that is not accessible by the access permission policy for each host, even if the mirroring unit 201 mirrors the content of the object, and accesses the content of the object allowed by the access permission policy. Can only collect. Of course, at this time, the identification information corresponding to the content that is not accessible according to the access permission policy may be pre-registered as spam in the spam processing unit 203.

Meanwhile, the object collection system 200 may include a pairing unit 205. When the identification information is extracted from the HTTP request packet and the content is collected from the HTTP response packet, the pairing unit 205 confirms a correspondence relationship between the extracted identification information and the collected content. That is, the specific identification information and the content of the corresponding object should be matched with each other. When collecting the identification information and the content in different packets, a procedure for checking their correspondence may be necessary. Therefore, the pairing unit 205 confirms the correspondence between the identification information and the content, and can confirm the correspondence between the packets as the correspondence between the packets.

For example, the pairing unit 205 may check whether the HTTP request packet from which the identification information is extracted and the HTTP response packet from which the content is collected correspond to each other. That is, it may be checked whether the HTTP response packet in which the content is collected is a response to the HTTP request packet from which identification information is extracted. At this time, the pairing unit 205 may check the correspondence between the two packets by comparing the protocol header included in the packet. In particular, the packet may be transmitted and received according to a plurality of protocols layered in two or more different layers, and thus may include headers of two or more layer protocols. Accordingly, the pairing unit 205 may compare the information included in the same layer protocol header included in the two packets with each other, and determine whether the two packets correspond to each other.

The pairing unit 205 compares the source address information (Source IP address) and the destination address information (Destination IP address) included in the TCP / IP protocol header included in the HTTP request packet and the HTTP response packet, respectively, and correspond to each other. If, the two packets may be determined to correspond to each other, and thus identification information and contents extracted from each of the two packets may be determined to correspond. Of course, the pairing unit 205 may refer to the information on the time when the two packets are transmitted and received together when determining the correspondence relationship.

Of course, in the embodiment of the present invention, the pairing unit 205 may be unnecessary when the identification information and the content are extracted in the same packet.

The object collection system 200 may include an indexing unit 206 for indexing each piece of identification information based on the contents included in the object collected by the content collecting unit 204. For example, the indexing unit 206 indexes each identification information by referring to a string such as text included in the collected content or a content name included in the collected content, thereby searching an object corresponding to each identification information. Incorporate

In addition, the object collection system 200 includes a search unit 207. When the search unit 207 receives a search request including a search word from the user terminal 100, the search unit 207 searches for identification information of an object in which a string corresponding to the received search word is indexed. In addition, the search unit 207 may arrange the searched identification information and provide the searched identification information to the user terminal 100.

The mirroring unit 201, the link processing unit 202, the spam processing unit 203, the content collecting unit 204, the pairing unit 205, and the indexing unit included in the object collection system 200 ( 206, the search unit 207 may be configured as a software module in which functions are separated from each other in one physical system, or may be configured as at least one software module or a combination of at least one hardware module.

Hereinafter, an object collection method using the object collection system 200 will be described.

3 and 4 are flowcharts illustrating an object collection method using packet mirroring according to an embodiment of the present invention.

As shown in FIG. 3, the object collection method according to the embodiment of the present invention begins with mirroring a packet transmitted and received by the user terminal 100 with the web server 300 (S11). The mirrored packet includes an object request packet requesting the user terminal 100 to transmit a specific object to the web server 300, and an object transmitted by the web server 300 to the user terminal 100. It may be a response packet. In particular, when the object request packet is an HTTP Request packet, a URL of the packet may be included as identification information of the requested object. In addition, when the object response packet is an HTTP response packet, the body of the packet may include the content of the requested object.

The object collecting method according to the embodiment of the present invention performs the step of extracting and collecting identification information included in the mirrored packet (S12). The identification information may be a URL included in the header of the mirrored packet as described above.

In addition, the object collection method according to an embodiment of the present invention includes the step (S13) for collecting the content contained in the mirrored packet in the object collection system 200. Here, the step S13 is, for example, if the identification information and the content is included in one packet, by collecting the content in the packet from which the identification information is extracted, for example, the identification information and the content in different packets If included, the packet from which the identification information is extracted and the packet for collecting the content are first paired, and then the content identified as having a corresponding relationship can be collected.

Furthermore, the object collecting method according to the embodiment of the present invention includes storing the content collected in step S13 in association with the identification information collected in step S12 (S14).

In this case, as described above, the content collected in step S13 may be stored in association with the identification information in the corresponding relationship through a process of confirming the correspondence with the identification information.

As such, the content stored in association with the identification information may be used to index the index word in the identification information.

On the other hand, as described in the object collection system 200 according to the embodiment of the present invention, the step (S13) of collecting the content of the object corresponding to the identification information may be made a plurality of times for one identification information. Even if the content corresponding to the object corresponding to the same identification information has already been collected and the collected content is stored in association with each other, if a predetermined time elapses, the process of collecting the content corresponding to the identification information may be repeated. Of course, whenever specific identification information is extracted, the corresponding content may be collected together to maintain the freshness of the content. However, a time may be set in advance, and a process of collecting content for the same identification information may not be repeated within the time.

On the other hand, even if a predetermined time has elapsed since the collection of content for a specific identification information and a limit time that is set longer than the above-described predetermined time has elapsed, the object collection system is not mirrored. 200 may directly send an object request packet requesting an object for the identification information to the web server 300, and collect the content from the response. This is to provide more accurate search results by maintaining the freshness of the content corresponding to the identification information.

The object collecting method using packet mirroring according to the embodiment described with reference to FIG. 3 may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, computer readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transmission mechanism, and includes any information delivery media.

The foregoing description of the present invention is intended for illustration, and it will be understood by those skilled in the art that the present invention may be easily modified in other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is shown by the following claims rather than the above description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

Claims

In the search system collects the content contained in the object to be searched,

A mirroring step of mirroring packets transmitted and received by a user terminal with a web server;

An identification information extraction step of extracting identification information of an object to be searched from the packet mirrored in the mirroring step;

A content collection step of collecting content of an object to be searched from the packet mirrored in the mirroring step; And

And a storage step of storing the content collected in the content collection step in association with the content and in association with the identification information extracted in the identification information extraction step.
The method of claim 1,

The packet mirrored in the mirroring step is at least one of an object request packet transmitted from a user terminal to a web server and an object response packet transmitted from the web server to the user terminal in response to the object request packet. How to collect objects.
The method of claim 1,

The identification information extraction step,

Extracting the identification information from the object request packet transmitted from the user terminal to the web server, which is mirrored in the mirroring step,

The content collection step,

And collecting the content from an object response packet transmitted from the web server to the user terminal in response to the object request packet mirrored in the mirroring step.
The method of claim 3,

The object collection method,

And a pairing step of confirming a correspondence relationship between the identification information extracted in the identification information extraction step and the content collected in the content collection step.
The method of claim 4, wherein

The pairing step,

And checking the correspondence between the source request information and the destination address information included in the object request packet and the object response packet, respectively.
The method of claim 4, wherein

The pairing step,

And comparing the information contained in one or more of the same layer protocol headers included in each of the object request packet and the object response packet with each other.
The method of claim 1,

The object collection method,

And updating the content associated with the identification information by re-collecting the content with respect to the identification information and the content stored in the storing step.
The method of claim 7, wherein

The update step,

If the packet mirrored in the mirroring step includes content for the identification information that has already been collected, the content stored in association with the identification information is changed to content included in the mirrored packet in the mirroring step. Comprising a step of collecting the object using the packet mirroring.
The method of claim 3,

The content collection step,

And determining whether to collect content included in the object response packet based on object state information included in the object response packet.
The method of claim 1,

The content collection step,

And determining whether to collect content included in the mirrored packet by the mirroring unit by referring to a pre-registered access permission policy for each host.
The method of claim 1,

The object collection method,

And indexing a character string in the identification information based on the content stored in association with the identification information in the storing step.
The method of claim 11,

The object collection method,

When a search request including a search word is received from the user terminal, the indexing step further comprises a search step of searching for the identification information to be provided to the user terminal by comparing the search string with the string indexed to the identification information, using packet mirroring How to collect objects.
An identification information database for storing identification information of an object which is a target of a search service and contents of an object corresponding to the identification information;

A mirroring unit for mirroring packets transmitted and received by a user terminal with a web server;

A link processing unit for extracting identification information from the packet mirrored by the mirroring unit and adding the extracted identification information to the identification information database; And

And a content collecting unit for extracting content of an object corresponding to identification information added to the identification information database by the link processing unit from the packet mirrored by the mirroring unit, and storing the extracted content in the identification information database. Object Collection System using Packet Mirroring.
The method of claim 13,

The link processing unit,

Extracting identification information from the object request packet transmitted from the user terminal to the web server, which is mirrored by the mirroring unit,

The content collecting unit,

And collecting content from an object response packet transmitted from the web server to the user terminal in response to the object request packet mirrored by the mirroring unit.
The method of claim 14,

The object collection system,

It further comprises a pairing unit for confirming the correspondence between the identification information extracted by the link processing unit and the content collected by the content collection unit,

The content collecting unit,

And storing the collected content in association with the collected information and the identification information corresponding to the collected content, based on the correspondence confirmed by the pairing unit.
The method of claim 13,

The object collection system,

Further comprising a spam processing unit for determining a part of the identification information extracted by the link processing unit as spam,

The content collecting unit,

The object collection system using packet mirroring does not collect the content corresponding to the identification information determined to be spam according to the spam processing unit.