KR102222287B1 - Web Crawler System for Collecting a Structured and Unstructured Data in Hidden URL - Google Patents

Abstract

The present invention provides a web crawler system for collecting structured and unstructured data included in a hidden URL. The web crawler system for collecting structured and unstructured data included in the hidden URL according to the present invention can find the hidden URL address link and process the structured and unstructured data of the web content included in the hidden URL address. By showing a high speed, it is possible to generate big data based on more accurate data from related websites for many fields connected by hyperlinks, and based on the generated big data, it is possible to create more accurate data requested by users for the relevant field. It has the effect of providing statistics and data much more quickly, and it will be used as the basis for building an artificial intelligence system used for shopping at the lowest price demanded by consumers, customized advertisements, and product recommendations by utilizing the accumulated big data. In order to be able to access the hidden URL address through a separate Casper service module, the content of the web document accessed through the URL address can be accessed through a specific area that the user wants to crawl and the type of data to be collected. Web crawler system for collecting structured and unstructured data included in hidden URLs that can collect structured and unstructured data in a specific area through an extension program that can perform settings, and monitor crawled results. It is about.

Description

Web Crawler System for Collecting a Structured and Unstructured Data in Hidden URL

The present invention relates to a web crawler system for collecting structured and unstructured data included in a hidden URL, and more particularly, a headless browser such as Chrome to access a hidden URL address in a web document. Regarding data extraction from the environment, a utility such as Casper.js is used to access the hidden URL address to extract web content, and an extension program to collect not only structured data such as text but also unstructured data such as audio or video. It relates to a web crawler system for collecting structured and unstructured data included in a hidden URL that can collect structured and unstructured data through a set value that sets the data type desired by the user and the area to be collected.

Recently, the importance of big data is increasing as it is combined with services in various fields such as shopping at the lowest price using big data, customized advertisements, and product recommendations. As a result, the importance of a web crawler that automatically collects data distributed on a website is also increasing.

Here, in the method of accessing other web sites through hyperlinks included in web documents, in the HTML language for creating web documents, the hyperlink corresponds to the A tag. Tag A has an href attribute, which has the URL address of another website. The web crawler accesses the web site through this URL address, but there are several problems with this.

The first problem is that the URL address can be concealed by using a JavaScript function, so existing web crawlers cannot access and collect the hidden URL address. The second problem is that the existing web crawlers have difficulty in distinguishing and collecting structured data such as tags and text and unstructured data such as video and audio.

Web crawler is an automation that collects data contained in web documents such as text, images, and videos in the Internet environment where web sites are connected like a spider web with hyperlinks, and collects data again by accessing other web sites through hyperlinks. It's a program.

A web crawler navigates the world wide web in an organized and automated way. The job of a web crawler is called web crawling or spidering, and is done in the form of a bot or software agent.

These web crawlers are largely divided into general web crawlers and distributed web crawlers.

1 is a diagram showing the overall flow of a general web crawler according to a conventional embodiment.

As shown in FIG. 1, the basic operation of a general web crawler is to retrieve a URL address included in a web document from the URL Frontier module and retrieve a web page of the URL address using the http protocol. And the Fetch module saves the web page to a temporary storage. Then, the Parse module extracts text and links from the web page of the temporary storage, sends the text to the Indexer, and determines whether the link should be added to the URL Frontier through the Content Seen module, the URL Filter module, and the Duplication URL Element module. .

However, the problem is that the URL address hidden in the link cannot be added because it is hidden even though it is the URL address to be added to the URL Frontier.

Therefore, it is virtually impossible to crawl all of the web documents around the world with a regular web crawler, so you have to use a distributed web crawler. Distributed web crawlers are largely divided into two types, one is the centralized method used by Google, and the other is the peer to peer or Fully-Distributed (P2P) method used by Mercator or elsewhere.

2 is a diagram showing a structure of a centralized distributed web crawler according to a conventional embodiment.

As shown in FIG. 2, in the centralized distributed web crawler, the URL Manager acts as a server and the crawler acts as a client. After downloading the document from the crawler, extracting the OutLink URL address and passing it to the URL Manager, the URL Manager checks whether the downloaded document is the URL address and removes the duplicate URL address. In other words, URL Manager replaces URL duplication and URL management in general web crawlers.

3 is a diagram showing a structure of a P2P distributed web crawler according to a conventional embodiment.

As shown in FIG. 3, in the P2P method, each crawler has a completely independent structure, and each crawler operates like a general web crawler. Each crawler downloads documents, extracts OutLink URLs, and even de-duplication of URLs, all operate independently in each crawler.

In order to operate in this way, the list of downloaded URL addresses managed by each crawler must be mutually exclusive. If not, the same document is downloaded from different crawlers. As a way to solve this problem, each crawler manages the list of URL addresses to be downloaded exclusively from each other. In other words, it manages only what belongs to the download list, and passes the remaining URL addresses to other crawlers so that each crawler can operate independently.

Conventional Patent No. 10-1153138 client-based web crawling system and method update the data of the crawler with the information collected by the web crawler of the server so that the distributed property of the server of the server is used to provide fast and accurate web crawling data. Data is compared with data retrieved by the client of the server to perform data comparison, and data comparison is accomplished using information scattered through search engine results pages, and data verification is performed by a client dictionary from the server that summarizes the web crawler data. A system and method for acquiring information from a networked system using a distributed web crawler that facilitates data analysis by providing a means to counter spoofing of web crawlers to achieve and improve data accuracy.

However, the conventional patent provides a means to counter spoofing of web crawlers in order to improve accuracy by verifying data in order to distribute the information collected by the web crawler of the server to the web crawler for distributed web crawling, thereby facilitating data analysis. There is a limit to the ability to find hidden URL addresses.

4 is a block diagram of a web content extraction system according to a conventional embodiment.

As shown in FIG. 4, the technology for extracting web content included in a web document provides a function of automatically extracting the title, author, publication date, and text, which are contents to be used for information analysis from the web document. The web content extraction system is a device that automatically generates rules for extracting content and extracts only content.A Rule Generator that automatically generates content extraction rules, and a Navigation Content Eliminator that removes navigation content from a given web document. ,Consists of a content extractor (Core Context Extractor) that extracts content through content extraction rule keyword similarity comparison.

Conventional Patent Publication No. 10-2018-0024360 A data collection method using a data agent container and a system thereof includes a data collection method using a data agent container and a data collector; A data collection controller controlling the data collector; And a control monitoring channel connecting the data collector and the data collection controller to each other, wherein the data collection controller includes: a data collector installation and removal control module; Data collector start and end control module; And a data collector status monitoring module. It provides a data collection system using a data agent container.

However, the conventional technology collects data through a website using a web crawler in a data collector, so it is impossible to find and search a hidden URL address. There is a problem in that there is no technology for collecting data.

Korean Patent Publication No. 10-1153138 “Client-based web crawling system and method” Republic of Korea Patent Laid-Open Publication No. 10-2018-0024360 “Data collection method and system thereof using data agent container”

Therefore, the present invention is a technology that improves the problems of the prior art and collects documents distributed and stored in a myriad of computers and includes them as an index of a search target. This is the factor that determines the advantage. Web documents accessed through the URL address by allowing access to the hidden URL address through a separate Casper service module for web documents through crawling, which is an important element of recent web searches. Through an extension program that allows users to set the specific area to crawl and the type of data to be collected for the contents of the user, it collects structured and unstructured data in a specific area, and performs monitoring to view the crawled results. The purpose of this is to provide a web crawler system for collecting structured and unstructured data included in hidden URLs.

According to a feature of the present invention for achieving the above object, the present invention is a client 100 that receives input from a user a specific area to be crawled and a data type to be collected, and outputs the crawl result to the user; and the client ( A web server 200 that performs a crawl according to a specific area and data type received from 100), stores the crawl result in the storage module 300, and transmits the crawl result to the client 100 according to a user's request; And a storage module that stores the URL address, structured data and unstructured data transmitted through the web server 200, and transmits a crawl result file received from the web server 200 according to a user's request to the web server 200. (300); It provides a web crawler system for collecting the structured and unstructured data included in the hidden URL, characterized in that consisting of a configuration including.

The client 100 of the web crawler system for collecting the structured and unstructured data included in the hidden URL according to the present invention receives a crawl rule and a collection area to be extracted from the user, and the crawler data to be output according to the user's request A setting input module 110 receiving the type of input; and a crawling rule received through the setting input module 110, a collection area, a main URL address, and a user's request required for output to the web server 200, and , A client communication module 120 that receives the structured and unstructured crawler collection data transmitted from the web server 200; and the client communication module 120 according to a user's request received from the setting input module 110 An output module 130 for outputting the collected data from the structured and unstructured crawler to the user; And a client control module 140 that processes an event from a user and an internal event of the client 100 generated in the setting input module 110, the client communication module 120, and the output module 130. It is characterized by consisting of a configuration.

The web server 200 of the web crawler system for the collection of structured and unstructured data included in the hidden URL according to the present invention, the crawling rule, the collection area, and the main URL address received through the client communication module 120 A server communication module 210 that transmits to the collection engine module 220 and transmits the crawling data received through the storage access module 250 to the client communication module 120 according to the received user's request; and the server By analyzing the crawling rule received through the communication module 210 and the collection area included in the main URL address, driving the crawler module 230 or the casper module 240 to perform crawling, and store the collected data through the crawling. A collection engine module 220 that transmits to the access module 250; and a crawler rule driven by the scheduler module 222 of the collection engine module 220 to receive a seed including a main URL address, a crawler rule, and a URL. Crawler module 230 that crawls according to the data type defined in; And it is driven by the scheduler module 222 of the collection engine module 220 to receive a seed including a main URL address, a crawler rule, and an XPath, and the crawler rule Casper module 240 for interpreting and crawling the hidden URL address included in the seed according to the data type defined in; And the storage module for storing the URL address, structured data, and unstructured data transmitted from the collection engine module 220 ( A storage access module 250 that transmits to 300, receives a crawl result file according to the request of the client 100, and transmits it to the server communication module 210; And an exception processing module 260 that processes exceptional events that may occur in the web server 200.

The collection engine module 220 of the web crawler system for collecting the structured and unstructured data included in the hidden URL according to the present invention includes the crawling rule transmitted from the server communication module 210 and the collection included in the main URL address. A collection rule module 221 that analyzes the region, creates a seed included in the collection region, and transmits it to the scheduler module 222; and the crawler module 230 or casper based on the seed defined in the collection rule module 221 A scheduler module 222 that drives the module 240 and transmits a main URL address, a crawler rule, and a seed, respectively; And the file transmitted from the scheduler module 222 is transmitted to the casper module 240 or the URL address and structured data transmitted from the scheduler module 222 and crawled from the crawler module 230 or the casper module 240 It characterized in that it consists of a configuration including; an engine file module 223 for receiving unstructured data among the results and transmitting it to the storage access module 250.

The crawler module 230 of the web crawler system for collecting the structured and unstructured data included in the hidden URL according to the present invention analyzes the crawler rule received from the scheduler module 222 and sends it to the crawler collection module 232. A crawler rule translation module 231 that determines and transmits the type of data to be crawled; and crawls the types of data transmitted from the crawler rule translation module 231 with respect to URL addresses included in the main URL included in the seed. A crawler collection module 232 which transmits the crawled data together with the URL address to the scheduler module 222 or the crawler file module 233 and notifies the scheduler module 222 of a completion message; And a crawler file module 233 for transmitting the unstructured data received from the crawler collection module 232 to the engine file module 223 in a form in which a URL address is displayed.

The casper module 240 of the web crawler system for collecting the structured and unstructured data included in the hidden URL according to the present invention analyzes the crawler rule received from the scheduler module 222 and sends it to the casper collection module 242. A casper rule translation module 241 that determines the type of data to be crawled, analyzes the XPath of the seed, analyzes the hidden URL address, and transmits it to the casper collection module 242; and the casper rule translation module 241 Casper collection that transmits the crawled data for each data type of the hidden URL addresses interpreted by the URL address together with the URL address to the scheduler module 222 or the casper file module 243, and then notifies the completion message to the scheduler module 222 Module 242; And a casper file module 243 that transmits the structured data and unstructured data received from the casper collection module 242 to the engine file module 223 in a form in which a URL address is displayed. .

According to the crawler system for collecting structured and unstructured data included in the hidden URL address according to the present invention and the data collection method using the same, the URL address link hidden than the conventional Node.js-based web crawler module, which is a conventional crawler. By showing a high speed in processing the structured and unstructured data of web content included in the hidden URL address, it is possible to create big data based on more accurate data from related websites in many fields connected by hyperlinks. And, based on the generated big data, it has the effect of providing more accurate statistics and data for the data requested by users for the relevant field much faster, and consumers demand by utilizing the accumulated big data based on this. It has an advantage that can be used as a basis for building an artificial intelligence system used for low-cost shopping, customized advertisement, and product recommendation.

1 is a view showing the overall flow of a general web crawler according to a conventional embodiment;
2 is a view showing a structure of a centralized distributed web crawler according to a conventional embodiment;
3 is a diagram showing a structure of a P2P distributed web crawler according to a conventional embodiment;
4 is a block diagram of a web content extraction system according to a conventional embodiment;
5 is a block diagram of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention;
6 is a configuration diagram of a client of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention;
7 is an exemplary diagram of a setting input module of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention;
8 is an exemplary diagram of a setting input module of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention;
9 is a block diagram of a web server of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention;
10 is a block diagram of a collection engine module of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention;
11 is a block diagram of a crawler module of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention;
12 is a block diagram of a casper module of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention.

In order to fully understand the present invention, a preferred embodiment of the present invention will be described in detail with reference to FIGS. 1 to 12 of the accompanying drawings. The embodiments of the present invention may be modified in various forms, and the scope of the present invention should not be construed as being limited to the embodiments described in detail below. This embodiment is provided to more completely explain the present invention to those of ordinary skill in the art. Accordingly, the shape of the element in the drawings may be exaggerated to emphasize a clearer description. It should be noted that in each drawing, the same member may be indicated by the same reference numeral. On the other hand, in the drawings and detailed description, the illustration and reference of the configuration and operation that can be easily recognized by those in the field, such as a client, a web server, a web crawler system, and an exception handling module, have been simplified or omitted. In particular, in the illustration and detailed description of the drawings, a detailed description and illustration of a specific technical configuration and operation of elements not directly related to the technical features of the present invention are omitted, and only the technical configuration related to the present invention is briefly shown or Explained.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

5 is a block diagram of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention.

As shown in FIG. 5, the web crawler system includes a client 100, a web server 200, and a storage module 300.

6 is a block diagram of a client of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention.

As shown in FIG. 6, the client 100 includes a configuration input module 110, a client communication module 120, an output module 130, and a client control module 140. The client 100 receives a specific area to be crawled and a data type to be collected from the user, and outputs the crawl result to the user.

7 is an exemplary diagram of a setting input module of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention.

As shown in FIG. 7, the setting input module 110 receives a crawl rule and a collection area to be extracted from the user, and receives the type of crawler data to be output according to the user's request. The crawling rule is defined in the setting input module 110 and the collection area is extracted through an extended function provided by a web browser.

The extension function is a program for web browsers that is separately provided to change the behavior of existing functions of the web browser or to add completely new functions. Extensions are provided by Chrome, Firefox, Microsoft Edge, and Opera web browsers.

The setting input module 110 may first select a data type that the user wants to crawl, such as an image, text, or video, by checking a check box in the input menu for determining the crawling rule, as indicated in ①. If not selected, all data types included in the area can be imported.

If a specific area is not selected in the setting input module 110, the entire displayed web page is crawled, and when a specific area is selected, a user interface (UI) is displayed as in ②. You can turn on/off the specific area extraction function by pressing the button displayed at the top. When the ON function is activated, an element of the Document Object Model (DOM) pointed by the mouse pointer is brought to the Mouse Up event of the document object.

In the setting input module 110, as ③, the background color property of the DOM pointed to by the mouse pointer is modified to recognize which DOM the user points to, and when the user clicks the mouse, the imported DOM element is stored. And, in the UI of ②, the DOM element is converted into XPath and displayed as a list. Finally, if you click the crawl button at the bottom of ①, a crawl rule is created with the selected data type and the XPath host URL data of a specific area and delivered to the web server.

③ is implemented so that the user can recognize which DOM the user points to by modifying the back-ground-color property of the DOM pointed to by the mouse pointer. When the mouse is clicked, the imported DOM element is saved, and the DOM element is converted to XPath in UI ② and displayed as a list.

8 is an exemplary diagram of a setting input module of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention.

Finally, if you click the "Crawl" button located at the bottom of ①, as shown in Fig. 8, a crawl rule and a file for a specific area are created with the selected data type and XPath host URL data for a specific area, and the client communication module ( 120). Crawl rules and files for specific areas are described in Json syntax, and data_type, specific_area, and host_url properties exist. The data_type property has a Boolean value for each type, the default value is set to false, and the specific_area property is the DOM extracted by the user. It is converted into XPath syntax and added.

The setting input module 110 requests the form of all or a desired data type for the structured and unstructured data collected by crawling through the web server 200.

The client communication module 120 transmits a crawling rule, a collection area, and a main URL address received through the setting input module 110 and a user's request for output to the web server 200, and transmits a request from the web server 200 to the web server 200. Receives the received data collected from the structured and unstructured crawlers. The client communication module 120 is responsible for communicating with the web server 200.

The output module 130 outputs the structured and unstructured crawler collection data transmitted from the client communication module 120 according to the user's request received from the setting input module 110 to the user. The output module 130 outputs a result of crawling through the web server 200 through the client communication module 120 to the user.

The client control module 140 processes events from the user and internal events of the client 100 generated in the setting input module 110, the client communication module 120, and the output module 130. The client control module 140 processes events by various user actions, such as a user clicking a check box or a specific button using a mouse, or moving or dragging a mouse through the setting input module 110, and the client ( 100) controls the driving of events occurring between modules.

9 is a block diagram of a web server of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention.

9, the web server 200 includes a server communication module 210, a collection engine module 220, a crawler module 230, a casper module 240, a storage connection module 250, and an exception processing module. It consists of a configuration including 260. The web server 200 performs crawling according to the specific area and data type received from the client 100, stores the crawl result in the storage module 300, and sends the crawl result to the client 100 according to the user's request. send.

The web server 200 is a high-performance server that supports non-blocking I/O on a single thread basis, but because it is single thread, there is a limit to crawling multiple web pages.

Therefore, to solve the above problem, a cluster is used. The cluster is a module of Node.js, which creates a number of processes that share a port of the web server 200, creates and collects the crawler module 230 or the casper module 240 as threads, thereby solving the limitation of speed.

The server communication module 210 transmits the crawling rule, the collection area, and the main URL address received through the client communication module 120 to the collection engine module 220, and the storage access module ( The crawling data received through 250) is transmitted to the client communication module 120. The server communication module 210 is responsible for communication with the client 100.

10 is a block diagram of a collection engine module of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention.

As shown in FIG. 10, the collection engine module 220 includes a collection rule module 221, a scheduler module 222, and an engine file module 223. The collection engine module 220 analyzes the crawling rule received through the server communication module 210 and the collection area included in the main URL address, and drives the crawler module 230 or the casper module 240 to perform crawling. , And transmits the data collected through the crawling to the storage access module 250.

The collection rule module 221 analyzes the crawling rule transmitted from the server communication module 210 and the collection area included in the main URL address, creates a seed included in the collection area, and transmits it to the scheduler module 222. The collection rule module 221 defines the type of data to be collected in the received crawling rule.

The collection rule module 221 generates a seed, which is a list of URL addresses or XPaths described in the collection area included in the main URL address received from the server communication module 210. XPath (XML Path Language) is a language that describes how to place and process items using the syntax specified on the path through the document structure of the extended generation language as a standard of W3C, and includes a hidden URL address.

The scheduler module 222 drives the crawler module 230 or the casper module 240 based on the seed defined in the collection rule module 221, and transmits a main URL address, a crawler rule, and a seed, respectively. The scheduler module 222 receives a crawl rule and a seed from the collection rule module 221 and, when the seed value is a URL, transmits the crawl rule to the crawler module 230 to perform crawling. When the seed value is XPath, the scheduler module 222 transmits a crawl rule to the casper module 240 to perform crawling.

The scheduler module 222 drives the casper module 240 when the seed value is XPath. The scheduler module 222 may use the engine file module 223 to store the main URL address, the seed including the XPath, and the crawling rule as a file and transmit it to the casper module 240 . This is because Casper module 240 cannot get the URL of the link tag from the web page of the main URL address, so the location path of the link tag in the DOM structure must be described in XPath syntax and sent to the Casper module. After finding the link tag by analyzing the link tag, you can access the URL you want to crawl by forcibly generating a click event.

Therefore, since the scheduler module 222 needs to separately analyze the seed made of the XPath, it is reasonable to transmit it in the form of a file to process it.

The engine file module 223 transmits the file received from the scheduler module 222 to the casper module 240 or the URL address and structured data received from the scheduler module 222, and the crawler module 230 or casper module. Among the crawled results from 240, unstructured data is received and transmitted to the storage access module 250.

The engine file module 223 receives a file corresponding to unstructured data such as a URL address and structured data and an image, audio and video in which the URL address is displayed, or a location where the file is stored, for example, Receive the names of folders and files. And it transmits this to the storage connection module 250.

11 is a block diagram of a crawler module of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention.

As shown in FIG. 11, the crawler module 230 includes a crawler rule translation module 231, a crawler collection module 232, and a crawler file module 233. The crawler module 230 is driven by the scheduler module 222 of the collection engine module 220 to receive a seed including a main URL address, a crawler rule, and a URL, and crawl according to the data type defined in the crawler rule.

The crawler module 230 determines the collected DOM tag according to the crawler rule. DOM tags differ according to the type of multimedia data. For example, video tag is used for video, img tag is used for image, and audio tag is used for music. Data, which are images, images, and music, are stored as separate files, and the stored location is transmitted to the scheduler module 222.

The crawler rule translation module 231 is a module that analyzes the crawler rule received from the scheduler module 222, determines the type of data to be crawled to the crawler collection module 232, and transmits it.

The crawler collection module 232 crawls the types of data transmitted from the crawler rule translation module 231 with respect to URL addresses included in the main URL included in the seed, and converts the crawled data together with the URL address to the scheduler module 222 ) Or, after transmission to the crawler file module 233, a completion message is notified to the scheduler module 222.

The crawler collection module 232 transmits unstructured data such as images, images, and music among the crawled data to the crawler file module 233. In addition, when crawling is completed, the crawler collection module 232 transmits a crawled URL address and structured data such as text to the scheduler module 222 along with a completion message.

The crawler file module 233 transmits the unstructured data received from the crawler collection module 232 to the engine file module 223 in the form of a URL address displayed. The crawler file module 233 may transmit unstructured data as a file or may transmit information on a location where the file is stored.

12 is a block diagram of a casper module of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention.

As shown in FIG. 12, the casper module 240 includes a casper rule translation module 241, a casper collection module 242, and a casper file module 243. Casper module 240 is driven by the scheduler module 222 of the collection engine module 220 to receive a seed including a main URL address, a crawler rule, and an XPath, and include it in the seed according to the data type defined in the crawler rule. The hidden URL address is analyzed and crawled.

The casper module 240 reads the file received from the engine file module 223 through the casper file module 243 after driving, interprets the XPath-based hidden URL address included in the seed, and interprets the URL address. Structured data and unstructured data can be crawled by analyzing the data type according to the crawler rule.

The casper rule translation module 241 interprets the crawler rule received from the scheduler module 222 to determine the type of data to be crawled in the casper collection module 242, and interprets the XPath of the seed to determine the hidden URL address. It is a module that analyzes and transmits it to the casper collection module 242.

The Casper collection module 242 collects data crawled by data type for the hidden URL addresses interpreted by the Casper rule translation module 241 together with the URL address to the scheduler module 222 or the Casper file module 243. After transmission, a completion message is notified to the scheduler module 222.

The casper collection module 242 transmits the crawled structured data and unstructured data together with the crawled URL address to the casper file module 243, and when the crawling is completed, transmits a completion message to the scheduler module 222. Casper collection module 242 transmits structured data as a file because it can only transmit data as a file due to the implementation of the program. If it is considered to be efficient, it is more efficient to transfer it through memory rather than transfer it as a file like the crawler collection module.

The casper file module 243 transmits the structured data and unstructured data received from the casper collection module 242 to the engine file module 223 in a form in which a URL address is displayed. The casper file module 243 may transmit the crawled data as a file or transmit information on a location where the file is stored.

The storage access module 250 transmits the URL address, structured data, and unstructured data transmitted from the engine file module 223 of the collection engine module 220 to the storage module 300, and requests from the client 100 According to the following, the crawling result file is received and transmitted to the server communication module 210. The storage access module 250 transmits the crawled structured data and unstructured data crawled by URL addresses that have been hidden or not contained in the main URL to be stored in the storage module 300.

The exception processing module 260 processes exceptional events that may occur in the web server 200. The exception handling module 260 responds to the occurrence of various exceptional events, such as the crawler module 230 or the casper module 240 not being driven, or the storage access module 250 being unable to access the storage module 300. A message is generated so that it can be checked and notified to the user through the server communication module 210.

The storage module 300 stores the URL address, structured data and unstructured data transmitted through the web server 200, and stores the crawling result file according to the user's request received from the web server 200 to the web server 200. Transfer to. The storage module 300 is generally formed in the form of a database, but may store a URL address, structured data crawled by URL address, and unstructured data in the form of a file.

As described above, a preferred embodiment of a web crawler system for collecting structured and unstructured data included in a hidden URL according to an embodiment of the present invention has been illustrated according to the above description and drawings, but this is only the description of the present invention. It is only described as an example to easily explain the contents and aid understanding of the invention, and is not intended to limit the scope of the present invention, and various changes and modifications such as other modified examples within the scope not departing from the technical spirit of the present invention It will be apparent to those of ordinary skill in the art that this is possible.

100: client 110: setting input module
120: client communication module 130: output module
140: client control module 200: web server
210: server communication module 220: collection engine module
221: collection rule module 222: scheduler module
223: engine file module 230: crawler module
231: crawler rule translation module 232: crawler collection module
233: crawler file module 240: casper module
241: Casper rule translation module 242: Casper collection module
243: Casper file module 250: storage connection module
260: exception handling module 300: storage module

Claims (7)

A setting input module 110 that receives a crawl rule and a collection area to be extracted from a user, and receives a type of crawler data to be output according to a user's request; The crawling rule, collection area and main URL address received through the setting input module 110 and the user's request for output are transmitted to the web server 200, and the structured and unstructured A client communication module 120 receiving the crawler collection data; An output module 130 for outputting the structured and unstructured crawler collection data received from the client communication module 120 according to the user's request received from the setting input module 110 to the user; And a client control module 140 for processing an event from a user and an internal event of the client 100 generated in the setting input module 110, the client communication module 120, and the output module 130. A client 100 configured to receive input from a user for a specific area to be crawled and a data type to be collected, and output a crawl result to the user;
The crawling rule, collection area, and main URL address received through the client communication module 120 are transmitted to the collection engine module 220, and the crawl data received through the storage access module 250 according to the received user's request. A server communication module 210 that transmits a message to the client communication module 120; By analyzing the crawling rule received through the server communication module 210 and the collection area included in the main URL address, driving the crawler module 230 or the casper module 240 to perform crawling, and data collected through the crawling A collection engine module 220 that transmits the data to the storage connection module 250; A crawler module 230 driven by the scheduler module 222 of the collection engine module 220 to receive a seed including a main URL address, a crawler rule, and a URL, and crawl according to the data type defined in the crawler rule; It is driven by the scheduler module 222 of the collection engine module 220 to receive a seed including a main URL address, a crawler rule, and an XPath, and store the hidden URL address included in the seed according to the data type defined in the crawler rule. Casper module 240 for analyzing and crawling; The server communication module 210 transmits the URL address, structured data, and unstructured data transmitted from the collection engine module 220 to the storage module 300, and receives a crawl result file according to the request of the client 100. Storage connection module 250 for transmitting to the; And an exception processing module 260 that processes exceptional events that may occur in the web server 200, and performs crawling according to a specific area and data type received from the client 100 A web server 200 that stores the crawl result in the storage module 300 and transmits the crawl result to the client 100 according to a user's request; And
A storage module that stores the URL address, structured data and unstructured data transmitted through the web server 200, and transmits a crawl result file received from the web server 200 according to the user's request to the web server 200 ( 300); Web crawler system for collecting structured and unstructured data included in the hidden URL, characterized in that consisting of a configuration including.
delete delete The method of claim 1,
The collection engine module 220,
A collection rule module 221 that analyzes the crawling rule transmitted from the server communication module 210 and the collection area included in the main URL address, creates a seed included in the collection area, and transmits it to the scheduler module 222;
A scheduler module 222 that drives the crawler module 230 or the casper module 240 based on the seed defined in the collection rule module 221 and transmits a main URL address, a crawler rule, and a seed, respectively; And
The file received from the scheduler module 222 is transmitted to the casper module 240 or the URL address and structured data transmitted from the scheduler module 222 and the crawler module 230 or the casper module 240 crawled An engine file module 223 that receives the unstructured data and transmits it to the storage access module 250; a web crawler system for collecting structured and unstructured data included in a hidden URL.
The method according to claim 1 or 4,
The crawler module 230,
A crawler rule translation module 231 which analyzes the crawler rule received from the scheduler module 222 and determines the type of data to be crawled to the crawler collection module 232 and transmits it;
A scheduler module 222 or a crawler file module 233 crawls the types of data transmitted from the crawler rule translation module 231 with respect to URL addresses included in the main URL included in the seed, together with the URL address. ), a crawler collection module 232 for notifying the scheduler module 222 of a completion message; And
A crawler file module 233 for transmitting the unstructured data received from the crawler collection module 232 to the engine file module 223 in the form of a URL address displayed; included in the hidden URL, characterized in that it comprises a configuration including Web crawler system for the collection of structured and unstructured data.
The method according to claim 1 or 4,
The casper module 240,
Casper collection module 242 by analyzing the crawler rule received from the scheduler module 222 to determine the type of data to be crawled in the casper collection module 242, and analyzing the hidden URL address by analyzing the XPath of the seed. Casper rule translation module 241 to transmit to;
After transmitting the crawled data for each data type of the hidden URL addresses interpreted by the Casper Rule Translation Module 241 to the scheduler module 222 or the Casper File Module 243 together with the URL address, a completion message is sent to the scheduler. A casper collection module 242 notifying the module 222; And
Concealed URL, characterized in that it comprises a configuration comprising a; Casper file module (243) for transmitting the structured data and unstructured data received from the Casper collection module 242 to the engine file module 223 in the form of a URL address displayed. Web crawler system for the collection of structured and unstructured data contained in.
The method of claim 4,
The scheduler module 222 uses the engine file module 223 to store the main URL address, the seed including the XPath, and the crawling rule as a file and transmits it to the casper module 240. Web crawler system for the collection of structured and unstructured data.

Images