KR102407312B1 - Apparatus and method for crawling data using web archive - Google Patents

Abstract

A data crawling device using a web archive that can reduce the time for sufficient collection of meaningful data by collecting data from various websites by the crawling module, while also collecting historical data of the website by the web archive module, and A method and a recording medium are disclosed. An apparatus for crawling data using a web archive according to an embodiment of the present invention includes: a crawling module configured to crawl basic list configuration information including title, summary, and creation time from various web sites; a web archive module for collecting past data of the web site from a web archive site that collects and records change data of the web site by time; a database configured to store the basic list configuration information collected from the web site and the historical data collected from the web archive site; and a user target site creation module configured to generate a user target site in which the various web sites are integrated from the basic list configuration information and the past data.

Description

Apparatus and method for crawling data using web archive

The present invention relates to an apparatus and method for crawling data using a web archive, and more particularly, to collect data from various web sites by a crawling module, while also collecting past data of web sites from a web archive site By doing so, it relates to a data crawling apparatus and method using a web archive that can reduce the time for sufficient collection of meaningful data.

Recently, service providers that provide services such as shopping malls through the Internet can increase user inflow channels and share user pools. . Such meta-services generally collect information from web sites through crawling. The current web site crawling method is implemented by receiving a value called using an application program interface (API) or by directly accessing each page of the web site to collect data within the corresponding page.

Such a conventional web site direct access method has a problem in that access to the web site is easily blocked when frequent access attempts are made within a predetermined time. A website may be set to prevent an unspecified party from scraping its own web page by default, and this has limitations in which access from the website may be blocked or data crawling may be restricted.

In addition, in the prior art, since the user has to go through a process of collecting data by accessing various sites for which a user wants to compare, and comparing and analyzing the data collected from each site, the inquiry process is cumbersome and inconvenient. In addition, in the case of trend-sensitive commerce, it is necessary to always maintain an up-to-date list on the website due to the nature of it. there is a problem.

In addition, the conventional crawling method is to periodically collect data from the time when the crawling is started. In the case of such a crawling method, it is necessary to accumulate more than a certain amount of data for analysis of meaningful data. In addition, when a single machine periodically collects a specific site, a load occurs on both the collecting site and the collecting site. There are cases where the process crashes beyond the collection unit time.

The present invention collects data from various web sites by a crawling module, while also collecting historical data of web sites from web archive sites, thereby reducing the time for sufficient collection of meaningful data. To provide an apparatus and method, and a recording medium.

In addition, the present invention maintains the basic product information list on the integrated platform up to date, and at the same time reduces excessive or frequent crawls that occur in this process, thereby reducing the resource load of the crawling server and the crawling target system. Data using a web archive It is to provide a crawling apparatus and method, and a recording medium.

In addition, the present invention reduces the crawling burden of the server and reduces the crawling burden of the server by collecting detailed product data by indirectly receiving detailed product information of a website through a user device that has requested detailed product information after crawling only the basic list of products. An object of the present invention is to provide an apparatus and method for crawling data using a web archive that can prevent crawling restrictions and a recording medium.

In addition, the present invention selectively collects detailed product information based on product preference (priority) of the crawling target shopping mall, and collects only detailed information on some products, thereby exhibiting an effect close to collecting detailed information on all products. It is to provide a data crawling apparatus and method using a web archive, and a recording medium.

An apparatus for crawling data using a web archive according to an embodiment of the present invention includes: a crawling module configured to crawl basic list configuration information including title, summary, and creation time from various web sites; a web archive module for collecting past data of the web site from a web archive site that collects and records change data of the web site by time; a database configured to store the basic list configuration information collected from the web site and the historical data collected from the web archive site; and a user target site creation module configured to generate a user target site in which the various web sites are integrated from the basic list configuration information and the past data.

The web archive module may be configured to collect historical data of the web site from the web archive site based on a wayback machine.

A data crawling apparatus using a web archive according to an embodiment of the present invention includes: when a detailed information request is input from a user device in the user target site, the basic list in the target website related to the detailed information request through the user device It may further include an emulator module configured to collect detailed information related to the configuration information and store it in the database.

the emulator module is configured to cause the user device to view the detailed information on the target website; receiving the detailed information collected by browsing the target website in the user device from the user device; And it may be configured to store the detailed information received from the user device in the database in association with the basic list configuration information.

the emulator module, when the detailed information request is input from the user device, determines whether the detailed information is stored in the database; when the detailed information is stored in the database, providing the detailed information stored in the database to the user device through the user target site; when the detailed information is not stored in the database, execute an application program on the user device to access the target website; And it may be configured to collect the detailed information from the target website by an application program running on the user device and store it in the database.

The detailed information may include a detailed image of a product related to the basic list configuration information, product size, weight, material, color, and additional description information.

The emulator module is configured to: determine whether a preference ranking of a user target product to which the detailed information request is input from the user target site, among products included in the target website, satisfies a preset ranking condition; collecting detailed information of the user target product from the target website through the user device when the preference ranking of the user target product satisfies the preset ranking condition; And when the preference ranking of the user target product does not satisfy the preset ranking condition, it may be configured not to collect detailed information of the user target product through the user device.

The crawling module may include: a crawling target analyzer configured to analyze an arrangement pattern of the basic list configuration information for each of the various target web sites, and to designate a crawling target block and properties and fields of the crawling target block based on the arrangement pattern; and a crawler configured to crawl the basic list configuration information based on the crawling target block and attributes and fields of the crawling target block from the various target web sites. The user target site creation module may be configured to integrate product data for various products collected from the various target web sites into a designated object pattern and display it on the user target site.

A data crawling method using a web archive according to an embodiment of the present invention includes: crawling, by a crawling module, basic list configuration information including title, summary, and creation time from various web sites; collecting, by a web archive module, past data of the web site from a web archive site that collects and records change data of the web site by time; storing the basic list configuration information collected from the web site and the past data collected from the web archive site in a database; and generating, by a user target site creation module, a user target site in which the various web sites are integrated from the basic list configuration information and the past data.

A data crawling method using a web archive according to an embodiment of the present invention includes: when a detailed information request is input from a user device in the user target site by an emulator module, a target web related to the detailed information request through the user device The method may further include collecting detailed information related to the basic list configuration information from a site and storing it in the database.

The step of collecting and storing the detailed information in the database includes: allowing the user device to view the detailed information on the target website; receiving the detailed information collected by browsing the target website in the user device from the user device; And it may include the step of linking the detailed information received from the user device with the basic list configuration information to the database.

The collecting and storing of the detailed information in the database may include: when the detailed information request is input from the user device, determining whether the detailed information is stored in the database; providing the detailed information stored in the database to the user device through the user target site when the detailed information is stored in the database; when the detailed information is not stored in the database, executing an application program on the user device to access the target website; and collecting the detailed information from the target website by an application program executed on the user device and storing the detailed information in the database.

The step of collecting the detailed information and storing the detailed information in the database may include: among the products included in the target website, a preference rank of the user target product for which the detailed information request is inputted from the user target site satisfies a preset ranking condition determining whether and when the preference ranking of the user target product satisfies the preset ranking condition, collecting detailed information of the user target product from the target website through the user device.

In the data crawling method using a web archive according to an embodiment of the present invention: If the preference ranking of the user target product does not satisfy the preset ranking condition, detailed information of the user target product is not collected through the user device can prevent it

The crawling of the basic list configuration information includes: analyzing an arrangement pattern of the basic list configuration information for each of the various target websites, and designating a crawling target block and properties and fields of the crawling target block based on the arrangement pattern step; and crawling the basic list configuration information from the various target web sites based on the crawling target block and attributes and fields of the crawling target block. The generating of the user target site may include integrating product data for various products collected from the various target web sites into a designated object pattern and displaying the integrated product data on the user target site.

According to an embodiment of the present invention, there is provided a computer-readable recording medium in which a program for executing a data crawling method utilizing the web archive is recorded.

According to an embodiment of the present invention, by collecting data from various web sites by the crawling module, while also collecting historical data of web sites from the web archive site, a web archive that can shorten the time for sufficient collection of meaningful data A data crawling apparatus and method using the , and a recording medium are provided.

In addition, according to an embodiment of the present invention, it is possible to reduce the resource load of the crawling server and the crawling target system by reducing the excessive or frequent crawling that occurs in this process while maintaining the product basic information list up to date on the integrated platform. An apparatus and method for crawling data using a web archive, and a recording medium are provided.

In addition, according to the embodiment of the present invention, after crawling only the basic list of products, the detailed product information of the website is indirectly transmitted through the user device requesting the detailed product information, and the detailed product data is collected by crawling the server. It can reduce the burden and prevent crawling restrictions by the website.

In addition, according to an embodiment of the present invention, detailed information about products is selectively collected based on product preference (priority) of the shopping mall to be crawled, and detailed information about all products is collected by collecting only detailed information about some products. can have a close effect.

1 is a block diagram of a data crawling apparatus utilizing a web archive according to an embodiment of the present invention.
2 is a block diagram of a crawling module constituting an apparatus for crawling data using a web archive according to an embodiment of the present invention.
3 is a block diagram of a crawling target analyzer constituting a data crawling apparatus using a web archive according to an embodiment of the present invention.
4 is a flowchart of a data crawling method using a web archive according to an embodiment of the present invention.
5 is a flowchart illustrating step S110 of FIG. 4 .
6 is a flowchart illustrating step S160 of FIG. 4 .
7 is a flowchart illustrating another example of step S160 of FIG. 4 .
8 is a flowchart illustrating another example of step S160 of FIG. 4 .

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

In the present specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. As used herein, '~ unit, ~ module' is a unit that processes at least one function or operation, and may refer to, for example, software, FPGA, or hardware component. The functions provided by '~ unit, ~ module' may be performed separately by a plurality of components, or may be integrated with other additional components. 'Part, ~ module' in this specification is not necessarily limited to software or hardware, and may be configured to be in an addressable storage medium, or may be configured to reproduce one or more processors. Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

A data crawling apparatus and method using a web archive according to an embodiment of the present invention collect data from various web sites by a crawling module, while also collecting past data of a web site from a web archive site by a web archive module. , can be provided to shorten the time for sufficient collection of meaningful data.

In the data crawling apparatus and method using a web archive according to an embodiment of the present invention, a crawling server crawls a basic product list configuration for various shopping mall sites, and then, when a user device requests detailed product information from the crawling server, the user By allowing the device to access the target shopping mall site and receiving data obtained by inquiring the target shopping mall site from the user device to the crawling server, only detailed information about the product requested by the user device among various products can be crawled.

Accordingly, according to the data crawling apparatus and method using the web archive according to the embodiment of the present invention, it is possible to always keep a list of basic information about products, such as online shopping malls, etc. up to date on the integrated platform, and at the same time, detailed information System resource load can be reduced by reducing excessive or frequent crawls that may occur in the process of collecting data.

1 is a block diagram of a data crawling apparatus utilizing a web archive according to an embodiment of the present invention. Hereinafter, an apparatus and method for crawling data using a web archive according to an embodiment of the present invention will be described, taking the case of collecting product data in an online shopping mall as an example. However, it is stated in advance that the data crawling apparatus and method using a web archive according to an embodiment of the present invention can be applied to collect various data other than product-related data from various web sites in addition to an online shopping mall.

Referring to FIG. 1 , the data crawling apparatus 100 using a web archive according to an embodiment of the present invention provides basic list configuration information (eg, title, product name, creation date and time, Summary, thumbnail, product price, etc.) are integrated and provided on the user target site, and the product requested by the user among the products can be provided as a crawling server that collects detailed information through the user device 20 have.

The user device 20 may be a terminal of various users who want to inquire, compare, and analyze product information on various shopping mall websites, for example, a laptop such as a smartphone, a smart pad, a tablet PC, a notebook computer, or a desktop computer. It may be a terminal such as a computer, but is not limited thereto.

The basic list configuration information of products may be information included in a web page of a web site provided on a screen displaying schematic information about products of the shopping mall sites 12 , 14 , and 16 . Accordingly, the basic list configuration information of products may be different for each shopping mall site 12 , 14 , and 16 .

The data crawling apparatus 100 using the web archive according to the embodiment of the present invention is first set in the data of the web page displayed on the screen displaying the outline information of the products in each shopping mall site (12, 14, 16). You can collect information corresponding to the list configuration.

The data crawling apparatus 100 using a web archive according to an embodiment of the present invention provides product data to a plurality of users through a user target site, and collects detailed information about the product through a plurality of user devices 20 , It may be provided as a server that provides product information integrated with a plurality of shopping mall sites.

The data crawling apparatus 100 using a web archive according to an embodiment of the present invention includes a crawling module 110 , a web archive module 120 , an emulator module 130 , a user target site creation module 140 , and a database 150 . ) may be included.

The crawling module 110 provides basic list configuration information (eg, title, product name, summary, thumbnail, product price and date of creation, etc.) in various shopping mall sites 12 , 14 , 16 corresponding to the crawling target shopping mall 10 . And, it may be configured to preferentially crawl only basic list configuration information among detailed product information (eg, product detailed image, size, weight, material, color, and additional description of the product for the user).

The crawling target shopping mall 10 may be a shopping mall website having original data including basic list configuration information and detailed information of products, and may be stored on a shopping mall website designated in a server or an administrator terminal (not shown). It may be a shopping mall website designated by the company, but is not limited thereto.

2 is a block diagram of a crawling module constituting an apparatus for crawling data using a web archive according to an embodiment of the present invention. 1 and 2 , the crawling module 110 may include a crawling target analyzer 112 and a crawler 114 .

In an embodiment, the crawling module 110 may crawl basic list configuration information by analyzing a pattern of a document object model for each shopping mall site and a crawling target block to be provided to the user device 20 at the shopping mall site. As an example, the crawling module 110 may include one or several virtual machines.

The crawling target analyzer 112 is configured to analyze the arrangement pattern of basic list configuration information for each various shopping mall sites 12, 14, and 16, and to designate the crawling target block and properties and fields of the crawling target block based on the arrangement pattern. can

The crawling module 110 recognizes the pattern of the document object model for each of the various shopping mall sites 12, 14, and 16, and based on the document object model pattern recognized for each of the shopping mall sites 12, 14, 16, the shopping mall site 12, 14, 16), the necessary information can be extracted from the web page.

The document object model of the shopping mall site may be an interface that allows a program or script to access components in a web page to change content or style. When a web page such as HTML or XML is loaded in the browser, the document object model composes the elements in the document in the form of a tree in the browser.

The crawler 114 may be configured to crawl basic list configuration information based on the crawling target block and the properties and fields of the crawling target block from various target shopping mall sites 12 , 14 , and 16 .

The crawler 114 may perform crawling by accessing each website at a set period. Data crawled by the crawler 114 (basic list configuration information) may be stored in the database 150 .

The basic list configuration information of the product may be additionally collected by an administrator terminal (not shown) in addition to the crawler module 110 . The manager terminal may designate mining blocks, attributes and fields for each of the plurality of shopping mall sites 12, 14, and 16, and may perform URL list management, error log management, authentication management, and the like.

At this time, the administrator uses the manager terminal to designate mining blocks, designate attributes, designate fields, manage URL list, manage errors, based on the structure of web pages collected from the shopping mall sites 12, 14, and 16 Log management, authentication management, etc. can be performed.

An administrator may manage a shopping mall website list, a mining area, a collection cycle, an error log, and the like in the form of a graphic user interface (GUI) by using the manager terminal. The function of the manager terminal may be integrated into a server that performs crawling.

3 is a block diagram of a crawling target analyzer constituting a data crawling apparatus using a web archive according to an embodiment of the present invention. 1 to 3 , the crawling target analyzer 112 may analyze a crawling target for web crawling in a shopping mall site.

The crawling target analyzer 112 includes a mining block designator 1122, an attribute designator 1124, a field designator 1126, a URL list management part 1128, an error log management part 1130, an authentication management part 1132, a pattern It may include a recognizer 1134 , a scrap unit 1136 , a scheduler 1138 , an error handler 1140 , an API interworking module 1142 , and a data storage unit 1144 .

The mining block designation unit 1122 may include a mining block designation UI (User Interface) for designating a mining block for data collected from the shopping mall sites 12 , 14 , and 16 .

The mining block designation unit 1122 may set or designate positions and sizes of various blocks indicating information such as product names, thumbnails, and prices of products in the web pages of the shopping mall sites 12 , 14 , and 16 .

The attribute designator 1124 may include an attribute designation UI for designating an attribute for data collected from the shopping mall sites 12 , 14 , and 16 . The attribute designator 1124 may set or designate attributes of the mining blocks designated by the mining block designator 1122 .

The attribute designator 1124 may, for example, set or designate attributes related to basic list configuration information such as a product name, thumbnail, and price of a product for each web site of the shopping mall sites 12 , 14 , and 16 .

The field designator 1126 may include a field designation UI for designating a field for data collected from a website. The field designator 1126 may designate the fields of the mining blocks to which the attribute is designated by the attribute designator 1124 .

The field designator 1126 may set or designate each field for, for example, a product name, thumbnail, and price corresponding to basic list configuration information of products for each shopping mall website.

The URL list manager 1128 may manage a Uniform Resource Locator (URL) list of web pages for each shopping mall website. The error log management unit 1130 may manage an error log indicating error information when crawling data is collected from a web site.

The authentication manager 1132 may manage authentication information for website access and data designation/management. The authentication information managed by the authentication management unit 1132 may include authentication information of an administrator terminal, authentication information of a server performing crawling, and the like.

The pattern recognizer 1134 may recognize a pattern of the document object model for each web site. The pattern recognizer 1134 may recognize a pattern of the document object model based on mining blocks, properties, and fields designated for each website.

The scrap unit 1136 may extract necessary information based on the pattern of the document object model identified for each web site. The scraper 1136 may include a scraping module for scraping a web page of a web site based on the pattern of the document object model, and a parser module for parsing data of the web page.

The scheduler 1138 may perform a thread management function for crawling data of a web site. The error handler 1140 may perform an error handling function for managing errors in the process of crawling data of a web site.

The API interworking module 1142 may perform an external API interworking function for crawling data of a website. When the web site provides an application program interface (API), the server may perform data inquiry through the API by the API interworking module 1142 .

The data storage unit 1144 may store data crawled by the crawling module 110 (basic product list configuration information) and data (detailed product information) collected by the emulator module 130 in the database 150 .

Referring back to FIG. 1 , the web archive module 120 may collect past data of various web sites from a web archive site that collects and records change data of various web sites by time.

The web archive module 120 may be configured to collect historical data of web sites such as various shopping mall sites from the web archive site based on a wayback machine.

The web archive site may be a site that provides an internet archiving service, and collects and records changes in the shape of the web site to the source code unit by year, month, and day.

The web archive module 120 may collect past data corresponding to a time point at which information on a web site is updated, added, or changed by searching for data in the reverse direction of time from the current time point.

The past data of the web site may be collected by crawling the updated, added, or changed past web page information sequentially in reverse order from the current time point. In this case, in order to increase the crawling speed of the past data, only the past data related to the updated, added or changed part of the web pages may be crawled.

In an embodiment, the web archive module 120 may select a web site to perform web archiving preferentially among various web sites, and sequentially web archive the web sites according to the priority.

The web archive module 120 may perform web archiving, for example, by determining the priority of web sites in the order of the lowest data collection amount for the basic list configuration information. Accordingly, it is possible to reduce the time required to collect meaningful data for various websites.

According to the embodiment of the present invention as described above, additional data is collected from the past shape of the target site recorded in a wayback machine, which is an Internet archiving service, as well as crawling from the current time, until meaningful data collection is achieved. time can be shortened.

The database 150 stores not only product data about various products collected by crawling basic list configuration information for various products from various shopping mall sites 12 , 14 , and 16 , but also a web archive site by the web archive module 120 . may be configured to store historical data of the website collected from

The user target site creation module 140 generates various shopping mall sites 12, 14, and 16 from product data including basic list configuration information stored in the database 150 and past data of the shopping mall site collected using the web archive. You can create a unified user-facing site.

The user target site creation module 140 may be configured to receive a request for detailed information on at least one user target product at the user target site from the user device 20 .

The user target site creation module 140 may be configured to integrate product data for various products collected from the various target shopping mall sites 12 , 14 , and 16 into a designated object pattern and display it on the user target site.

The user target site creation module 140 may integrate data collected from various shopping mall sites 12 , 14 , and 16 to provide a service that is collectively provided to the user device 20 . Accordingly, the user can more quickly and efficiently grasp the information of various shopping mall web sites to perform mutual comparison and analysis.

In an embodiment, the user target site creation module 140 may provide information by unifying related fields (same or similar fields) of various web sites into one field name. In this case, the related fields may be classified based on word similarity between fields of various web sites, thesaurus, and the like.

A data crawling apparatus utilizing a web archive according to an embodiment of the present invention may return data of various web sites to a user at once. Accordingly, users can easily search data at once without moving to multiple sites, compare data from various websites, and find desired data quickly and conveniently.

In addition, if it is configured so that users can upload their own data to the crawling server, it is possible to show not only the data of the website but also the data that is uploaded to the crawling server at once.

When the detailed information request is input from the user device 20 for the user target product created on the user target site, the emulator module 130 performs the user target at the target shopping mall site corresponding to the user target product through the user device 20 . It may be configured to collect detailed product information and store it in the database 150 .

In an embodiment, the emulator module 130 may cause the user device 20 to inquire and view detailed information on a target shopping mall site that has collected product data corresponding to the user target product.

The emulator module 130 executes an application program in the user device 20 to call and inquire detailed product information on the target shopping mall site, and the user device 20 receives the detailed product information from the target shopping mall site As a method of collecting , detailed product information of a target shopping mall site may be crawled.

In addition, the emulator module 130 may receive detailed information of the product for the user who reads the target shopping mall site from the user device 20 from the user device 20 .

The emulator module 130 may be configured to store the detailed information of the product received from the user device 20 in the product data related to the user target product to be stored in the database 150 .

The emulator module 130 may determine whether detailed information of the user target product is stored in the database 150 when a detailed information request for the user target product is input from the user device.

The emulator module 130 may access a detailed page classified as a search target, read data, and transmit the analyzed content to the server by calling a server API. In addition to user devices such as mobile phones, the emulator module 130 can be set in various forms, such as virtual machines or physical devices with its own OS, such as virtual machines (VMs), desktop PCs, workstations, and Raspberry Pi. .

When the detailed information of the user target product is stored in the database 150 , the emulator module 130 may provide the detailed information stored in the database 150 to the user device 20 through the user target site.

When detailed information of the user target product is not stored in the database 150 , the emulator module 130 may execute an application program in the user device 20 to access the target shopping mall site.

Then, the emulator module 130 may be configured to collect detailed information from the target shopping mall site by an application program executed on the user device 20 and store it in the database 150 .

The emulator module 130 may collect detailed product information through distributed parallel processing through a plurality of virtual machines, and may increase the crawling processing speed. In addition, it is possible to secure high speed and reduce costs by running a lightweight mobile OS, which is an environment similar to user devices, in parallel in a virtual environment.

In addition, the emulator module 130 is excellent in terms of crawling speed because distributed calls are made in parallel processing, and it maintains the form of calling the API, so that general users inquire the detailed page on the crawl target service system side. The load can be relieved to a level without difference.

A bottleneck that may occur during collection due to network problems or site speed problems can be improved through distributed processing. That is, the task that needs processing can be divided into small units and distributed to multiple machines or processes so that they can be processed simultaneously or according to each schedule, reducing the workload and increasing the processing speed.

In an embodiment, the emulator module 130 determines a preference rank (priority) of a user target product for which a detailed information request is input, among products included in the target shopping mall sites 12, 14, and 16, and determines the user target product It may be determined whether the preference rank of ' satisfies a preset rank condition.

If it is determined that the preference ranking of the user target product satisfies the preset ranking condition, the emulator module 130 collects detailed information of the user target product from the target shopping mall sites 12 , 14 , 16 through the user device 20 . can do.

If the preference ranking of the user target product does not satisfy the preset ranking condition, the emulator module 130 may be configured not to collect detailed information of the user target product through the user device 20 .

The preference ranking of the user target product may be calculated based on, for example, the product popularity ranking of each shopping mall site, the product exposure order, the product popularity ranking calculated or predicted by the crawling server, and the like, but is not limited thereto.

According to the data crawling apparatus using the web archive according to the embodiment of the present invention described above, while data from various web sites is collected by the crawling module, the past data of the web site is also collected by the web archive module, thereby providing meaningful data It can shorten the time for sufficient collection of

On the other hand, in general, if a website is equipped with a function to prevent an unspecified party from scraping a web page, access to the website when the administrator terminal or server repeatedly attempts to access the website to collect data from the website This can easily be blocked and prevent crawling of website data.

However, according to an embodiment of the present invention, since the server is configured to crawl by indirectly collecting and crawling the data of the website by causing the user device to call data from the website, the crawling of the server is limited by the website. can be prevented

That is, the embodiment of the present invention crawls only the information necessary for the basic list configuration instead of the detailed information that the user does not know whether to inquire when collecting information about the crawling target, and maintains a new list, while maintaining a new list. By analyzing the characteristics of , detailed information can be collected focusing on product information that is judged to have high priority.

Therefore, according to the data crawling apparatus using the web archive according to the embodiment of the present invention, the entire data is collected by collecting some data by efficiently collecting detailed product information based on product preference (priority) using user product access data. can have an effect close to

In addition, according to the embodiment of the present invention, it is possible to reduce the crawling burden of the server while maintaining the basic list configuration of various products, while widening the user's choice, and accessing the web page from the shopping mall site according to the frequent access of the server is possible. blockage can be prevented.

Therefore, according to the embodiment of the present invention, when the user accesses the detailed information of the product, additional information of the target service site is collected through the user device, delivered to the crawling subject system, and utilized when accessing the product, thereby using the user device resource. Thus, the load on the central system can be reduced, and the load on the target service system can also be reduced.

In addition, according to the embodiment of the present invention, since detailed product information is collected according to the request of the user device 20 , the product detailed information request history of the user device 20 may be collected in this process.

Accordingly, it is also possible to preferentially recommend product information suitable for the user by analyzing the product selection propensity data of the user based on the product detailed information request history of the user device 20 and feeding it back to the user target site.

4 is a flowchart of a data crawling method using a web archive according to an embodiment of the present invention. Hereinafter, a data crawling method using a web archive according to an embodiment of the present invention will be described with reference to FIGS. 1 and 4 .

The data crawling method using a web archive according to an embodiment of the present invention may provide a user target site by integrating basic list configuration information of products of the crawling target shopping mall 10 and past data of the crawling target shopping mall 10 into a user target site. . In addition, detailed information of the product requested by the user among the products may be collected through the user device 20 .

In addition, the data crawling method using the web archive according to the embodiment of the present invention provides product data to a plurality of users through a user target site based on basic list information and past data collected from the crawling target shopping mall 10, and , it is possible to additionally collect detailed information of the product through a plurality of user devices (20).

The crawling module 110 provides basic list configuration information (eg, title, product name, summary, thumbnail, product price, creation date and time, etc.) in various shopping mall sites 12 , 14 , 16 corresponding to the crawling target shopping mall 10 . And, only basic list configuration information among detailed product information (eg, detailed product image, size, weight, material, color, and additional description of the product for the user) may be crawled ( S110 ).

In this case, the crawling module 110 may crawl basic list configuration information by analyzing a pattern of a document object model for each shopping mall site and a crawling target block to be provided to the user device 20 at the shopping mall site.

5 is a flowchart illustrating step S110 of FIG. 4 . 1, 4, and 5, the crawling module 110 analyzes the arrangement pattern of basic list configuration information for each of the various shopping mall sites 12, 14, and 16 (S112), and based on the arrangement pattern, a crawling target The properties and fields of the block and the block to be crawled can be specified (S114).

The crawling module 110 recognizes the pattern of the document object model for each of the various shopping mall sites 12, 14, and 16, and based on the document object model pattern recognized for each of the shopping mall sites 12, 14, 16, the shopping mall site 12, 14, 16), the necessary information can be extracted from the web page. The crawling module 110 may crawl the basic list configuration information based on the crawling target block and the properties and fields of the crawling target block ( S116 ).

Referring back to FIGS. 1 and 4 , product data related to various products collected by crawling basic list configuration information for various products from various shopping mall sites 12 , 14 , and 16 may be stored in the database 150 . There is (S120).

The web archive module 120 may collect past data of various web sites from the web archive site that collects and records change data of various web sites by time (S130).

The web archive module 120 may be configured to collect historical data of web sites such as various shopping mall sites from the web archive site based on a wayback machine.

The web archive module 120 may sequentially search for data from the current time in the opposite direction of time to collect past data corresponding to the time when information on the web site is updated, added, or changed.

In an embodiment, the web archive module 120 may select a web site to preferentially perform web archiving among various web sites, and sequentially web archive the web sites according to the priority.

The web archive module 120 may perform web archiving, for example, by determining the priority of web sites in the order of the lowest data collection amount for the basic list configuration information. Accordingly, it is possible to shorten the time required to collect meaningful data for various websites.

According to the embodiment of the present invention as described above, additional data is collected from the past shape of the target site recorded in a wayback machine, which is an Internet archiving service, as well as crawling from the current time, until meaningful data collection is achieved. time can be shortened.

Product data on various products collected by crawling basic list configuration information for various products from various shopping mall sites 12, 14, and 16, respectively, and past data collected from the web archive site by the web archive module 120 are It may be stored in the database 150 .

The user target site creation module 140 generates various shopping mall sites 12, 14, and 16 from product data including basic list configuration information stored in the database 150 and past data of the shopping mall site collected using the web archive. An integrated user target site may be created (S140).

The user target site creation module 140 may be configured to integrate product data for various products collected from the various target shopping mall sites 12 , 14 , and 16 into a designated object pattern and display it on the user target site.

The user target site creation module 140 may provide services that are collectively provided to the user device 20 by integrating data collected from various shopping mall sites 12 , 14 , and 16 . Accordingly, the user can more quickly and efficiently grasp the information of various shopping mall web sites to perform mutual comparison and analysis.

In an embodiment, the user target site creation module 140 may provide information by unifying related fields (same or similar fields) of various web sites into one field name. In this case, the related fields may be classified based on word similarity between fields of various web sites, thesaurus, and the like.

According to the data crawling method using the web archive according to the embodiment of the present invention, data of various web sites can be returned to the user at once, and the users can easily search data at once without moving multiple sites to access the data of multiple web sites. You can compare data and find the data you want quickly and conveniently.

In addition, if it is configured so that users can upload their own data to the server, it is possible to show not only the data of the website but also the data that is uploaded to the server at once.

The user target site creation module 140 may receive a request for detailed information on at least one user target product on the user target site from the user device 20 ( S150 ).

When a request for detailed information on at least one user target product is input from the user device 20 on the user target site, the emulator module 130 collects detailed information of the user target product from the target shopping mall site through the user device 20 . to be stored in the database 150 (S160).

6 is a flowchart illustrating step S160 of FIG. 4 . 1, 4, and 6 , the emulator module 130 may allow the user device 20 to view detailed information on a target shopping mall site that has collected product data corresponding to a user target product (S162). ).

The emulator module 130 executes an application program in the user device 20 to call and inquire detailed product information on the target shopping mall site, and the user device 20 receives the detailed product information from the target shopping mall site As a method of collecting , detailed product information of a target shopping mall site may be crawled.

In addition, the emulator module 130 may receive detailed information of the product for the user who has viewed the target shopping mall site in the user device 20 from the user device 20 ( S164 ).

The emulator module 130 may store detailed information of the product delivered from the user device 20 in the product data related to the user target product and store it in the database 150 (S166).

7 is a flowchart illustrating another example of step S160 of FIG. 4 . 1, 4 and 7, the emulator module 130 determines whether detailed information of the user target product is stored in the database 150 when a detailed information request for the user target product is input from the user device. It can be (S172). In this case, the server may determine whether information requested by the user device is stored in the server through the URL information.

When the detailed information of the user target product is stored in the database 150, the emulator module 130 may provide the detailed information stored in the database 150 to the user device 20 through the user target site (S174, S176).

If the detailed information of the user target product is not stored in the database 150, the emulator module 130 may execute an application program in the user device 20 to access the target shopping mall site (S174, S178) .

Then, the emulator module 130 may collect detailed information from the target shopping mall site by an application program executed on the user device 20 and store it in the database 150 ( S170 ).

8 is a flowchart illustrating another example of step S160 of FIG. 4 . 1, 4, and 8 , the emulator module 130 sets a preference (priority) of a product for which a detailed information request is input, among products included in the target shopping mall sites 12 , 14 , and 16 . ) may be determined (S182), and it may be determined whether the preference ranking of the user target product satisfies a preset ranking condition (S184).

When it is determined that the preference ranking of the user target product satisfies the preset ranking condition, the emulator module 130 collects detailed information of the user target product from the target shopping mall sites 12 , 14 , 16 through the user device 20 . It can be done (S184, S186).

If the preference ranking of the user target product does not satisfy the preset ranking condition, the emulator module 130 may be configured not to collect detailed information of the user target product through the user device 20 (S184, S188).

According to the data crawling method using the web archive according to the embodiment of the present invention described above, while data from various web sites is collected by the crawling module, the past data of the web site is also collected by the web archive module, thereby providing meaningful data It can shorten the time for sufficient collection of

In addition, according to an embodiment of the present invention, since the server is configured to indirectly collect and crawl the data of the website by causing the user device to call data from the website, the crawling of the server is limited by the website. can be prevented

The data crawling method according to an embodiment of the present invention, when collecting information about a crawling target, crawls only the information necessary for configuring the basic list instead of the detailed information that the user does not know whether to inquire, while maintaining a new list, By analyzing the characteristics of the inquired product, detailed information is collected focusing on product information that is judged to have high priority.

Therefore, according to the data crawling method using the web archive according to the embodiment of the present invention, by efficiently collecting detailed product information based on product preference (priority) using user product access data, the entire data is collected by collecting some data. can have an effect close to

In addition, according to the embodiment of the present invention, it is possible to reduce the crawling burden of the server while maintaining the basic list configuration of various products, while widening the user's choice, and accessing the web page from the shopping mall site according to the frequent access of the server is possible. blockage can be prevented.

Therefore, according to an embodiment of the present invention, when a user accesses detailed information of a product, additional information of a target service site is collected through a user device, delivered to a crawling subject system, and utilized when accessing a product, thereby using user device resources Thus, the load on the central system can be reduced, and the load on the target service system can also be reduced. In addition, there is an advantage that the entire crawler is not affected by errors such as data collection failure and data collection delay.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, method, and component described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). Array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers.

The processing device may run an operating system and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, a processing device is sometimes described as being used, but a person of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It will be understood that this may include

For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a Parallel Processor. The software may include a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device.

The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded in the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software.

Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CDROMs and DVDs, and ROM, RAM, and flash memory. Hardware devices specially configured to store and execute program instructions, such as, etc. are included. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result. Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

10: Crawl target shopping mall
12, 14, 16: shopping mall site
20: user device
100: Data crawling device using web archive
110: crawling module
112: crawl target analyzer
114: crawler
120: web archive module
130: emulator module
140: User-targeted site creation module
150: database

Claims (15)

a crawling module, configured to crawl basic listing composition information, including title, summary, and creation time, from various websites;
a web archive module for collecting past data of the web site from a web archive site that collects and records change data of the web site by time;
a database configured to store the basic list configuration information collected from the web site and the historical data collected from the web archive site; and
From the basic listing configuration information collected from the various websites and the past data collected from the web archive site in relation to the various websites, from the basic listing configuration information collected from the various websites and the web archive site A data crawling device using a web archive, comprising a user target site creation module configured to generate a user target site in which the collected historical data is integrated. According to claim 1,
The web archive module is configured to collect past data of the web site from the web archive site based on a wayback machine, a data crawling device using a web archive. According to claim 1,
When a detailed information request is input from a user device at the user target site, detailed information related to the basic list configuration information is collected from a target website related to the detailed information request through the user device and stored in the database A data crawling device using a web archive, further comprising an emulator module. 4. The method of claim 3,
The emulator module is
cause the user device to view the detailed information on the target website;
receiving the detailed information collected by browsing the target website in the user device from the user device; and
Data crawling apparatus using a web archive, configured to store the detailed information received from the user device in the database in conjunction with the basic list configuration information. 5. The method of claim 4,
The emulator module is
when the detailed information request is input from the user device, determining whether the detailed information is stored in the database;
when the detailed information is stored in the database, providing the detailed information stored in the database to the user device through the user target site;
when the detailed information is not stored in the database, execute an application program on the user device to access the target website; and
Data crawling apparatus utilizing a web archive, configured to collect the detailed information from the target website by an application program running on the user device and store it in the database. 4. The method of claim 3,
The detailed information includes a detailed image of the product related to the basic list configuration information, product size, weight, material, color, and additional description information, a data crawling device using a web archive. 4. The method of claim 3,
The emulator module is:
determining whether a preference rank of a user target product to which the detailed information request is inputted from among the products included in the target website satisfies a preset ranking condition;
collecting detailed information of the user target product from the target website through the user device when the preference ranking of the user target product satisfies the preset ranking condition; and
If the preference ranking of the user target product does not satisfy the preset ranking condition, the data crawling apparatus using a web archive is configured not to collect detailed information of the user target product through the user device. 8. The method according to any one of claims 1 to 7,
The crawling module,
a crawling target analyzer configured to analyze an arrangement pattern of the basic list configuration information for each of the various target web sites, and to designate a crawling target block and properties and fields of the crawling target block based on the arrangement pattern; and
a crawler configured to crawl the basic list configuration information based on the crawling target block and the properties and fields of the crawling target block from the various target websites;
The data crawling device using a web archive, wherein the user target site creation module is configured to integrate product data for various products collected from the various target web sites into a specified object pattern and display it on the user target site. crawling, by the crawling module, basic list configuration information including title, summary, and creation time in various web sites;
collecting, by a web archive module, past data of the web site from a web archive site that collects and records change data of the web site by time;
storing the basic list configuration information collected from the web site and the past data collected from the web archive site in a database; and
The basic list collected from the various web sites from the basic list configuration information collected from the various web sites and the historical data collected from the web archive site in relation to the various web sites by the user target site creation module A method for crawling data using a web archive, comprising generating a user target site in which configuration information and the past data collected from the web archive site are integrated. 10. The method of claim 9,
When a detailed information request is input from a user device at the user target site by the emulator module, detailed information related to the basic list configuration information is collected from the target website related to the detailed information request through the user device and stored in the database A method of crawling data using a web archive, further comprising the step of storing. 11. The method of claim 10,
The step of collecting the detailed information and storing it in the database includes:
allowing the user device to view the detailed information on the target website;
receiving the detailed information collected by browsing the target website in the user device from the user device; and
Data crawling method using a web archive, comprising the step of linking the detailed information received from the user device with the basic list configuration information to the database. 12. The method of claim 11,
The step of collecting the detailed information and storing it in the database includes:
determining whether the detailed information is stored in the database when the detailed information request is input from the user device;
providing the detailed information stored in the database to the user device through the user target site when the detailed information is stored in the database;
when the detailed information is not stored in the database, executing an application program on the user device to access the target website; and
Data crawling method using a web archive, comprising the step of collecting the detailed information from the target website by an application program running on the user device and storing the detailed information in the database. 11. The method of claim 10,
The step of collecting the detailed information and storing it in the database includes:
determining whether a preference ranking of a user target product to which the detailed information request is inputted from among the products included in the target website satisfies a preset ranking condition; and
When the preference ranking of the user target product satisfies the preset ranking condition, collecting detailed information of the user target product from the target website through the user device,
When the preference ranking of the user target product does not satisfy the preset ranking condition, the data crawling method using a web archive for not collecting detailed information of the user target product through the user device. 10. The method of claim 9,
The steps of crawling the basic list configuration information include:
analyzing an arrangement pattern of the basic list configuration information for each of the various target web sites, and designating a crawling target block and properties and fields of the crawling target block based on the arrangement pattern; and
Crawling the basic list configuration information from the various target websites based on the crawling target block and the properties and fields of the crawling target block,
The step of creating the user target site includes integrating product data for various products collected from the various target web sites into a designated object pattern and displaying it on the user target site. Data crawling method using a web archive . A computer-readable recording medium in which a program for executing the data crawling method utilizing the web archive according to any one of claims 9 to 14 is recorded.

Images