KR102399642B1 - Apparatus and method for crawling item data using emulator - Google Patents

Abstract

Apparatus and method for crawling product data using an emulator that can reduce the resource load of the crawling server and the crawled system by reducing the excessive or frequent crawling that occurs in the process while keeping the list of basic product information up to date on the integrated platform , a recording medium is disclosed. A crawling device for crawling product data using an emulator according to an embodiment of the present invention is configured to crawl only basic list configuration information including product names, thumbnails, and prices of products on a shopping mall site, and basic list configuration information among detailed product information. module; Create a user target site in which various shopping mall sites are integrated based on the basic list configuration information collected for various products from various shopping mall sites, and input a detailed information request for at least one user target product in the user target site from the user device a user-targeted site creation module configured to receive; and an emulator module configured to collect detailed information of a user target product from a target shopping mall site through the user device and store the collected information in a database.

Description

Apparatus and method for crawling product data using an emulator {APPARATUS AND METHOD FOR CRAWLING ITEM DATA USING EMULATOR}

The present invention relates to an apparatus and method for crawling product data using an emulator, and more particularly, to keep a list of basic product information on an integrated platform up to date and at the same time reduce excessive or frequent crawling that occurs in this process By doing so, it relates to an apparatus and method for crawling product data using an emulator that can reduce the resource load of a crawling server and a crawling target system.

Recently, service providers that provide services such as shopping malls through the Internet can increase user inflow channels and share user pools. . Such meta-services generally collect information from web sites through crawling. The current web site crawling method is implemented by receiving a value called using an application program interface (API) or by directly accessing each page of the web site to collect data within the corresponding page.

Such a conventional web site direct access method has a problem in that access to the web site is easily blocked when frequent access attempts are made within a predetermined time. A website may be set to prevent an unspecified party from scraping its own web page by default, and this has limitations in which access from the website may be blocked or data crawling may be restricted.

In addition, in the prior art, since the user has to go through a process of collecting data by accessing various sites for which a user wants to compare, and comparing and analyzing the data collected from each site, the inquiry process is cumbersome and inconvenient. In addition, in the case of trend-sensitive commerce, it is necessary to always keep an up-to-date list on the website due to the nature of it. there is a problem.

Product data crawling using an emulator that can reduce the resource load of the crawling server and the crawling target system by reducing the excessive or frequent crawling that occurs in the process while keeping the basic product information list up to date on the integrated platform To provide an apparatus and method, and a recording medium.

In addition, the present invention reduces the crawling burden of the server and reduces the crawling burden of the server by collecting detailed product data by indirectly receiving detailed product information of a website through a user device that has requested detailed product information after crawling only the basic list of products. An object of the present invention is to provide an apparatus and method for crawling product data using an emulator capable of preventing crawling restrictions, and a recording medium.

In addition, the present invention selectively collects detailed product information based on product preference (priority) of the crawling target shopping mall, and collects only detailed information on some products, thereby exhibiting an effect close to collecting detailed information on all products. It is to provide an apparatus and method for crawling product data using an emulator, and a recording medium.

The device for crawling product data using an emulator according to an embodiment of the present invention crawls only basic list configuration information including product names, thumbnails, and prices of products in various shopping mall sites, and the basic list configuration information among detailed information of the products a crawling module configured to; a database configured to store product data on the various products collected by crawling the basic list configuration information for various products from the various shopping mall sites, respectively; A user target site is generated from the product data stored in the database, the various shopping mall sites are integrated, and a user target site is created, configured to receive a request for detailed information on at least one user target product from a user device at the user target site module; and when the detailed information request is inputted from the user device for the user target product created on the user target site, the detailed information of the user target product is collected from the target shopping mall site corresponding to the user target product through the user device and an emulator module configured to be stored in the database.

the emulator module causes the user device to view the detailed information on the target shopping mall site that has collected product data corresponding to the user target product; receiving, from the user device, detailed information of the product for the user who has viewed the target shopping mall site in the user device; And it may be configured to include the detailed information received from the user device in the product data related to the user target product to be stored in the database.

The emulator module, when the detailed information request for the user target product from the user device is input, determine whether the detailed information of the user target product is stored in the database; when the detailed information of the user target product is stored in the database, providing the detailed information stored in the database to the user device through the user target site; when the detailed information of the user target product is not stored in the database, execute an application program application on the user device to access the target shopping mall site; And it may be configured to collect detailed information from the target shopping mall site by an application program executed on the user device and store it in the database. The detailed information may include a detailed product image, size, weight, material, color, and additional description of the product for the user.

The crawling module may include: a crawling target analyzer configured to analyze an arrangement pattern of the basic list configuration information for each of the various target shopping mall sites, and to designate a crawling target block and properties and fields of the crawling target block based on the arrangement pattern; and a crawler configured to crawl the basic list configuration information based on the crawling target block and attributes and fields of the crawling target block from the various target shopping mall sites.

The user target site creation module may be configured to integrate product data for the various products collected from the various target shopping mall sites into a designated object pattern and display the integrated product data on the user target site.

The emulator module is configured to: determine a preference ranking of the user target product to which the detailed information request is input, among products included in the target shopping mall site; determining whether the preference ranking of the user target product satisfies a preset ranking condition; collecting detailed information of the user target product from the target shopping mall site through the user device when the preference ranking of the user target product satisfies the preset ranking condition; And when the preference ranking of the user target product does not satisfy the preset ranking condition, it may be configured not to collect detailed information of the user target product through the user device.

The product data crawling method using the emulator according to an embodiment of the present invention includes, by a crawling module, basic list configuration information including product names, thumbnails, and prices of products in various shopping mall sites, and the basic among detailed information of the products crawling only list configuration information; storing product data on the various products collected by crawling the basic list configuration information for various products from the various shopping mall sites in a database; generating, by a user target site creation module, a user target site in which the various shopping mall sites are integrated from the product data stored in the database; receiving, by the user target site creation module, a request for detailed information on at least one user target product at the user target site from a user device; and collecting, by the emulator module, detailed information of the user target product from the target shopping mall site corresponding to the user target product through the user device when the detailed information request for the user target product is input from the user device includes ;

The collecting of the detailed information of the user target product may include: allowing the user device to view the detailed information on the target shopping mall site that has collected product data corresponding to the user target product; receiving, from the user device, detailed information of the product for the user who has read the target shopping mall site from the user device; and storing the detailed information received from the user device in the product data related to the user target product and storing it in the database.

The collecting of the detailed information of the user target product may include, when the detailed information request for the user target product is input from the user device, determining whether the detailed information of the user target product is stored in the database; providing the detailed information stored in the database to the user device through the user target site when the detailed information of the user target product is stored in the database; when the detailed information of the user target product is not stored in the database, executing an application program in the user device to access the target shopping mall site; and collecting detailed information from the target shopping mall site by an application program executed on the user device and storing the detailed information in the database.

The crawling includes analyzing, by a crawling target analyzer, an arrangement pattern of the basic list configuration information for each of the various target shopping mall sites, and designating a crawling target block and properties and fields of the crawling target block based on the arrangement pattern to do; and crawling, by a crawler, the basic list configuration information from the various target shopping mall sites based on the crawling target block and attributes and fields of the crawling target block.

The generating of the user target site may include integrating product data for the various products collected from the various target shopping mall sites into a specified object pattern and displaying the integrated product data on the user target site.

The collecting of the detailed information of the user target product may include: determining a preference rank of the user target product to which the detailed information request is input, among products included in the target shopping mall site; determining whether the preference ranking of the user target product satisfies a preset ranking condition; collecting detailed information of the user target product from the target shopping mall site through the user device when the preference ranking of the user target product satisfies the preset ranking condition; and if the preference ranking of the user target product does not satisfy the preset ranking condition, not collecting detailed information of the user target product through the user device.

According to an embodiment of the present invention, there is provided a computer-readable recording medium in which a program for executing a product data crawling method using the emulator is recorded.

According to an embodiment of the present invention, an emulator capable of reducing the resource load of the crawling server and the crawling target system by reducing the excessive or frequent crawling that occurs in this process while maintaining the product basic information list up to date on the integrated platform A product data crawling apparatus and method, and a recording medium utilized are provided.

In addition, according to the embodiment of the present invention, after crawling only the basic list of products, the detailed product information of the website is indirectly transmitted through the user device requesting the detailed product information, and the detailed product data is collected by crawling the server. It can reduce the burden and prevent crawling restrictions by the website.

In addition, according to an embodiment of the present invention, detailed information about products is selectively collected based on product preference (priority) of the shopping mall to be crawled, and detailed information about all products is collected by collecting only detailed information about some products. can have a close effect.

1 is a block diagram of an apparatus for crawling product data using an emulator according to an embodiment of the present invention.
2 is a block diagram of a crawling module constituting an apparatus for crawling product data using an emulator according to an embodiment of the present invention.
3 is a block diagram of a crawling target analyzer constituting an apparatus for crawling product data using an emulator according to an embodiment of the present invention.
4 is a flowchart of a product data crawling method using an emulator according to an embodiment of the present invention.
5 is a flowchart illustrating step S110 of FIG. 4 .
6 is a flowchart illustrating step S150 of FIG. 4 .
7 is a flowchart illustrating another example of step S150 of FIG. 4 .
8 is a flowchart illustrating another example of step S150 of FIG. 4 .

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

In the present specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. As used herein, '~ unit, ~ module' is a unit that processes at least one function or operation, and may refer to, for example, software, FPGA, or hardware component. The functions provided by '~ unit, ~ module' may be performed separately by a plurality of components, or may be integrated with other additional components. 'Part, ~ module' in this specification is not necessarily limited to software or hardware, and may be configured to be in an addressable storage medium, or may be configured to reproduce one or more processors. Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

Product data crawling apparatus and method using an emulator according to an embodiment of the present invention, while always keeping the basic product information list up to date on an integrated platform, and reducing excessive or frequent crawling that occurs in the process, system resource load provided to reduce

An apparatus and method for crawling product data using an emulator according to an embodiment of the present invention allows the crawling server to crawl only the basic product list configuration for various shopping mall sites, and then, when the user device requests detailed product information from the crawling server, the user device access to the target shopping mall site from the user device, and transmit data obtained by inquiring into the target shopping mall site from the user device to the crawling server, so that only detailed information about the product requested by the user device among various products can be crawled.

1 is a block diagram of an apparatus for crawling product data using an emulator according to an embodiment of the present invention. Referring to FIG. 1 , the product data crawling apparatus 100 using the emulator according to an embodiment of the present invention provides basic list configuration information (product name, thumbnail, and price of the product) of the product of the crawling target shopping mall 10 to the user. It may be provided to a crawling server that integrates and provides the site, and collects detailed information of the product requested by the user among the products through the user device 20 .

The user device 20 may be a terminal of a user who inquires, compares, and analyzes product information from various shopping mall websites, for example, a smartphone, a smart pad, a tablet PC, a laptop such as a notebook computer, and a desktop computer. It may be a terminal such as, but not limited to.

The basic list configuration information of products may be information included in a web page provided on a screen that displays schematic information about products of the shopping mall sites 12 , 14 , and 16 . Accordingly, the basic list configuration information of products may be different for each shopping mall site 12 , 14 , and 16 .

The product data crawling apparatus 100 using the emulator according to the embodiment of the present invention is first set in the data of the web page displayed on the screen displaying the outline information of the product in each shopping mall site 12 , 14 , and 16 . You can collect information corresponding to the list configuration.

The product data crawling apparatus 100 using an emulator according to an embodiment of the present invention provides product data to a plurality of users through a user target site, and collects detailed information of the product through a plurality of user devices 20 It may be provided as a server that provides product information integrated with a plurality of shopping mall sites.

The product data crawling apparatus 100 using an emulator according to an embodiment of the present invention may include a crawling module 110 , an emulator module 120 , a user target site creation module 130 , and a database 140 .

The crawling module 110 includes basic list configuration information including product names, thumbnails, and prices of products in various shopping mall sites 12 , 14 , and 16 corresponding to the crawling target shopping mall 10 , and detailed product information (eg, , product detailed image, size, weight, material, color, and additional description of the user target product) may be configured to crawl only basic list configuration information.

The crawling target shopping mall 10 may be a shopping mall website having original data including basic list configuration information and detailed information of products, and may be stored on a shopping mall website designated in a server or an administrator terminal (not shown). It may be a shopping mall website designated by the company, but is not limited thereto.

2 is a block diagram of a crawling module constituting an apparatus for crawling product data using an emulator according to an embodiment of the present invention. 1 and 2 , the crawling module 110 may include a crawling target analyzer 112 and a crawler 114 .

In an embodiment, the crawling module 110 may crawl basic list configuration information by analyzing a pattern of a document object model for each shopping mall site and a crawling target block to be provided to the user device 20 at the shopping mall site. As an example, the crawling module 110 may include one or several virtual machines.

The crawling target analyzer 112 is configured to analyze the arrangement pattern of basic list configuration information for each various shopping mall sites 12, 14, and 16, and to designate the crawling target block and properties and fields of the crawling target block based on the arrangement pattern. can

The crawling module 110 recognizes the pattern of the document object model for each of the various shopping mall sites 12, 14, and 16, and based on the document object model pattern recognized for each of the shopping mall sites 12, 14, 16, the shopping mall site 12, 14, 16), the necessary information can be extracted from the web page.

The document object model of the shopping mall site may be an interface that allows a program or script to access components in a web page to change content or style. When a web page such as HTML or XML is loaded in the browser, the document object model composes the elements in the document in the form of a tree in the browser.

The crawler 114 may be configured to crawl basic list configuration information based on the crawling target block and the properties and fields of the crawling target block from various target shopping mall sites 12 , 14 , and 16 .

The crawler 114 may perform crawling by accessing each website at a set period. Data crawled by the crawler 114 may be stored in the database 140 .

The basic list configuration information of the product may be collected by an administrator terminal (not shown) in addition to the crawler module 110 . The manager terminal may designate mining blocks, attributes and fields for each of the plurality of shopping mall sites 12, 14, and 16, and may perform URL list management, error log management, authentication management, and the like.

At this time, the administrator uses the manager terminal to designate mining blocks, designate attributes, designate fields, manage URL list, manage errors, based on the structure of web pages collected from the shopping mall sites 12, 14, and 16 Log management, authentication management, etc. can be performed.

An administrator may manage a shopping mall website list, a mining area, a collection cycle, an error log, and the like in the form of a graphic user interface (GUI) by using the manager terminal. The function of the manager terminal may be integrated into a server that performs crawling.

3 is a block diagram of a crawling target analyzer constituting an apparatus for crawling product data using an emulator according to an embodiment of the present invention. 1 to 3 , the crawling target analyzer 112 may analyze a crawling target for web crawling in a shopping mall site.

The crawling target analyzer 112 includes a mining block designator 1122, an attribute designator 1124, a field designator 1126, a URL list management part 1128, an error log management part 1130, an authentication management part 1132, a pattern It may include a recognizer 1134 , a scrap unit 1136 , a scheduler 1138 , an error handler 1140 , an API interworking module 1142 , and a data storage unit 1144 .

The mining block designation unit 1122 may include a mining block designation UI (User Interface) for designating a mining block for data collected from the shopping mall sites 12 , 14 , and 16 .

The mining block designation unit 1122 may set or designate positions and sizes of various blocks indicating information such as product names, thumbnails, and prices of products in the web pages of the shopping mall sites 12 , 14 , and 16 .

The attribute designator 1124 may include an attribute designation UI for designating an attribute for data collected from the shopping mall sites 12 , 14 , and 16 . The attribute designator 1124 may set or designate attributes of the mining blocks designated by the mining block designator 1122 .

The attribute designator 1124 may, for example, set or designate attributes related to basic list configuration information such as a product name, thumbnail, and price of a product for each web site of the shopping mall sites 12 , 14 , and 16 .

The field designator 1126 may include a field designation UI for designating a field for data collected from a website. The field designator 1126 may designate the fields of the mining blocks to which the attribute is designated by the attribute designator 1124 .

The field designator 1126 may set or designate each field for, for example, a product name, thumbnail, and price corresponding to basic list configuration information of products for each shopping mall website.

The URL list manager 1128 may manage a Uniform Resource Locator (URL) list of web pages for each shopping mall website. The error log management unit 1130 may manage an error log indicating error information when crawling data is collected from a web site.

The authentication manager 1132 may manage authentication information for website access and data designation/management. The authentication information managed by the authentication management unit 1132 may include authentication information of an administrator terminal, authentication information of a server performing crawling, and the like.

The pattern recognizer 1134 may recognize a pattern of the document object model for each web site. The pattern recognizer 1134 may recognize a pattern of the document object model based on mining blocks, properties, and fields designated for each website.

The scrap unit 1136 may extract necessary information based on the pattern of the document object model identified for each web site. The scraper 1136 may include a scraping module for scraping a web page of a web site based on the pattern of the document object model, and a parser module for parsing data of the web page.

The scheduler 1138 may perform a thread management function for crawling data of a web site. The error handler 1140 may perform an error handling function for managing errors in the process of crawling data of a web site.

The API interworking module 1142 may perform an external API interworking function for crawling data of a website. When the web site provides an application program interface (API), the server may perform data inquiry through the API by the API interworking module 1142 .

The data storage unit 1144 may store data crawled by the crawling module 110 (basic product list configuration information) and data (detailed product information) collected by the emulator module 120 in the database 140 .

Referring back to FIG. 1 , the database 140 may be configured to store product data regarding various products collected by crawling basic list configuration information for various products from various shopping mall sites 12 , 14 , and 16 , respectively. .

The user target site creation module 130 generates a user target site in which various shopping mall sites 12 , 14 , 16 are integrated from product data stored in the database 140 , and generates at least one user target site from the user device 20 . may be configured to receive a request for detailed information about a user target product of

The user target site creation module 130 may be configured to integrate product data for various products collected from the various target shopping mall sites 12 , 14 , and 16 into a designated object pattern and display it on the user target site.

The user target site creation module 130 may provide services that are collectively provided to the user device 20 by integrating data collected from various shopping mall sites 12 , 14 , and 16 . Accordingly, the user can more quickly and efficiently grasp the information of various shopping mall web sites to perform mutual comparison and analysis.

In an embodiment, the user target site creation module 130 may provide information by unifying related fields (same or similar fields) of various web sites into one field name. In this case, the related fields may be classified based on word similarity between fields of various web sites, thesaurus, and the like.

The device for crawling product data using an emulator according to an embodiment of the present invention may return data of various web sites to the user at once. Accordingly, users can easily search data at once without moving to multiple sites, compare data from various websites, and find desired data quickly and conveniently.

In addition, if it is configured so that users can upload their own data to the crawling server, it is possible to show not only the data of the website but also the data that is uploaded to the crawling server at once.

When the detailed information request is input from the user device 20 for the user target product created on the user target site, the emulator module 120 performs the user target at the target shopping mall site corresponding to the user target product through the user device 20 . It may be configured to collect detailed product information and store it in the database 140 .

In an embodiment, the emulator module 120 may cause the user device 20 to inquire and view detailed information on a target shopping mall site that has collected product data corresponding to the user target product.

The emulator module 120 executes an application program on the user device 20 to call and inquire detailed product information on the target shopping mall site, and the user device 20 receives the detailed product information from the target shopping mall site As a method of collecting , detailed product information of a target shopping mall site may be crawled.

In addition, the emulator module 120 may receive detailed information of the product for the user who reads the target shopping mall site from the user device 20 from the user device 20 .

The emulator module 120 may be configured to store the detailed information of the product received from the user device 20 in the product data related to the user target product to be stored in the database 140 .

When a detailed information request for a user target product is input from the user device, the emulator module 120 may determine whether detailed information of the user target product is stored in the database 140 .

When the detailed information of the user target product is stored in the database 140 , the emulator module 120 may provide the detailed information stored in the database 140 to the user device 20 through the user target site.

When the detailed information of the user target product is not stored in the database 140 , the emulator module 120 may execute an application program in the user device 20 to access the target shopping mall site.

Then, the emulator module 120 may be configured to collect detailed information from the target shopping mall site by an application program executed on the user device 20 and store it in the database 140 .

The emulator module 120 may collect detailed product information through parallel processing through a plurality of virtual machines and may increase the crawling processing speed. In addition, it is possible to secure high speed and reduce costs by running a lightweight mobile OS, which is an environment similar to user devices, in parallel in a virtual environment.

In addition, the emulator module 120 is excellent in terms of crawling speed because distributed calls are made in parallel processing, and it maintains the form of calling the API, which makes it difficult for general users to view detailed pages on the service system side to be crawled. The load can be relieved to a level without difference.

In an embodiment, the emulator module 120 determines a preference rank (priority) of a user target product for which a detailed information request is input, among products included in the target shopping mall sites 12, 14, and 16, and determines the user target product It may be determined whether the preference rank of ' satisfies a preset rank condition.

When it is determined that the preference ranking of the user target product satisfies the preset ranking condition, the emulator module 120 collects detailed information of the user target product from the target shopping mall sites 12 , 14 , 16 through the user device 20 . can do.

If the preference ranking of the user target product does not satisfy the preset ranking condition, the emulator module 120 may be configured not to collect detailed information of the user target product through the user device 20 .

The preference ranking of the user target product may be calculated based on, for example, the product popularity ranking of each shopping mall site, the product exposure order, the product popularity ranking calculated or predicted by the crawling server, and the like, but is not limited thereto.

In general, if the website is equipped with a function to prevent unspecified parties from scraping the web page, when the administrator terminal or server repeatedly tries to access the website to collect data from the website, it is easy to access the website It can be blocked and prevent crawling of data on websites.

However, according to an embodiment of the present invention, since the server is configured to crawl by indirectly collecting and crawling the data of the website by causing the user device to call data from the website, the crawling of the server is limited by the website. can be prevented

That is, the embodiment of the present invention crawls only the information necessary for the basic list configuration instead of the detailed information that the user does not know whether to inquire when collecting information about the crawling target, and maintains a new list, while maintaining a new list. By analyzing the characteristics of the product, detailed information is collected focusing on product information that is judged to have high priority.

Therefore, according to the product data crawling device using the emulator according to the embodiment of the present invention, by efficiently collecting detailed product information based on product preference (priority) using user product access data, the entire data is collected by collecting some data. can have an effect close to

In addition, according to the embodiment of the present invention, it is possible to reduce the crawling burden of the server while maintaining the basic list configuration of various products, while widening the user's choice, and accessing the web page from the shopping mall site according to the frequent access of the server is possible. blockage can be prevented.

Therefore, according to an embodiment of the present invention, when a user accesses detailed information of a product, additional information of a target service site is collected through a user device, delivered to a crawling subject system, and utilized when accessing a product, thereby using user device resources Thus, the load on the central system can be reduced, and the load on the target service system can also be reduced.

In addition, according to the embodiment of the present invention, since detailed product information is collected according to the request of the user device 20 , the product detailed information request history of the user device 20 may be collected in this process.

Accordingly, it is also possible to preferentially recommend product information suitable for the user by analyzing the product selection propensity data of the user based on the product detailed information request history of the user device 20 and feeding it back to the user target site.

4 is a flowchart of a product data crawling method using an emulator according to an embodiment of the present invention. Hereinafter, a product data crawling method using an emulator according to an embodiment of the present invention will be described with reference to FIGS. 1 and 4 .

The product data crawling method using the emulator according to an embodiment of the present invention provides the basic list configuration information of products of the crawling target shopping mall 10 by integrating it into the user target site, and the product requested by the user among the products is detailed This is for collecting information through the user device 20 .

The product data crawling method using the emulator according to an embodiment of the present invention may provide product data to a plurality of users through a user target site, and collect detailed product information through a plurality of user devices 20 .

The crawling module 110 includes basic list configuration information including product names, thumbnails, and prices of products in various shopping mall sites 12 , 14 , and 16 corresponding to the crawling target shopping mall 10 , and detailed product information (eg, , it is possible to crawl only the basic list configuration information among the detailed product image, size, weight, material, color, and additional description of the user target product (S110).

In this case, the crawling module 110 may crawl basic list configuration information by analyzing a pattern of a document object model for each shopping mall site and a crawling target block to be provided to the user device 20 at the shopping mall site.

5 is a flowchart illustrating step S110 of FIG. 4 . 1, 4, and 5, the crawling module 110 analyzes the arrangement pattern of basic list configuration information for each of the various shopping mall sites 12, 14, and 16 (S112), and based on the arrangement pattern, a crawling target The properties and fields of the block and the block to be crawled can be specified (S114).

The crawling module 110 recognizes the pattern of the document object model for each of the various shopping mall sites 12, 14, and 16, and based on the document object model pattern recognized for each of the shopping mall sites 12, 14, 16, the shopping mall site 12, 14, 16), the necessary information can be extracted from the web page. The crawling module 110 may crawl the basic list configuration information based on the crawling target block and the properties and fields of the crawling target block ( S116 ).

Referring back to FIGS. 1 and 4 , product data related to various products collected by crawling basic list configuration information for various products from various shopping mall sites 12 , 14 and 16 may be stored in the database 140 . There is (S120).

The user target site creation module 130 may create a user target site in which various shopping mall sites 12 , 14 , and 16 are integrated from product data stored in the database 140 ( S130 ).

The user target site creation module 130 may be configured to integrate product data for various products collected from the various target shopping mall sites 12 , 14 , and 16 into a designated object pattern and display it on the user target site.

The user target site creation module 130 may provide services that are collectively provided to the user device 20 by integrating data collected from various shopping mall sites 12 , 14 , and 16 . Accordingly, the user can more quickly and efficiently grasp the information of various shopping mall web sites to perform mutual comparison and analysis.

In an embodiment, the user target site creation module 130 may provide information by unifying related fields (same or similar fields) of various web sites into one field name. In this case, the related fields may be classified based on word similarity between fields of various web sites, thesaurus, and the like.

The device for crawling product data using an emulator according to an embodiment of the present invention can return data of various websites to a user at once, and users can easily search data at once without moving multiple sites to view data from multiple websites. You can compare and find the data you want quickly and conveniently.

In addition, if it is configured so that users can upload their own data to the server, it is possible to show not only the data of the website but also the data that is uploaded to the server at once.

When a request for detailed information on at least one user target product is input from the user device 20 on the user target site ( S140 ), the emulator module 120 performs the detailed information of the user target product at the target shopping mall site through the user device 20 . Information may be collected and stored in the database 140 (S150).

6 is a flowchart illustrating step S150 of FIG. 4 . 1, 4, and 6 , the emulator module 120 may allow the user device 20 to view detailed information on a target shopping mall site that has collected product data corresponding to the user target product (S152). ).

The emulator module 120 executes an application program on the user device 20 to call and inquire detailed product information on the target shopping mall site, and the user device 20 receives the detailed product information from the target shopping mall site As a method of collecting , detailed product information of a target shopping mall site may be crawled.

In addition, the emulator module 120 may receive detailed information of the product for the user who has viewed the target shopping mall site in the user device 20 from the user device 20 ( S154 ).

The emulator module 120 may store the detailed information of the product received from the user device 20 in the product data related to the user target product and store it in the database 140 (S156).

7 is a flowchart illustrating another example of step S150 of FIG. 4 . 1, 4 and 7, the emulator module 120 determines whether detailed information of the user target product is stored in the database 140 when a detailed information request for the user target product is input from the user device. It can be (S162). In this case, the server may determine whether information requested by the user device is stored in the server through the URL information.

When the detailed information of the user target product is stored in the database 140, the emulator module 120 may provide the detailed information stored in the database 140 to the user device 20 through the user target site (S164, S166).

When the detailed information of the user target product is not stored in the database 140, the emulator module 120 may execute an application program on the user device 20 to access the target shopping mall site (S164, S168) .

Then, the emulator module 120 may collect detailed information from the target shopping mall site by an application program executed on the user device 20 and store it in the database 140 ( S170 ).

8 is a flowchart illustrating another example of step S150 of FIG. 4 . 1, 4, and 8 , the emulator module 120 determines a preference (priority) of a product for which a detailed information request is input among products included in the target shopping mall sites 12, 14, and 16. ) is determined (S172), and it can be determined whether the preference ranking of the user target product satisfies a preset ranking condition (S174).

When it is determined that the preference ranking of the user target product satisfies the preset ranking condition, the emulator module 120 collects detailed information of the user target product from the target shopping mall sites 12 , 14 , 16 through the user device 20 . You can (S174, S176).

If the preference ranking of the user target product does not satisfy the preset ranking condition, the emulator module 120 may be configured not to collect detailed information of the user target product through the user device 20 (S174, S178).

According to an embodiment of the present invention, since the server is configured to indirectly collect and crawl data of the website by causing the user device to call data from the website, it is possible to prevent the crawling of the server from being restricted by the website. can

The product data crawling method according to an embodiment of the present invention, when collecting information on a crawling target, crawls only the information necessary for the basic list configuration instead of the detailed information that the user does not know whether to inquire while maintaining a new list, By analyzing the characteristics of products that are frequently inquired, detailed information is collected focusing on product information that is judged to have high priority.

Therefore, according to the product data crawling method using the emulator according to the embodiment of the present invention, by efficiently collecting detailed product information based on product preference (priority) using user product access data, the entire data is collected by collecting some data. can have an effect close to

In addition, according to the embodiment of the present invention, it is possible to reduce the crawling burden of the server while maintaining the basic list configuration of various products, while widening the user's choice, and accessing the web page from the shopping mall site according to the frequent access of the server is possible. blockage can be prevented.

Therefore, according to an embodiment of the present invention, when a user accesses detailed information of a product, additional information of a target service site is collected through a user device, delivered to a crawling subject system, and utilized when accessing a product, thereby using user device resources Thus, the load on the central system can be reduced, and the load on the target service system can also be reduced.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). Array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers.

The processing device may run an operating system and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, a processing device is sometimes described as being used, but a person of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It will be understood that this may include

For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a Parallel Processor. The software may include a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device.

The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software.

Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CDROMs and DVDs, and ROM, RAM, and flash memory. Hardware devices specially configured to store and execute program instructions, such as, etc. are included. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result. Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

10: Crawl target shopping mall
12, 14, 16: shopping mall site
20: user device
100: Product data crawling device using emulator
110: crawling module
112: crawl target analyzer
114: crawler
120: emulator module
130: user target site creation module
140: database

Claims (15)

a crawling module configured to crawl only basic list configuration information including product names, thumbnails, and prices of products in various shopping mall sites, and the basic list configuration information among detailed information of the products;
a database configured to store product data on the various products collected by crawling the basic list configuration information for various products from the various shopping mall sites, respectively;
Generating a user target site in which the various shopping mall sites are integrated from the product data stored in the database, and generating a user target site configured to receive a request for detailed information on at least one user target product from a user device at the user target site module; and
When the detailed information request is inputted from the user device for the user target product created on the user target site, detailed information of the user target product is collected from the target shopping mall site corresponding to the user target product through the user device, an emulator module configured to store in the database;
The emulator module is
when the detailed information request is input from the user device, causing the user device to view the detailed information on the target shopping mall site that has collected product data corresponding to the user target product;
receiving, from the user device, detailed information of the product for the user who viewed the target shopping mall site in the user device; And
It is configured to include the detailed information received from the user device in the product data related to the user target product and store it in the database,
The emulator module is
When the detailed information request for the user target product is input from the user device, it is determined whether the detailed information of the user target product is stored in the database;
when the detailed information of the user target product is stored in the database, providing the detailed information stored in the database to the user device through the user target site;
when the detailed information of the user target product is not stored in the database, execute an application program application on the user device to access the target shopping mall site; And
It is configured to collect detailed information from the target shopping mall site by an application program executed on the user device and store it in the database,
The crawling module is
a crawling target analyzer configured to analyze an arrangement pattern of the basic list configuration information for each of the various target shopping mall sites, and to designate a crawling target block and properties and fields of the crawling target block based on the arrangement pattern; and
a crawler configured to crawl the basic list configuration information based on the crawling target block and attributes and fields of the crawling target block from the various target shopping mall sites; and
The emulator module is
determining a preference ranking of the user target product to which the detailed information request is input, among products included in the target shopping mall site;
determining whether the preference ranking of the user target product satisfies a preset ranking condition;
collecting detailed information of the user target product from the target shopping mall site through the user device when the preference ranking of the user target product satisfies the preset ranking condition; And
Product data crawling apparatus using an emulator, configured not to collect detailed information of the user target product through the user device if the preference ranking of the user target product does not satisfy the preset ranking condition. delete delete According to claim 1,
The detailed information includes a detailed product image, size, weight, material, color, and additional description of the product for the user, a product data crawling device using an emulator. delete According to claim 1,
The user target site creation module,
The device for crawling product data using an emulator is configured to integrate product data for the various products collected from the various target shopping mall sites into a specified object pattern and display it on the user target site. delete crawling, by a crawling module, only basic list configuration information including product names, thumbnails, and prices of products in various shopping mall sites, and the basic list configuration information among detailed information of the products;
storing product data on the various products collected by crawling the basic list configuration information for various products from the various shopping mall sites in a database;
generating, by a user target site creation module, a user target site in which the various shopping mall sites are integrated from the product data stored in the database;
receiving, by the user target site creation module, a request for detailed information on at least one user target product at the user target site from a user device; and
collecting, by the emulator module, detailed information of the user target product from a target shopping mall site corresponding to the user target product through the user device when the detailed information request for the user target product is input from the user device; including,
The step of collecting detailed information of the product for the user includes:
allowing the user device to view the detailed information on the target shopping mall site that has collected product data corresponding to the user target product;
receiving, from the user device, detailed information of the product for the user who has read the target shopping mall site from the user device; and
Including; including the detailed information received from the user device in the product data related to the user target product and storing in the database;
The step of collecting detailed information of the product for the user includes:
When the detailed information request is input for the user target product from the user device, determining whether the detailed information of the user target product is stored in the database;
providing the detailed information stored in the database to the user device through the user target site when the detailed information of the user target product is stored in the database;
when the detailed information of the user target product is not stored in the database, executing an application program in the user device to access the target shopping mall site; and
The method further comprising: collecting detailed information from the target shopping mall site by an application program executed on the user device and storing it in the database;
The crawling step is
analyzing, by a crawling target analyzer, an arrangement pattern of the basic list configuration information for each of the various target shopping mall sites, and designating a crawling target block and properties and fields of the crawling target block based on the arrangement pattern; and
Crawling, by a crawler, the basic list configuration information from the various target shopping mall sites based on the crawling target block and attributes and fields of the crawling target block;
The step of collecting detailed information of the product for the user includes:
determining a preference ranking of the user target product to which the detailed information request is input, among products included in the target shopping mall site;
determining whether the preference ranking of the user target product satisfies a preset ranking condition;
collecting detailed information of the user target product from the target shopping mall site through the user device when the preference ranking of the user target product satisfies the preset ranking condition; and
If the preference ranking of the user target product does not satisfy the preset ranking condition, not collecting detailed information of the user target product through the user device; Containing, Product data crawling method using an emulator. delete delete 9. The method of claim 8,
The detailed information includes a detailed product image, size, weight, material, color, and additional description of the product for the user, a product data crawling method using an emulator. delete 9. The method of claim 8,
The step of creating the user target site includes:
Product data crawling method using an emulator, comprising the step of integrating the product data for the various products collected from the various target shopping mall sites into a designated object pattern and displaying it on the user target site. delete A computer-readable recording medium in which a program for executing the product data crawling method using the emulator according to any one of claims 8, 11 and 13 is recorded.

Images