KR102191458B1 - Apparatus and method for scrapping web data - Google Patents

Abstract

A web data scraping system and its operation method are provided. The system simulates the target site designated by the user and renders it in a separate frame. In this process, even sites that do not comply with web standards can be accurately simulated through proxy settings. Users can intuitively and easily select DOM in the frame and create events that control them as data collection rules. This is reflected in the scenario, and web scraping services such as automatic repetition can be provided.

Description

Web data collection device and method {APPARATUS AND METHOD FOR SCRAPPING WEB DATA}

It relates to a web data scraping apparatus and its operation method, and more specifically, to a web data scraping service that can be easily utilized by users without specialized knowledge.

Web scraping (also called "web harvesting", "web data extraction", "web crawling", etc.) is the extraction and acquisition of data from a web site. It can be done by hand, but web scraping software is also available. Existing web scraping software or service sites were difficult to use by general users without specialized knowledge, not developers.

On the other hand, according to the same-origin policy of the World Wide Web Consortium (W3C), documents or scripts loaded from one source are restricted from interacting with resources from other sources. It is difficult to provide a dedicated frame for ping.

In addition, there is a limitation in that standard input/output such as physical keyboard and mouse events in the OS kernel cannot be injected into the web browser during the web scraping process. The existing input method uses the function of forcibly injecting a Javascript event after acquiring resource permission through a Javascript Selector or XPath for a HTML5-based specific tag that manages elements. However, if the function is used maliciously, it may generate excessive traffic in an illegal way, and it may be driven in the same format as a web macro and damage the website. Therefore, in the case of recently developed large portals and web services with excellent security, the method of forcibly injecting events through filtering in a Javascript module or server program is blocked.

Korean Patent Registration No. 10-1125083 (announcement date March 21, 2012) US registered patent US 7,647,351 (announced date January 12, 2010)

According to one side, a method of operating a web data scraping device is provided. The method includes: simulating a target site designated by a user and rendering it on a target site display unit; Performing graphic processing on a first DOM selected by a user among at least one Document Object Model (DOM) included in the rendered result to be distinguished from other objects; And displaying on the input event display unit in the form of a menu through a relay module that transmits at least one physical input event of the OS area applicable to the first DOM to an application higher than the OS area.

According to an embodiment, the method adds a first scenario line corresponding to a rule for applying a first physical event selected by a user to the first DOM among the displayed physical input events of the at least one OS area to the data collection rule display unit. It may further include a step of reflecting in the scenario.

In addition, the method may further include a scenario execution step of generating a physical input event in an application higher than the OS area through the relay module when executing a physical input event in the OS area among the created scenarios.

Meanwhile, in the scenario creation or execution step, it may include determining whether the DOM corresponding to the DOM coordinates of the physical input event of the OS area is an object controllable through the relay module.

According to an embodiment, the method may include that the upper application is a web browser or a big data collection program.

1 is a block diagram showing the configuration of a web data collection system according to an embodiment.
2 is a diagram illustrating a screen configuration provided to a user by a web data collection system according to an embodiment.
3 is a diagram illustrating a process of selecting a DOM in a web data collection system according to an exemplary embodiment.
4 is a diagram illustrating a process of providing an event menu corresponding to a selected DOM according to an embodiment.
5 is a diagram for explaining setting and editing of a scenario according to an embodiment.
6 is a diagram for describing a process of providing a physical text input event to an event menu corresponding to a selected DOM according to an embodiment.
7 is a diagram illustrating a process of providing a physical element addition event to an event menu corresponding to a selected DOM according to an embodiment.
8 is a diagram illustrating a process of setting a web data proxy and modifying a path according to an embodiment.
9 is a diagram illustrating an operation flow of a web data collection system according to an embodiment.
10 is a diagram illustrating a configuration of a data collection robot according to an embodiment.
11 is a diagram for describing a process of providing an event through a physical input to a higher level application according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the rights is not limited or limited by these embodiments. The same reference numerals in each drawing indicate the same members.

The terms used in the description below have been selected as general and universal in the related technology field, but there may be other terms depending on the development and/or change of technology, customs, preferences of technicians, and the like. Therefore, terms used in the following description should not be understood as limiting the technical idea, but should be understood as exemplary terms for describing embodiments.

In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, detailed meanings will be described in the corresponding description. Therefore, terms used in the following description should be understood based on the meaning of the term and the contents throughout the specification, not just the name of the term.

Web scraping interface

Software and/or web services that perform web scraping remained in the domain of developers. General users without specialized knowledge could not design code for web scraping. According to embodiments, a user intuitively selects such a web scraping object based on a graphic user interface, and sets an event to be performed on the selected object. Setting a series of events for each of one or more objects is referred to as scraping scenario configuration. According to embodiments, a user can intuitively and easily configure a scenario. In addition, if you set the cycle to execute the scenario, the robot automatically performs web scraping according to the same scenario in each cycle. Therefore, a user can create a web scraping scenario as if recording a macro service in an editing application, and web scraping can be automatically performed by setting a set period and/or repetition number.

DOM control of web scraping software

In the web scraping interface according to the embodiment, an object to be applied to an event is referred to as a document object model (hereinafter referred to as'DOM'). DOM is an application programming interface (API) that processes html and xml documents. Html is a document composed of numerous tags such as head, body, table, tr, td, etc., and all of these tags are made to create a visual representation that users can see. DOM refers to a part of a web document made up of this tag, and the user selects it with a user interfacing device (eg, keyboard, mouse, touch screen, etc.), and selects some action (e.g., keystroke, mouse click, Drag, touch, etc., but are not limited thereto). According to embodiments, a web screen is implemented in an i-frame, which is a separate browsing frame, while providing a user to select and control an individual DOM. When a specific DOM is selected, multiple events that can be applied to the DOM are provided on the menu screen on the right, and when the user selects one or more of them and completes the event to be applied to a specific DOM, the event is It is added as a scenario line. Hereinafter, embodiments will be described in more detail with reference to the drawings.

Example system configuration

The system according to an embodiment may be configured as shown in FIG. 1. The web data collection system includes a web data search area 100 and a web data collection area 140. The web data search area 100 includes a target site management unit 110, a data collection rule management unit 120, a control server 130, and a data collection robot 150. The web data collection area 140 includes a data collection rule generation unit 180 and a target site setting unit 190.

The target site management unit 110 receives information on the target site as a request signal 170 from the target site setting unit 190 of the web data collection area 140 and stores it, and when necessary, the information on the target site Can provide. In addition, the data collection rule management unit 120 receives information about the data collection rule as a request signal 170 from the data collection rule generation unit 180 of the web data collection area 140 and stores it, and when necessary, You can provide information about collection rules. The control server 130 may receive information on a target site from the target site management unit 110 and control a data collection procedure to download data from the target site. The control server 130 will be described in detail in FIG. 2.

The data collection robot 150 receives information on the target site from the target site management unit 110, receives information on the data collection rule from the data collection rule management unit 120, and periodically and/or It is possible to repeatedly collect target data from target sites according to data collection rules.

The data collection rule generation unit 180 may generate a data collection rule through a mouse click, a keyboard input, or the like using a user interface of a web data collection system provided by a user. The data collection rule generation unit 180 receives an event designated by the user for the DOM selected by the user through a graphic user interface. Although there may be one such event, preferably a plurality of events can be generated as a scenario with a series of time flows. The user may set a target site by vomiting a dedicated web and/or app. In this case, the target site setting unit 190 processes this setting. The unit 190 may set a target site for collecting web data by using a user interface of a web data collection system provided by the user.

The block diagram shown in FIG. 1 may mean classification of physical elements, but may mean logical division rather than physical division. Accordingly, it is also possible for one or more units to be implemented together by substantially the same hardware element. Therefore, it should be understood that the exemplary block diagram is not limited to FIG. 1, and configurations that perform substantially the same function are included in the scope represented by the invention.

Implementation of'i-frame'

According to an embodiment, a web page to be scrapped (hereinafter, it may be referred to as a'target page') is not a general-purpose web browser itself, but a separate frame provided through it (the'i-frame' throughout the specification. Rendered in). A web data collection system frame 200 is provided in order to render a target page in an i-frame and enable DOM selection and control event selection. 2 is a diagram illustrating an implementation frame 200 of a web data collection system according to an embodiment. The frame 200 is a graphical user interface that allows a user to easily and intuitively select/control a target site and a DOM included therein, and select an event accordingly to create/manage a web data collection scenario according to embodiments. As a set of, it may be web-based, but is not limited thereto. For example, the frame 200 may be provided by an app-based dedicated program. Even if there is no particular mention in the following, the form or provision method of the frame 200 is limited to some exemplified applications and should not be understood.

The frame 200, which enables a user to provide web data collection based on a graphic, includes a target site display unit 210, an input event display unit 220, and a data collection rule display unit 230. By way of example, but not limited to, the target site display unit 210 may be the aforementioned i-frame. The web data collection system is based on a graphical user interface. The target site display unit 210 may recognize a target site for collecting web data as a converted path through a proxy server and appear on the screen. A proxy server and a proxy site using the same will be described in detail later. The target site display unit 210, which is an i-frame, simulates and renders a target site, displays a DOM selectable from the target site, and provides a graphic that can be classified and selected. Such a graphic may provide a graphic such as highlighting a corresponding DOM or displaying an outline when a mouse-over is performed, for example. For the selected DOM, it is also possible to provide graphical processing that is distinct from the others temporarily or continuously until another DOM is selected.

The input event display unit 220 may be a menu window including a plurality of sub-menus. The input event display unit 220 may display data collection rule events applicable to the DOM selected by the user. For example, input events such as a corresponding mouse click, drag, keyboard input, and touch may be indicated. In addition, it can include clicking on physical elements and entering physical text. It is possible to provide all events regardless of the selected DOM, but preferably only those applicable to the selected DOM can be displayed. In addition, it is possible to display the entire event menu, but enable selection by activating only applicable to the selected DOM, and inactivating other event menus so that they are not selected. When the user selects a specific event appearing on the target site display unit 210, this is generated as a rule that applies the event to the selected DOM, and this may be reflected as part of a scenario. For example, when DOM selection and event selection are performed, one event line may be added to the scenario. In this way, the user can select the DOM and intuitively select the event to be applied to it, and the actual code line for it is automatically generated, so even if the user does not have expertise in html or computing code for web scraping, the web data collection scenario Can be created.

As described above, the data collection rule display unit 230 may be a scenario display unit that displays the flow of individual processing applying a specific event to a specific DOM in time series. When the user sees the target site rendered in the i-frame, selects the DOM included in it, and selects an event from the menu, it is added one by one as an individual rule to the scenario. Scenarios displayed on the data collection rule display unit 230 can be subsequently edited, such as deletion, addition, editing, order change, etc. for each line and/or related line group. It is understood that the types of such edits are not limited to the illustrated ones, and others are included if necessary to improve data collection efficiency in the process of scraping web data and/or to improve some errors or as user needs change. Should be. The editability of the scenario itself provided according to the embodiments has a great advantage because it provides the flexibility to intuitively change the web data collection rules desired by the non-developers even if they do not understand the source code.

The rendering of the target site, menu processing for the selected DOM, creation or editing of scenarios, etc. will be described in more detail with reference to subsequent drawings.

Target site display and DOM selection

3 to 5 show a frame layout presented to a user by a web data collection system according to an embodiment, where a user selects a DOM while viewing a target site and a DOM included therein, and creates an event rule to be applied thereto to create a scenario. It shows a series of processes to go.

In FIG. 3, the screen that the user encounters includes a target site display unit 300 also called an i-frame, an input event display unit 320 also called a menu window, and a data collection rule display unit 330 called a scenario window. When the user selects a target site, such as "http://www.goole.com", the target site is rendered in the i-frame. According to an embodiment, the general-purpose browser itself does not browse the actual target site, but renders a proxy site that simulates the target site. This has the effect of bypassing the W3C's same-origin policy, which will be described later, and enabling site rendering, DOM selection, and data collection for sites that do not comply with web standards.

The target site can contain multiple DOMs. For example, individual DOMs, such as objects that are presented in the form of text, image, or flash, have hyperlinks, and text input windows for inputting queries.

When a user takes an action to select a DOM, such as hovering over the target site display unit 300, which is an i-frame, hovering an input pen, touching or clicking, the system according to an embodiment is currently reserved. Graphics can be processed so that the selected DOM can be distinguished from other parts. As described above, such graphic processing can be various things such as highlight processing, outline display, color or shade change, flickering, etc., and should be understood as an arbitrary method of graphic processing that can distinguish the currently selected DOM from others. . In the illustrated example, the DOM, which is a text input window for inputting a text query, is selected, and an outline 310 of the selected DOM is displayed. Therefore, users can intuitively recognize which DOM they have selected.

Event menu applicable to the selected DOM

As shown in FIG. 4, when the text input window DOM to be graphic processed 410 is selected in the i-frame 400, the input event display unit 420, which is a menu window, displays a plurality of event menus applicable to the selected DOM. Is provided. If the user selects the text input event 440 as shown, a rule to input text for the DOM is generated, and a scenario line 450 is generated on the data collection rule display unit 430 which is a scenario window. An edit button 460 for deleting a corresponding individual event is provided in the scenario line, so that the user can undo the creation of the scenario line. In addition, although not shown, editing buttons such as changing the order of scenario events or copying and reproducing specific events may be provided.

Additional settings within the scenario line

As illustrated in FIG. 5, in the scenario window 530, a user may additionally input specific text for a text input scenario. If the event selected by the user is text input, this is the process of subsequent input/setting of what text to put. For example, if the user inputs text to the DOM as "KBS 9 o'clock news", the user can input it in the corresponding input window 540. Then, the user selects the search icon DOM 510 in the i-frame 500. Then, events applicable to the selected DOM are displayed on the menu 520 window 520, and the user can select a specific event 550 called'element click'. Then, the scenario line 560 for the DOM is added to the scenario window 530 again. Through this process, the user can intuitively and easily compose/edit the scenario for web data scraping.

OS area physical element additional setting

As shown in FIGS. 6 and 7, it is also possible to add a scenario through a physical input in the OS area. The element click and text input of Figs. 4 and 5 use a function of forcibly injecting an event after acquiring resource rights through a selector or XPath of Javascript for a HTML5-based specific tag that manages the DOM. This is different from the physical input method in that the event is executed in a web browser that is an application higher than the OS kernel, not input by a user in the OS area. As for the method of executing an event by clicking on a general element or inputting text, web services with excellent security have recently blocked the execution of the event. However, the method of clicking a physical element and inputting text is an input of an OS area, not an input of a web browser, and cannot be blocked in the web service. A method of adding and executing a scenario through physical input will be described in detail in FIG. 11.

As shown in Fig. 6, when the text input window DOM that is graphic processed 610 in the i-frame 600 is physically selected, the input event display unit 620, which is a menu window, displays a plurality of events applicable to the selected DOM. A menu is provided. When the user selects the physical text input event 640 as shown, a rule to input text for the corresponding DOM is generated, and a scenario line 650 is generated on the data collection rule display unit 630 which is a scenario window. An edit button 660 for deleting a corresponding individual event is provided in the scenario line, so that the user can undo the creation of the scenario line. In addition, although not shown, editing buttons such as changing the order of scenario events or copying and reproducing specific events may be provided.

As illustrated in FIG. 7, in the scenario window 730, a user may additionally input specific text for a text input scenario. If the event selected by the user is text input, this is a process of subsequent input/setting of what text to put. For example, if the user inputs text into the DOM as "KBS 9 o'clock news", the user can input it into the corresponding input window 740. Then, the user selects the search icon DOM 710 in the i-frame 700. Then, events applicable to the selected DOM are displayed on the menu 720 window 720, and the user can select a specific event 750 called'clicking a physical element'. Then, a scenario line 760 for the DOM is added to the scenario window 730 again. Through this process, the user can intuitively and easily compose/edit the scenario for web data scraping.

Overcoming W3C's same-origin policy and rendering target sites

According to an embodiment, the system does not browse the target site itself in the i-frame, downloads the target site, sets it as a proxy site, and renders the proxy site in the i-frame. In this process, the user may not be able to distinguish whether the rendered page is an actual target page or a copy page. In this sense, this processing can be referred to as a target site simulation within an i-frame.

There are several reasons why this embodiment is provided. First, it is to overcome the W3C's same-origin policy. The same-origin policy of the World Wide Web Consortium (W3C) stipulates restrictions on which documents or scripts loaded from one source cannot interact with resources from other sources. . Therefore, it is difficult to provide a dedicated frame for web scraping, and hyperlinks, Flash, or Java-based scripting may not be normally rendered in a separate copy frame. In addition, the fact that the target site does not comply with the web standard may be the reason why the target site is not normally copied in the frame for web scraping. In particular, web sites serviced in Korea (eg, portal sites or media content providing sites) do not comply with web standards in many cases. Therefore, existing web crawler bots often show errors in simulating these web sites and collecting data. Another cause can also be a script that takes a loop regression, such as self-referencing within a DOM object.

For any reason, in the existing web crawling services, the target site was not properly copied, and all DOMs were not displayed. However, the system according to the embodiments simulates the target site through proxy settings to prevent such errors. 8 illustrates a process of setting a web data proxy path and modifying a link path according to an embodiment.

PROXY site/page settings

According to an embodiment, as shown in FIG. 8, the control server 820 includes a proxy dynamic setting unit 824, an HTML downloader 822, an HTML parser 826, and an HTML converter 828. The proxy dynamic setting unit 824 may perform a proxy server 830 and a proxy dynamic setting 835 in order to download HTML resources from a target site. The HTML downloader 822 may download HTML resources necessary for controlling the target site in the web data collection system. During this download process, some objects may not be downloaded for various reasons such as the same-origin policy described above. According to one embodiment to be described later, the incompleteness of the download is overcome through various alternatives and the target page is accurately copied.

The HTML parser 826 parses the path of the HTML resource, does not recognize the downloaded HTML resource as it is in the web data collection system, and proxy the resource path on the HTML resource through the proxy server 830. It is converted into a path, and the target site is recognized through the converted path and displayed on the screen. Through this, the target site recognized through the converted path is displayed on the output screen of the web data collection system. Therefore, a state in which the user can select and control the DOM while viewing the target site in the same web data collection system is provided.

The HTML converter 828 may convert a resource path on an HTML resource into a proxy path. It describes in detail the conversion of the HTML resource path of the target page through the proxy server. By way of example, but not limitation, the system may receive the URL of the target site page, set it in the proxy server for the corresponding host, and proceed with key generation. This key is an identifier that identifies individual pages newly set in the proxy server corresponding to the corresponding URL, and when a key is attached to the proxy site path, a proxy URL dedicated to the target site can be generated. For example, when'http://www.A.com' is a target site, a key for'http://www.A.com' may be generated. For example, if the key is "abc", then "http://proxy.B.com/abc" may be a dedicated proxy URL for the target site "http://www.A.com". In this case, B may be a host of a web data collection system that provides a service according to an embodiment.

Meanwhile, correction of the proxy path of web data will be described. The proxy dynamic setting unit 824 of the control server 820 may perform a proxy dynamic setting 835 with the proxy server 830 in order to download HTML resources from a target site. The HTML downloader 820 may access the target site 860 850 and download 840 HTML resources from the target site 860. This downloaded HTML resource can be used to control the target site in the web data collection system. The HTML parser 830 parses the path of the downloaded HTML resource, does not recognize the downloaded HTML resource in the web data collection system as it is, and converts the resource path on the HTML resource into a proxy path through the proxy server 830 The target site is recognized through the converted path and displayed on the screen of the web data collection system. The resource path of the HTML resource may be converted into a proxy path through the HTML converter 828. The HTML resource path converted to the proxy path may be transmitted to the web data collection system (810). The web data collection system recognizes the target site through the HTML resource path converted to the proxy path, displays the target site screen on the target site display unit 805 of the web data collection system frame 800, and allows the user to enter the web data collection system. You can make it accessible.

Resource saving by preventing duplicate setting of PROXY path

Meanwhile, according to an embodiment, the proxy URL generated once may be reused. If different users create each proxy URL for the same target URL, this may be counter to the efficient use of resources. Therefore, if a key is assigned to a proxy URL that has been created once, it is managed separately to prevent duplicate proxy settings for the same target URL. For example, if the first user sets'http://www.C.com' as the target site and the key is set to "def" for this, this key is managed, and then the second user is' When'http://www.C.com' is set as the target site, the managed key is checked so that the target site of the second user can recycle the preset proxy URL. Therefore, server resources can be greatly reduced through such consideration.

System operation flow for scenario creation

9 is a flowchart illustrating an operation of a web data collection system according to an embodiment. The web data collection system includes a target site path correction step 900, a user's web data collection system control step 920, and a data collection step 940. First, in step 903 of the target site path correction step 900, a user may access a target site for which data is to be collected through a web data collection system, and a URL of the target site may be designated. Then, in step 906, the control server of the search area of the web data collection system may download the HTML resource of the target site designated by the user through the HTML downloader. Then, in step 909, after the control server dynamically sets the proxy, the downloaded HTML resource path may be converted into a proxy path using an HMTL parser and an HTML converter.

In addition, the target site recognized as the converted path in step 926 of the web data collection system control step 920 may be output to the target site display unit of the web data collection system frame so that the user can control it. Then, in step 924, the web data collection system frame provides a DOM event control environment in which a user can create a web data collection rule through a target site display unit, an input event display unit, and a data collection rule display unit. In step 922, the user may control the DOM using a user interface in the provided DOM control environment and select an input event to generate a data collection rule.

Next, in step 942 of the data collection step 940, the collection area of the web data collection system generates a data collection rule database for the data collection rule created by the user. In step 946, data may be searched periodically and repeatedly according to a data collection rule generated by a user. After this process, in step 948, the HTML resource of the result page for the searched data may be downloaded. Then, for the HTML resource downloaded in step 950, the path of the HTML resource may be converted according to the file type of resources such as images and css.

In addition, the result page may be reproduced with the path converted in step 928 of the web data collection system control step 920 and output to the web data collection system frame to provide the data collection result to the user.

Data collection bot behavior

10 is a diagram showing the configuration of a data collection robot according to an embodiment. The data collection bot 1000 may include a search performing robot 1010 and a data storage 1020. The search performing robot 1010 may periodically and repeatedly collect web data at predetermined time intervals according to a generated data collection rule. The collected web data may be stored in the data storage 1020.

Upon receiving the data output request 1040 from the user terminal 1050, web data collected and stored in the data storage 1020 may be immediately fetched and transmitted 1030 to the user terminal 1050.

Event occurrence through physical input of OS area

11 is a diagram showing a process in which a physical input such as a monitor or a keyboard in an OS area is executed by a web browser. First, when a physical input is generated on the web browser, the event occurrence coordinates and the event type are transmitted to the relay module 1130 (1160). The relay module functions to control each other through communication between the OS and a web browser, which is an upper application. When transmitted from the relay module to the application program 1140, the standard input/output event generating module 1150 in the application program requests the occurrence of an event according to the physical input by using a system call. Thereafter, an event is generated based on the event occurrence coordinates transmitted to the standard input/output event generation module (1170). When a physical input event is executed in the OS area, the event control module 1120 may further include selecting a type of the physical input event and checking whether a DOM corresponding to the event is controllable.

The process of generating an event through the relay module is similar to the known macro program that was input only at the OS level by turning on the web browser, but the known macro program operates based on specific coordinates, so the stored coordinates must be used. On the other hand, the event generation process is different in that the OS and the web browser are controlled through the relay module in that the coordinate values of the DOM tagged on the browser can be accurately recognized and a physical event value can be forcibly injected into the corresponding coordinates. In addition, it is not a method of using fixed coordinates, but because different dynamic coordinates are used each time depending on the scenario, it is possible to input various event values in various scenarios.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It can be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodyed in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

Although the embodiments have been described by the limited drawings, various modifications and variations are possible from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the claims to be described later.

Claims (5)

In the method of operating a web data scraping device, the device:
Simulating a target site designated by a user and rendering it on a target site display unit;
Performing graphic processing on a first DOM selected by a user among at least one Document Object Model (DOM) included in the rendered result to be distinguished from other objects; And
Displaying on the input event display unit in the form of a menu through a relay module that transmits at least one OS area physical input event applicable to the first DOM to a web browser that is an application higher than the OS area
Including,
The displaying step,
Transmitting coordinates of event occurrence to the relay module in response to a physical input being generated in the web browser;
Requesting the occurrence of an event according to the physical input by using a system call to a standard input/output event generation module in the application program when the event occurrence coordinates are transmitted from the relay module to an application program in the OS area; And
Generating a physical input event using the event occurrence coordinates transmitted to the standard input/output event generation module
Including
Way.
The method of claim 1,
Adding a first scenario line corresponding to a rule for applying a first physical event selected by a user to a first DOM among the displayed at least one physical input event of the OS area to the data collection rule display unit and reflecting it to the scenario
How to further include.
The method of claim 2,
In the case of executing a physical input event in the OS area among the created scenarios, a scenario execution step of generating a physical input event in an application higher than the OS area through the relay module
How to further include.
The method of claim 3,
The scenario execution step,
Determining whether the DOM corresponding to the DOM coordinates of the OS area physical input event is an object controllable through the relay module
How to include.
The method of claim 4,
The upper application method includes a web browser or a big data collection program.

Images