KR102436357B1 - Web scraping method and system using method of making web scraping script - Google Patents

Abstract

The present invention relates to a web scraping method using a web scraping system, wherein according to one embodiment, disclosed is the web scraping method comprising: a step of generating a job script comprising a command enabling one or more web scrapers connected through a communication network to scrape a digital content of a target site; a step of generating a target site subject to web scraping and a job schedule specifying an execution schedule of web scraping; a step of generating a job to be transmitted to each web scraper according to the job schedule; and a step of transmitting the job to each web scraper based on a job status of each web scraper. Therefore, the present invention is capable of more conveniently and efficiently perform an execution and script management of a web scraping job.

Description

{Web scraping method and system using method of making web scraping script}

The present invention relates to a web scraping method and system, and more particularly, to a method for generating a script for a web scraping operation, and a web scraping system for performing web scraping using the generated script.

In general, shipping companies operate a site where they can post schedules for ships they own (eg, container ships) and make reservations. There are many shipping companies in Korea, and each shipping company separately operates a site that provides schedules for multiple ships it owns. It is very inconvenient to visit each shipping site to find the most appropriate one.

Meanwhile, a web scraping technology that collects specific data from a web page is sometimes used in order to solve the inconvenience of such information being scattered on several sites. However, for web scraping work, for each target (target site) to be scraped, an expert (developer) who is proficient in each programming language had to write a script suitable for each target site. However, writing a script for each target site is very difficult, difficult, and time-consuming for general users.

Patent Document 1: Korean Patent Registration No. 10-1982085 (Announced on May 27, 2019) Patent Document 2: Korean Patent Publication No. 10-2021-0117039 (published on September 28, 2021)

An object of the present invention is to provide a web scraping method and system for scraping information scattered on a plurality of target sites and providing it in one integrated web server.

Another object of the present invention is to provide a script generation module that enables non-professional developers to easily and conveniently write scripts used for web scraping.

According to an embodiment of the present invention, there is provided a web scraping method using a web scraping system, the method comprising: generating a work script including a command for scraping digital content of a target site by one or more web scrapers connected through a communication network; generating a job schedule specifying a target site to be subjected to web scraping and an execution schedule of web scraping; generating a job to be delivered to each web scraper according to the job schedule; and transmitting the job to each web scraper based on the job status of each web scraper.

According to an embodiment of the present invention, there is disclosed a computer-readable recording medium in which a computer program for executing the web scraping method is recorded.

According to an embodiment of the present invention, even a non-professional developer can easily and conveniently write, modify, and change a script used for web scraping, so that the execution and script management of the web scraping operation can be performed more conveniently and efficiently.

1 is a diagram illustrating a network configuration including a web scraping system according to an embodiment of the present invention;
2 is a diagram illustrating a web scraping system according to an embodiment;
3 is a diagram illustrating an operational relationship between a management node and a work node according to an embodiment;
4 is a view for explaining a user interface of a script generating program according to an embodiment;
5 is a view for explaining an operation of generating a script using a script generating program according to an embodiment;
6 is a diagram illustrating a web browser screen displaying an exemplary target site;
7 is a diagram illustrating an exemplary operation list in a script generating program;
8 is a view for explaining a user interface of a work schedule generating program according to an embodiment;
9 is a view for explaining an operation of a work schedule generating program according to an embodiment.

The above objects, other objects, features and advantages of the present invention will be easily understood through the following preferred embodiments in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided so that the disclosed subject matter may be thorough and complete, and that the spirit of the present invention may be sufficiently conveyed to those skilled in the art.

In this specification, when terms such as first, second, etc. are used to describe components, these components should not be limited by these terms. These terms are only used to distinguish one component from another. The embodiments described and illustrated herein also include complementary embodiments thereof.

In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. The expressions 'comprising', 'consisting of', and 'consisting of' as used in the specification do not exclude the presence or addition of one or more other elements in addition to the stated elements.

As used herein, the term 'software' refers to a technology that moves hardware in a computer, and the term 'hardware' refers to a tangible device or device (CPU, memory, input device, output device, peripheral device, etc.) constituting the computer, and , the term 'step' means a series of processing or manipulations linked in time series to achieve a predetermined goal, and the term 'computer program', 'program', or 'algorithm' refers to a set of instructions that are summed up for processing by a computer. means, and the term 'program recording medium' refers to a computer-readable recording medium in which a program used for installing, executing, or distributing a program is recorded.

Terms such as '~ node', '~ unit', '~ module', '~ unit', '~ block', and '~ board' used in this specification to refer to the elements of the present invention are at least one function. It may mean a physical, functional, or logical unit that processes an operation, which may be implemented as one or more hardware, software, or firmware, or a combination of one or more hardware, software, and/or firmware.

As used herein, 'mobile device', 'computer', 'computing device', 'server device', or 'server' refers to an operating system such as Windows, Mac, or Linux, a computer processor, memory, applications, storage device (eg For example, HDD, SDD), and a device having a monitor may be implemented. The computer may be, for example, a device such as a desktop computer, a notebook computer, or a mobile terminal, but these are exemplary and not limited thereto. The mobile terminal may be one of a smart phone, a tablet PC, or a mobile wireless communication device such as a PDA.

In this specification, a reference to component 'A' transmitting information, details, and/or data to component 'B' means that component 'A' transmits directly to component 'B' or component 'A' is used in the meaning including transmitting to component 'B' through at least one or more other components.

Hereinafter, the present invention will be described in detail with reference to the drawings. In describing the specific embodiments below, various specific contents have been prepared to more specifically describe the invention and help understanding. However, a reader having enough knowledge in this field to understand the present invention may recognize that the present invention may be used without these various specific details. In addition, it is mentioned in advance that parts that are well-known or commonly used techniques in describing the invention, but not largely related to the invention, are not described in order to avoid confusion in describing the present invention.

1 schematically shows a network configuration including a web scraping system 100 according to an embodiment of the present invention. Referring to FIG. 1 , a web scraping system 100 according to an embodiment may communicate with one or more web scrapers 10 and one or more target sites 20 through a communication network 30 .

The web scraping system 100 generates and manages a command (script) defining a scraping job to be performed by each web scraper 10, and gives the web scraper 10 a job (a 'job' in a general sense) according to a schedule. In order to distinguish it from 'job' in this specification, it serves to convey the word 'job' or 'job').

The web scraper 10 is a device for scraping digital content from each target site 20 by accessing a web site to be scraped (hereinafter referred to as a “target site”) 20 . Each web scraper 10 may be a computer device independent of each other in a cloud environment. That is, each web scraper 10 is physically and/or logically independent of the web scraping system 100 and other web scrapers 10, including a processor (CPU), memory, storage (eg HDD or SSD), and It may be an independent computer device having an operating system (OS) and operating based thereon, and may be communicatively connected to the web scraping system 100 and the target site 20 through a wired and/or wireless communication network 30 . .

Each web scraper 10 executes a web scraping operation according to a job received from the web scraping system 100 , and transmits the digital content scraped from the target site 20 to the web scraping system 100 . have.

Meanwhile, in this specification, the term "web scraping system" may refer to only the web scraping system 100 of FIG. 1 in a narrow sense, but is also used in a broader sense to include the web scraper 10, or in a broader sense. It may also be used to mean including the raw web scraper 10 and the manager terminal (40 in FIG. 2 ).

The web scraping system 100 may store the scraping result received from the web scraper 10, ie, digital content, in a database, process or standardize the digital content as needed, and provide it to the user through a web server. For example, assuming that the target site 20 to be scraped is the site of shipping companies owning container ships, and the web scraper 10 scrapes the ship schedule at each shipping site, each web scraper 10 is Receives a job defining a scraping job for a specific shipping company site from the web scraping system 100, accesses each shipping company site accordingly, scrapes the shipping schedule of the shipping company, and transmits it to the web scraping system 100 . The web scraping system 100 may process or standardize the vessel schedules received from all web scrapers 10 to provide vessel schedules and receive reservations from one integrated vessel reservation site through a web server.

On the other hand, as the web scraper 10 and the target site 20 in Fig. 1, respectively, three web scrapers 11, 12, 13 and three target sites 21, 22, 23 are shown, but these numbers are exemplary, It will be understood that scraping for a plurality of target sites 20 is performed using two or more arbitrary plurality of web scrapers 10 in actual implementation of the present invention.

In addition, in FIG. 1, the communication network 30 may mean at least one of a wired and/or wireless communication network capable of communicating between the web scraping system 100, the web scraper 10, and the target site 20, Internet, Local Area Network (LAN), Metropolitan Area Network (MAN), Global System for Mobile Network (GSM), High Speed Downlink Packet Access (HSDPA), Wi-Fi, Voice over VoIP Internet Protocol), 3GPP Long Term Evolution (LTE), etc. may mean one or more wired and/or wireless communication networks.

2 is a diagram illustrating a web scraping system according to an embodiment.

Referring to FIG. 2 , the web scraping system 100 according to an embodiment may include a management node 110 , a database (DB) node 120 , and a service node 130 . However, as described above, when the web scraping system 100 is used in a broad sense, it will be understood that the web scraping system 100 may further include at least one of the work node 10 and the manager terminal 40 .

The work node 10 includes one or more web scrapers 11 , 12 , and 13 , and each of the web scrapers 11 , 12 , 13 will be referred to as a work node hereinafter. Each work node 10 (ie, each web scraper 11 , 12 , 13 ) has the same or similar configuration and function as the web scraper 10 described with reference to FIG. 1 . Each work node 10 may be a computer device independent of each other and communicates with the web scraping system 100 through the communication network 30 . Each job node 10 receives a job from the web scraping system 100, performs a scraping operation on each target site 20 accordingly, and then returns the scraping result (scrapped digital content) to the web scraping system ( 100) can be transmitted.

The management node 110 of the web scraping system 100 serves to manage and control the overall operation of the web scraping system 100 . In the illustrated embodiment, the management node 110 includes a job management module 111 , a node management module 112 , and a script management module 113 .

The job management module 111 is a functional unit for managing a scraping job. For example, the job management module 111 creates a schedule (“job schedule”) for the scraping job, how many job nodes 10 to execute the scraping, and when each job node 10 executes the scraping. Various management for scraping can be performed by setting whether or not to do so.

In an embodiment, the system administrator may create a job schedule through the job schedule generating program and input the job schedule to the job management module 111 . In this case, the 'job schedule' includes, for example, data that specifies a target site to be web scraped, specifies a specific object to be scraped, or specifies an execution schedule (eg, execution time, execution cycle, etc.) of web scraping. In addition, the job schedule generation program may be installed in the job management module 111 or may be installed outside the web scraping system 100 such as the manager terminal 40 . In one embodiment, the work schedule generating program may provide a template so that an administrator can conveniently create a work schedule, which will be described later with reference to FIGS. 8 and 9 .

The node management module 112 is a functional unit that manages the work node 10 . The node management module 112 may monitor the status of each work node 10 , that is, each web scraper 11 , 12 , and 13 , and deliver a job for scraping to each work node 10 . . For example, the node management module 112 may check the on/off state of the power of each work node 10 and monitor whether an abnormal operation or an error of each work node 10 occurs. In addition, the node management module 112 may monitor the usage status of resources of each work node 10, for example, memory usage, processor (CPU) usage, and storage device (HDD or SSD, etc.) usage, and based on the monitoring result, Based on it, a scraping job can be given by delivering a job to the currently available job node 10 .

The script management module 113 is a functional unit that generates and manages a work script to be delivered to each work node 10 to perform scraping. The job script contains an algorithm that defines the scraping operation. Since a method of defining and arranging data is different for each target site 20 , a corresponding work script may be different for each target site 20 .

In an embodiment of the present invention, a script generating module (script generating program) may be provided so that an administrator can easily create a work script for each target site 20 . The script generation module may provide a template so that an administrator can conveniently create a job script. For example, the script generation module may provide a template to a user through a user interface (UI) and generate a job script according to a user input input to the template. 2 exemplarily shows that such a script generation module 200 is installed in the manager terminal 40 . However, alternatively, the script generating module 200 may be installed in any functional part of the web scraping system 100 . For example, it will be understood that the script generation module 200 may be installed integrated into the modules 111 , 112 , and 113 in the management node 110 , or may be installed as a program independent of the modules 111 , 112 , and 1130 .

Database (DB) node 120 includes a database (DB) for storing various data for web scraping. For example, the DB of the DB node 120 may store a job schedule managed by the job management module 111 and a job script generated by the script management module 113 or the script generation module 200 . In addition, the DB of the DB node 120 may store the scraping result received from each work node 10 , that is, the digital content scraped from each target site.

The service node 130 is a functional unit that processes or standardizes digital content collected from the target site and stored in the DB of the DB node 120 and provides it to general users. In an embodiment, the service node 130 may include a web server that retrieves data from the DB node 120 and provides it to the user. For example, when the web scraping system 100 of the present invention is used to scrape a ship schedule from a shipping site, the service node 130 provides the vessel schedule collected from each shipping site through a web server, and a general user (ie, a user who wants to reserve a ship) may access a corresponding web server through the user terminal 50 and use services such as ship schedule inquiry and cargo transportation reservation.

The manager terminal 40 may be connected to communicate with the web scraping system 100 through a wired or wireless communication network, and provides a user interface so that the administrator of the web scraping system 100 can manage the web scraping system 100 It is a computer device. For example, the manager terminal 40 may be any computer device such as a desktop computer, a notebook computer, a tablet, or a mobile phone. In one embodiment, the script generating module 200 may be installed in the manager terminal 40, and the manager may write a script for web scraping by using it. In addition, according to an embodiment, a work schedule generating program may be installed in the manager terminal 40, and the manager may create a work schedule for web scraping by using it.

3 is a diagram for explaining the operational relationship between the management node 110 and the work node 10 of the web scraping system 100 for web scraping according to an embodiment.

Referring to FIG. 3 , first, in step S10 , the management node 110 generates and stores a job script. For example, an administrator may generate a job script using the script generation module 200 installed in the management node 110 or the administrator terminal 40 . At this time, the 'job script' may be data including a command for each job node 10 (ie, each web scraper 11 , 12 , 13 ) to scrape digital content of the target site 20 .

In one embodiment, the job script is composed of commands that cause each job node 10 to perform web scraping in a virtual browser manner. In this case, the 'virtual browser' method means that a person actually uses a web browser to access a specific target site and operates in a similar way to scraping data. For example, just as a person accesses the main page of the target site and logs in and accesses the web page to be scraped and performs scraping, each work node 10 uses a main page, a login page, a web page with content to be scraped, etc. After connecting step by step, a working script is written to scrape.

In addition, in an embodiment of the present invention, the script generating module 200 includes a user interface (UI) that provides a template so that an administrator can conveniently write a work script. In this regard, an exemplary script generating module 200 that provides a template will be described with reference to FIGS. 4 to 6 .

4 schematically illustrates an exemplary user interface (UI) of the script generation module 200 according to an embodiment. The user interface (UI) shown in Fig. 4 may be, for example, a screen displayed on the monitor of the manager terminal 40 when the script generation module 200 is executed in the manager terminal 40. Script generation in one embodiment The module 200 may provide a user interface (UI) partitioned into at least a work template area 210, an object display area 220, and a script display area 230. In the illustrated embodiment, on the left side of the screen, Although the work template area 210 , the object display area 220 , and the script display area 230 are arranged to the right, these are exemplary and the arrangement of each area 210 , 220 , and 230 may vary.

The work template area 210 is an area in which the items input to the script are provided in advance in a template format so that the user can conveniently write the work script, and the user can write data in each input window.

In the object display area 220, when the user inputs data into the input window in the work template area 210 to create a unit of 'object', the object 221 of one unit thus created is displayed in a single rectangular box. It is an area that is displayed as an icon so that the user can easily and intuitively recognize it. Here, the 'object' is a unit of instruction set that defines the operation of each step to be sequentially performed by the operation node 10 accessing the target site for the virtual browser-type scraping operation.

For example, in the virtual browser method, when it is assumed that the task node 10 needs to perform a scraping operation by going through the main page and the login page of the target site 20 to a web page containing content to be scraped, the first object is The operation node 10 defines the operation of accessing the main web page (homepage) of the target site, the second object defines the operation of the operation node 10 clicking the login button of the main web page, the third object defines an operation in which the work node 10 inputs an ID to the ID input window on the login page accessed by clicking the login button, and the fourth object is the work node 10 enters a password in the password input window on the login page. Defines an input operation, and the fifth object defines an operation in which the work node 10 presses the 'login' or 'confirm' button on the login page (in a state where both ID and password are input), and then, for example, 6 object may define an operation of scraping specific digital content from a member-only webpage accessed by clicking the 'login' or 'confirm' button.

In this way, for example, for one target site, from the step of accessing the main web page to the step of completing the scraping, an object corresponding to each step is sequentially created to perform web scraping for the target site. can be completed, and in this case, whenever each object is created in the work template area 210 , the corresponding object is sequentially displayed in the object display area 220 in the form of a single rectangular box icon. In the rectangular box of the object 221 displayed in the object display area 220, buttons 222 such as 'modify' and/or 'delete' may be displayed, for example, as shown, so that the user can edit or modify the object 221 . Deletion can be done conveniently. For example, when the user presses the 'edit' button 222, the template input content entered by the user for creating the object 221 is displayed in each template input window of the work template area 210, and the user enters the template can be modified.

The script display area 230 is an area showing the work script 231 written by the user so far. That is, the object display area 220 may display the work script written by the user in a simple box shape, and the script display area 230 may be an area that displays the work script 231 according to a specific programming language or string format. In the illustrated embodiment, the job script 231 was written and displayed as string data in JSON format. However, this is an example, and it goes without saying that the job script may be displayed in a script writing language other than JSON or in a string format according to specific embodiments.

Meanwhile, in the illustrated embodiment, the work template area 210 includes a template name input window 211 , a title input window 212 , a URL input window 213 , a key input window 214 , an action designation input window 215 , It may include input windows such as a time setting input window 216 , a return key input window 217 , and a script execution input window 218 .

The template name input window 211 is an input window for selecting a list or type of target sites. For example, when scraping a shipping company site with the web scraping system 100 of the present invention, a list of each shipping company site may be input in advance, and a specific shipping company may be selected after expanding the list by clicking the input window.

The title input window 212 is a window for the author to input the title of the object currently being created, and the user can input the object title with an arbitrary name.

The URL input window 213 is a window for inputting a web page address of the target site, and the key input window 214 is for inputting a key for designating a specific button such as 'login' and 'confirm' displayed on a web page, for example. it's a window

The operation designation input window 215 is an input window for designating which operation the work node 10 should perform in the current web page. In one embodiment, the operation designation input window 215 is displayed in a hide/unfold method, and when the user clicks the input window 215 with a mouse pointer, a list 215a of various operations as shown in FIG. 7 is displayed. can For example, in the action list 215a of FIG. 7 , 'click' is an action of clicking a specific button designated by the key input window 214 on a web page, and 'moveClick' is an action of clicking while moving a mouse point, for example. . 'sendkeys' is an operation to transmit the value entered in the 'Send_text' field. In this case, 'Send_text' is, for example, a window for inputting a character string to be entered into an ID or password input box or a search box on a web page. 'getText' is an action to get text, and this action is selected when performing the actual scraping action. And 'iframe' is the action to recognize the contents of the frame when an iframe tag is used to put the page in the page, 'window' and 'close' are the actions to bring or close the window (window) object itself, and 'alert' is the notification It may be an action that pops up a window.

The time setting input window 216 is an input window for setting a delay time when the work node 10 sequentially performs a specific task within the target site. In general, depending on a web server environment or network environment that provides a web page, it may take time to switch from one web page to the next by, for example, a click operation. For example, if an ID and password are entered on the login screen and clicked, the login screen is displayed after waiting for a predetermined time. If the operation node 10 is programmed to immediately execute the next step without waiting for a predetermined time when accessing the web page in the virtual browser method, the scraping operation may not be performed normally. Therefore, by intentionally setting the delay time through the time setting input window 216 , it is possible to ensure that the scraping operation can be performed normally by switching after the delay time when switching from one job to the next.

The return key input window 217 is a window for inputting a key for a result value to be returned. For example, it is possible to set which key of a web page corresponds to data received as a scraping result.

The script execution input window 218 is a window for inputting a command or algorithm for executing another operation or program in a web page according to the user's need when executing another operation or program. For example, while accessing a specific web page, there may be cases where it is necessary to perform a specific operation (eg, an operation such as addition or multiplication) using the value values displayed on the web page as variables and scrape the result of this operation. have. In this case, a script defining the operation may be created in the script execution input window 218 and included in the job script 231 .

As described above, the user writes predetermined content in each input window of the above-described work template area 210 and finally clicks the add button 219 to create one object 221 , and the object 221 displays the object. It will be understood that the area 220 is displayed as an icon of a square box, and the actual work script content of this object 221 is displayed in the script display area 230 in JSON format.

Now, with reference to Figs. 5 and 6, a specific example of writing a job script that defines an operation of logging in and accessing an arbitrary target site will be described.

5 is a user interface (UI) screen for creating an object of each step using the script generation module 200 according to an embodiment, and FIG. 6 shows an exemplary web browser screen 300 for displaying a target site.

First, a case in which a user actually accesses a target site through a web browser will be described with reference to FIG. 6 . As shown in Fig. 6(a), the user executes a web browser such as Microsoft Edge or Google Chrome to access the main page (eg, homepage) of the target site. A button 310 for login is displayed on the upper right of the main page, and when the user puts the mouse pointer on the button 310 and clicks it, the login page as shown in FIG. 6(b) can be accessed.

The user inputs an ID and password in the ID input window 321 and the password input window 322 of the login page, respectively, and clicks the 'login' button 323, thereby logging in the web as shown in FIG. 6(c), for example. A page screen may be displayed. Also, a 'logout' button 330 may be displayed on the web page, for example, in the upper right corner.

Now, a process of creating a job script so that the job node 10 executes these operations will be described with reference to FIG. 5 . Fig. 5 (a) is a screen for creating an object (hereinafter referred to as a “first object”) defining the operation of the work node 10 to access the main page (homepage) of the target site. The user inputs the title of the first object in the title input window 212 and inputs the home page address of the target site in the URL input window 213 . Thereafter, the 'add' button 219 is clicked to add the first object to the job script.

Fig. 5 (b) is a screen for creating an object (“second object”) defining the operation of the work node 10 clicking the login button (310 in Fig. 6 (a)) on the upper right of the main page. At this time, it will be understood that the icon of the previously created first object 221a is displayed on the object display area 220 and the actual script 231a of the first object 221a is displayed on the script display area 230 in JSON format. will be. Also, when the user clicks the edit button 222a of the first object 221a here, the template input content of the first object appears in the work template area 210 and the user can modify the first object.

To create the second object, the user inputs the title of the second object into the title input window 212 . Then, "xpath" is selected from the "BY" item of the key input window 214 and a key value indicating a login button is inputted into the "KEY" item. At this time, the key value of the login button can be found by searching, for example, in the html code for the web page in the web browser (eg, the html code displayed when the F12 button is pressed in a web browser such as Microsoft Edge or Google Chrome). In addition, "click", which means a mouse click, may be designated in the action designation input window 215 . Then, click the 'Add' button 219 to add the second object to the job script.

FIG. 5(c) is a screen for creating an object (“third object”) defining an operation in which the work node 10 inputs an ID to the ID input window ( 321 in FIG. 6( b )) of the login page. At this time, the icons of the previously generated first object 221a and the second object 221b are displayed on the object display area 220, and the scripts 231a and 231b of the first and second objects 221a and 221b are displayed as scripts. displayed on the display area 230 .

To create the third object, the user inputs the title of the third object in the title input window 212 , selects “xpath” in the key input window 214 and inputs a key value indicating an ID input window, and operates In the designation input window 215, "sendkeys" is selected and an ID is input in the "Send_text" field. Thereafter, the third object is added to the job script by clicking the 'add' button 219.

Fig. 5 (d) is a screen for creating an object (“fourth object”) defining the operation of the work node 10 inputting a password into the password input window (322 of Fig. 6 (b)) of the login page. At this time, the icons of the previously created first to third objects 221a, 221b, and 221c are displayed in the object display area 220, and the scripts 231a, 231b, and 231c of the first to third objects are displayed in the script display area ( 230) is shown.

To create the fourth object, the user inputs the title of the fourth object in the title input window 212 , selects “xpath” in the key input window 214 and inputs a key value indicating a password input window, and operates In the designation input window 215 , “sendkeys” is selected and a password is entered in the “Send_text” field. Thereafter, the fourth object is added to the job script by clicking the 'add' button 219.

FIG. 5(e) is a screen for creating an object (“fifth object”) defining an operation in which the work node 10 clicks a 'login' button (323 in FIG. 6(b)) of a login page. At this time, the icons of the previously generated first to fourth objects 221a, 221b, 221c, and 221d are displayed on the object display area 220, and the scripts 231a, 231b, 231c, and 231d of the first to fourth objects are displayed. It is displayed in the script display area 230 .

To create the fifth object, the user inputs the title of the fifth object in the title input window 212 , selects “xpath” in the key input window 214 and inputs a key value that means a login button, and specifies an operation In the input window 215, “click” is selected.

After that, if the fifth object is added to the job script by clicking the 'add' button 219, a user interface (UI) screen as shown in FIG. 5(f) may be displayed. That is, as shown in FIG. 5(f) , the icons of the first to fifth objects 221a, 221b, 221c, 221d, and 221e created so far are displayed on the object display area 220, and the icons of the first to fifth objects are displayed. The scripts 231a, 231b, 231c, 231d, and 231e are displayed in the script display area 230 .

In addition, although not shown in the drawing, an object defining an operation of scraping and retrieving specific data from the next operation of the fifth object, for example, a logged-in web page, an object defining an operation of logging out, ... etc. can be further created sequentially, and in this way, as necessary, each object defining the individual operation performed by the work node 10 is created and connected, thereby completing the entire work script for the scraping work on the target site. you will understand that you can

Meanwhile, in an embodiment of the present invention, the script generating module 200 may modify the script by clicking and dragging the icon of each object in the object display area 220 . For example, if the user clicks the icon of the fourth object in the object display area 220 and drags it under the icon of the fifth object, the arrangement order of the fourth object and the fifth object may be changed. In addition, at this time, the icon of each object in the object display area 220 and the script corresponding to each object in the script display area 230 (hereinafter also referred to as “object script”) are made to correspond to each other so that the script display area 230 also displays the object script. It is configured so that the position can be changed in units.

According to this configuration, when the order of two arbitrary objects in the object display area 220 is changed, the order of the corresponding object scripts is also changed and displayed in the script display area 230 in response thereto. Therefore, according to the present invention, even an ordinary person or a novice developer unfamiliar with script writing can easily and conveniently create a script and perform various edits, such as modifying or changing the order in units of objects, easily and conveniently.

Meanwhile, in the illustrated embodiment, the icons representing the object 221 are displayed in the form of a rectangular box and the icons are arranged vertically on the screen, but this is an example, and the shape or arrangement of the icons on the screen may vary depending on the specific embodiment of the invention. of course there is

Referring back to FIG. 3 , if the above-described work script is generated ( S10 ), then the management node 110 may create a work schedule ( S20 ). The job schedule may include information specifying a target site to be subjected to web scraping and an execution schedule of web scraping.

In an embodiment, the job schedule may be generated by the job management module 111 or generated by the manager terminal 40 . In addition, the job schedule may be generated by the script generating module 200 or may be generated by a separate job schedule generating program.

In order to generate the work schedule, the work schedule generating program (or the script generating module 200 ) may provide a management template as shown in FIG. 8 . 8 is an exemplary user interface (UI) screen of the work schedule generating program 400, and the management template includes a schedule name input window 401, a cycle unit input window 402, a cycle interval input window 403, and input data. An input window such as an input window 404 may be provided. The administrator may set how many time intervals the web scraping operation for the target site is to be executed through the cycle unit input window 402 and the cycle interval input window 403 .

The input data input window 404 is an input window for importing the contents of the pre-written work script as it is and changing only some variables so that it can be used when it is necessary to write a script defining a similar scraping operation that is repeated. For example, in the illustrated embodiment, the work script 231 prepared by the method described in FIG. 5 is inputted to the "base" item of the input window 404 as it is, and a changed variable is inputted to the "dates" item.

9 schematically shows the relationship between the job script 231 and the job schedule script 410 when data is input to the input data input window 404 as described above. In FIG. 9 , the job script 231 may be, for example, a script generated using the script generating module 200 in the manner described with reference to FIG. 5 . The job schedule script 410 is generated in JSON format by inputting contents into the administrative templates 401 to 404 on the user interface (UI) screen of the job schedule generating program 400 of FIG. 8, for example, and the job schedule script 410 ) may include dynamic data 410a and schedule data 410b. The dynamic data 410a is data input to the input data input window 404 of FIG. 8, for example, the work script 231 is inputted as it is in the "base" item, and the "dates" item is changed for each repeated web scraping operation. A variable has been entered. For example, if a job script 231 for scraping ship schedules on the Incheon-Shanghai route is created, this script 231 is used for scraping ship schedules of various routes such as Shanghai-Incheon, Incheon-Hong Kong, Hong Kong-Incheon, etc. It can be reused. At this time, most of the work scripts do not need to be changed, and only the routes such as Shanghai-Incheon, Incheon-Hong Kong, and Hong Kong-Incheon can be changed to generate scripts. Therefore, in the "base" item of the dynamic data 410a, for example, the pre-written work script 231 for the Incheon-Shanghai route is input as it is, and the variables for the remaining routes such as Shanghai-Incheon, Incheon-Hong Kong, Hong Kong-Incheon are "dates" "You can type in the item.

Meanwhile, the schedule data 410b may include, for example, schedule information input through the period unit input window 402 and the period interval input window 403 on the user interface (UI) screen of FIG. 8 .

Referring back to FIG. 3 , if a job schedule is created in step S20 as described above, then the job management module 111 generates a job to be delivered to each job node 10 ( S30 ), The generated job (job) is transmitted to each job node 10 to instruct the scraping operation (S50). Here, a job may mean a scraping job to be performed by one job node 10 for a specific target site.

In one embodiment, the job management module 111 selects which job node 10 to give the scraping job according to the current state of each job node 10, and selects a different job for the selected job node 10. to convey For example, the node management module 112 monitors the memory usage, the processor (CPU) usage, and the storage device usage of the working node 10 from each working node 10 (S40), and based on the monitoring result, a scraping operation You can select the work node 10 to perform.

Each job node 10 that has received a job from the management node 110 web scrapes the target site 20 assigned to it, respectively (S60), and scrapes the result, that is, digital content extracted from each target site. It is transmitted to the management node (110). The management node 110 may receive digital content from each work node 10 and store it in the DB node 120 (S80), process and standardize the content as necessary, and provide it to the user through a web server. .

As described above, those of ordinary skill in the art to which the present invention pertains can understand that various modifications and variations are possible from the description of this specification. Therefore, the scope of the present invention should not be limited to the described embodiments and should be defined by the claims described below as well as the claims and equivalents.

10: work node 20: target site
30: communication network 40: manager terminal
50: user terminal
100: web scraping system
200: script generation module

Claims (7)

As a web scraping method using a web scraping system,
generating a work script including instructions for causing one or more web scrapers connected through a communication network to scrape digital content of a target site;
generating a job schedule specifying a target site to be subjected to web scraping and an execution schedule of web scraping;
generating a job to be delivered to each web scraper according to the job schedule; and
and delivering the job to each web scraper based on the job status of each web scraper.
The web scraping system includes a script generation module 200 that provides a user interface (UI) including a work template area 210 , an object display area 220 , and a script display area 230 ,
The work template area 210 provides items to be input in units of objects of the work script in a predefined template format in order to create a work script for web scraping in a virtual browser method, and receives user input through each item of the template. is the area,
The object display area 220 is an area that displays icons representing one or more objects created by the user in the order of object creation,
The script display area 230 is an area for displaying the work scripts created so far,
By changing the order of the icon of the first object and the icon of the second object displayed in the object display area 220 , the order of the script of the first object and the script of the second object displayed in the script display area 230 is changed. A web scraping method using a web scraping system, characterized in that it is configured to be able to. The method of claim 1,
receiving digital content scraped by each web scraper from each web scraper; and
Storing the received digital content in a database; Web scraping method using a web scraping system, characterized in that it further comprises. The method of claim 1,
Each web scraper is an independent computer device that is communicatively connected to the web scraping system by wire or wirelessly and executes a web scraping operation according to a job received from the web scraping system. Web scraping method using the system. The method of claim 1,
Prior to the step of delivering the job to each web scraper, based on the memory usage, processor usage, and storage usage of each web scraper, further comprising the step of selecting a web scraper to which the job is to be delivered A web scraping method using a web scraping system, characterized in that. The method of claim 1,
The web scraping system provides an input template to the user through a user interface (UI) and includes a script generation module capable of generating the job script according to the user input received as the input template, the web Web scraping method using scraping system. The method of claim 1,
The web scraping system further comprises a manager terminal connected to communicate with the web scraping system by wire or wirelessly,
A web scraping method using a web scraping system, characterized in that the script generation module is installed in the administrator terminal. A computer-readable recording medium in which a computer program for executing the web scraping method according to any one of claims 1 to 6 is recorded.

Images