KR20200022289A - System and method for scraping based on web browser - Google Patents

Abstract

The present invention relates to a web browser-based scraping system and a method thereof. The web browser-based scraping system includes: a web browser installed in a client device, storing a user certificate for access to a target server, and receiving a scraping request of a user; a scraping engine receiving the scraping request from the web browser; a relay server generating a preamble satisfying requirements of the target server for scraping; a plurality of security gateways receiving the preamble and the scraping request from the relay server, accessing and scraping the target server with an IP different from the IP of the relay server, delivering scrapped data to the relay server, and having different IPs; and a scraping management server (SMS) managing the operation state and IPs of the security gateways. The security gateways transmit a use-ready state and own IP information to the scraping management server when actuated. The relay server requests the scraping management server for the IPs of the security gateways when receiving the scraping request from the web browser. The scraping management server selects one of the security gateways to deliver the IP of the selected security gateway to the relay server. The scraping engine delivers scraping request information including a service script and certificate information of the client to the relay server.

Description

Web browser-based scraping system and method {SYSTEM AND METHOD FOR SCRAPING BASED ON WEB BROWSER}

TECHNICAL FIELD The present invention relates to scraping, and more particularly, to a web browser-based scraping system and method.

Scraping is a technology or program designed to automatically connect to a networked Internet system to display data on a screen and then extract only the data you need. It collects information about a website or program and stores it in another program or database. It can be viewed or used at any time, and new data can be generated by comparing and analyzing stored data.

These scraping technologies are used to consolidate financial assets scattered in each financial institution, to manage transactions such as retrieval and transfer at once, and to integrate all mail at once when using multiple web mails. E-mail integrated inquiry can be found.

Conventionally, scraping was performed in two ways. First, PC scraping installed a separate scraping module on the PC and scraping through the scraping module received the results. In PC scraping, the certificate is stored on the PC. The second is mobile app scraping, which sends information from the mobile app and receives the results. Mobile app scraping has a certificate stored on the mobile device.

PC scraping requires that the scraping module is installed on the PC and the certificate is stored on the PC.In order to scrape using a mobile device (for example, a smartphone), a separate app for scraping is installed on the mobile and the certificate is stored on the mobile device. There was inconvenience of scraping.

Patent Registration No. 10-1815235 (2017.12.28)

The problem to be solved by the present invention was created to solve the above-described problems, can provide a standard web-based flexible service without operating system constraints to reflect various customer demands and changes, the app within the customer system To provide a web browser-based scraping system and method that can be easily applied without a separate installation, such as PC module, server.

Web browser-based scraping system according to the present invention for achieving the above technical problem, is installed in the client device, the user certificate for accessing the target server with the information to be scraped is stored, the web browser accepting the user's scraping request ; A scraping engine that receives a scraping request from the web browser; A relay server (WSGS) for generating information (full text) satisfying the requirements of the target server for scraping; And receiving a scraping request and a full message from the relay server, accessing and scraping the target server with an IP different from that of the relay server, and transferring the scraped data to the relay server, each of which has a different IP. Gateway (SG); And a scraping management server (SMS) that manages IPs and operating states of the plurality of security gateways, and when the security gateway is activated, transmits its IP information and ready-to-use status to the scraping management server, and relays When the server receives the scraping request from the web browser, the server requests the IP of the security gateway from the scraping management server, and the scraping management server selects one of the plurality of security gateways and delivers the IP of the selected security gateway to the relay server. And, the scraping engine is characterized in that for transmitting the scraping request information including the service script and the certificate information of the client to the relay server.

The web browser-based scraping system according to the present invention further includes a client management server (SSLS) that manages a service script and scraping license information of a client required for scraping, and the web browser receives the scraping engine upon receiving a scraping request from a client. Request a service script to the client management server (SSLS) through, and the client management server is characterized in that the service script to pass to the scraping engine if the normal user to verify the scraping license of the client. The relay server, the plurality of security gateways, the scraping management server and the client management server is characterized in that provided in the cloud web service platform. IP of the plurality of security gateways is characterized by being provided by the cloud web service platform. The plurality of security gateways provide an asynchronous connection, provide the same level of security as directly connected to the target server for end-to-end communication, and provide sessions for SSL (Secure Socket Layer) communication. It is characterized by maintaining the ciphertext state without decrypting in the middle.

According to an aspect of the present invention, there is provided a web browser-based scraping method comprising: when a web browser receives a scraping request for a target server from a client, transmitting a scraping request and a user certificate of the client to a scraping engine (120); Sending a service script for the scraping and the user certificate to a relay server by a scraping engine; The relay server constructs information (full text) that satisfies the requirements of the target server for scraping, and transmits the user certificate, service script, and the full text to a secure gateway using a personal information maintaining protocol (SSL); The security gateway accessing and scraping the target server using the full text and user certificate information and a service script through an IP different from the IP of the relay server; And receiving, by the web browser, the scraped information through the security gateway, the relay server, and the scraping engine, and each time the relay server accesses the target server, the IP is assigned a different security gateway. And accessing the target server through a gateway.

According to an aspect of the present invention, a web browser-based scraping method includes: requesting, by a scraping engine, a service script to a client management server when a scraping request of a client is received from the web browser; And the client management server verifying the scraping license of the client and providing a service script to the scraping server if the client is a legitimate client.

The web browser-based scraping method according to the present invention includes the steps of: when the security gateway is activated, transmitting its IP and ready to use message to the scraping management server; The scraping management server storing and managing an IP and a ready state of use of the security gateway; When the relay server receives a client's scraping request from the scraping engine, requesting a security gateway IP from the scraping management server; The scraping management server further includes the step of viewing the operating state of the security gateway and transmitting the IP of the security gateway ready for use to the relay server.

The secure gateway provides an asynchronous connection, provides the same level of security as directly connected to the target server for end-to-end communication, and in the middle of a session during SSL (Secure Socket Layer) communication. Maintain ciphertext without decrypting.

Web browser-based scraping system according to the present invention for achieving the above technical problem, is installed in the client device, the user certificate for accessing the target server with the information to be scraped is stored, the web browser accepting the user's scraping request ; A scraping engine that receives a scraping request from the web browser; A relay server (WSGS) for generating information (full text) satisfying the requirements of the target server for scraping; Receives a scraping request and a full message from the relay server, accesses and scrapes the target server with an IP different from that of the relay server, transfers the scraped data to the relay server, and each of the plurality of security gateways having different IPs. (SG); And a client company server managing IPs and operation states of the plurality of security gateways. The security gateway transmits its IP information and a ready-to-use state to the client company server when the security gateway is activated, and the relay server transmits the web. Upon receiving a scraping request from the browser, the client server requests the IP of the security gateway, and the client server selects one of the plurality of security gateways in a ready state, and delivers the IP of the selected security gateway to the relay server. The scraping engine delivers the scraping request information including the service script and the certificate information of the client to the relay server.

According to the web browser-based scraping system and method according to the present invention, it can be used without limitation in the service app of the customer company, and if the browser supports HTML 5 can be executed without limitation on the type of the terminal.

And according to the present invention, it is composed of one individual language, can be used only by applying a script without a separate app installation or update (update) and is convenient for maintenance.

In addition, it is easy to expand the service by freely changing additional target organizations or data types and easily applying it without additional work such as module installation.

1 is a block diagram showing an embodiment of the configuration of a web browser-based scraping system according to the present invention.
2 is a timing diagram illustrating an embodiment of a web browser-based scraping method according to the present invention.
3 is a block diagram illustrating another embodiment of the configuration of the web browser based scraping system according to the present invention.
4 and 5 are overall configuration diagrams of a first embodiment of a web browser based scraping system according to the present invention.
6 and 7 are overall configuration diagrams of a second embodiment of a web browser-based scraping system according to the present invention.
8 is a block diagram illustrating a configuration of an HTML 5 based client.
9 is a comparison between Secure Gateway (SG) and Proxy in terms of end-to-end (E2E) encryption.
10 illustrates a communication procedure between the relay server WSGS, the security gateway SG, and the scraping management server SMS in the scraping process.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Configurations shown in the embodiments and drawings described herein are only one preferred embodiment of the present invention, and do not represent all of the technical spirit of the present invention, various equivalents that may be substituted for them at the time of the present application It should be understood that there may be variations and variations.

1 is a block diagram showing an embodiment of the configuration of a web browser-based scraping system according to the present invention. One embodiment of the configuration of the web browser-based scraping system according to the present invention is a web browser 110, scraping engine 120, relay server (WSGS, 130), a plurality of security gateways (140, 145) and scraping management server (SMS, 150). The server may further include a client management server (SSLS) 160.

The web browser 110 is installed in a client device (not shown), and stores a user certificate for accessing the target servers 170 and 175 having the scraping target information, and receives a scraping request from the client. When the web browser 110 receives a scraping request from the client, the web browser 110 requests a service script from the client management server (SSLS) 160 through the scraping engine 120.

The scraping engine 120 receives a scraping request from the web browser 110.

The relay server WSGS 130 generates information (full text) satisfying the requirements of the target server 170 necessary for scraping. When the relay server 130 receives the scraping request from the web browser 110, the relay server 130 requests the IP of the security gateway (SG) from the scraping management server 150.

The plurality of security gateways SG, 140, and 145 receive the scraping request and the full text from the relay server 130, access the target server 170 with an IP different from that of the relay server 130, and then scrape and scrape. The data is transmitted to the relay server 130, and each has a different IP. When the security gateway 140 starts, it transmits its IP information and ready to use state to the scraping management server 150. IPs of the plurality of security gateways 140 and 145 may be provided by a cloud web service platform. In addition, the plurality of security gateways 140 and 145 provide an asynchronous connection, provide the same level of security as that directly connected to the target server 170 during end-to-end communication, and SSL (Secure Socket) Layer) During communication, a ciphertext state can be maintained without decrypting in the middle of a session.

The scraping management server (SMS) 150 manages IPs and operating states of the plurality of security gateways 140 and 145, and stores the IP information and the ready to use state from the security gateway 140. When the relay server 130 requests the IPs of the security gateways 140 and 145, the scraping management server 150 selects one of the plurality of security gateways and transfers the IP of the selected security gateway to the relay server 130. At this time, the scraping management server (SMS) 150 may reassign the IP of the security gateway according to the scraping request and the instance usage (number of calls per SG). The scraping engine 120 transmits the scraping request information including the service script and the certificate information of the client to the relay server 130.

The client management server (SSLS) 160 manages a service script required for scraping and scraping license information of the client. The client management server 160 verifies the scraping license of the client and delivers the service script to the scraping engine 120 when the client is a normal user.

The relay server 130, the plurality of security gateways 140 and 145, the scraping management server 150, and the client management server 160 may be provided by a cloud web service platform, for example, a cloud.

2 is a timing diagram illustrating an embodiment of a web browser-based scraping method according to the present invention. When the web browser 110 receives the scraping request for the target server 170 from the client 105 (step S200), the web browser 110 transmits the scraping request and the user certificate of the client 105 to the scraping engine 120 (step S205). When the scraping engine 120 receives the client's scraping request from the web browser 110, the scraping engine 120 requests a service script from the client management server 160 (step S210). The client management server 160 applies the scraping license of the client. If it is verified (step S215) and a legitimate client, the service script is provided to the scraping engine 120. (step S220).

The scraping engine 120 transmits the service script for the scraping and the user certificate to the relay server 130 (step S225).

On the other hand, the security gateway 140 transmits its own IP and the usage readiness completion message to the scraping management server 150 (step S212). The scraping management server 150 prepares the IP and use of the security gateway 140. Store and manage state

When the relay server 130 receives the client's scraping request from the scraping engine 120, the relay server 130 may request the security gateway IP from the scraping management server 150. (Step S230) The scraping management server 150 is secure gateway 140 IP address of the prepared security gateway may be transmitted to the relay server 130 (step S235). For example, the scraping management server (SMS) 150 may secure the service according to the scraping request and the instance usage (number of calls per SG). You can reassign the gateway's IP.

If the relay server 130 knows the IP of the security gateway 140 in advance, steps S230 and S235 may be omitted.

The relay server 130 configures information (full text) that satisfies the requirements of the target server 170 for scraping, and secures the user certificate, service script, and the full text using a personal information maintaining protocol (SSL). The security gateway 170 accesses and scrapes the target server 170 using the full text and user certificate information and a service script through an IP different from the IP of the relay server 130 (step S240). (S245) Here, the relay server 130 is assigned to the security gateway 140 having a different IP every time it connects to the target server 170 to the target server 170 through the assigned security gateway 140. Connect.

The web browser 110 receives the scraped information through the security gateway 140, the relay server 130, and the scraping engine 120 (steps S250, S255, and S260).

The secure gateway 140 provides an asynchronous connection, provides the same level of security as directly connected to the target server 170 for end-to-end communication, and sessions during SSL communication. (session) Maintain ciphertext without decrypting in the middle. The relay server 130, the plurality of security gateways 140 and 145, the scraping management server 150, and the client management server 160 may be provided by a cloud web service platform. The IP of the security gateway 140 may be provided in the cloud web service platform.

3 is a block diagram illustrating another embodiment of the configuration of the web browser based scraping system according to the present invention. Another embodiment of the configuration of the web browser-based scraping system according to the present invention is a web browser 310, scraping engine 320, relay server (WSGS, 330), a plurality of security gateway (340, 345) and customer company server ( 350). And it may further include a client management server (SSLS, 360).

The web browser 310 is installed in a client device (not shown), and stores a user certificate for accessing target servers 370 and 375 having scraping target information, and receives a scraping request from the client. When the web browser 310 receives a scraping request from the client (user), the web browser 310 requests a service script from the client management server (SSLS) 360 through the scraping engine 320.

The scraping engine 320 receives a scraping request from the web browser 110. The relay server WSGS 330 generates information (full text) satisfying the requirements of the target server 370 required for scraping. When the relay server 330 receives a scraping request from the web browser 310, the relay server 330 requests the IP of the security gateway from the client company 350.

The plurality of security gateways (SG, 340, 345) receives the scraping request and the full text from the relay server 330, accesses and scrapes by accessing the target server 370 with an IP different from that of the relay server 330. The data is transmitted to the relay server 330, and each has a different IP. When the security gateway 340 starts, it transmits its IP information and ready to use state to the customer company server 350. The IPs of the plurality of security gateways 340 and 345 may be provided by the cloud web service platform. In addition, the plurality of security gateways 340 and 345 provide an asynchronous connection, provide the same level of security as directly connected to the target server 170 in end-to-end communication, and SSL (Secure Socket) Layer) During communication, a ciphertext state can be maintained without decrypting in the middle of a session.

The customer company server (SMS) 350 manages IPs and operating states of the plurality of security gateways 340 and 345, and stores the IP information and the ready to use state from the security gateway 340. When the relay server 330 requests the IPs of the security gateways 340 and 345, the client company server 350 selects one of the plurality of security gateways and transfers the IP of the selected security gateway to the relay server 330. At this time, the client server 350 may reassign the IP of the security gateway according to the scraping request and the instance usage (number of calls per SG). The scraping engine 120 transmits the scraping request information including the service script and the certificate information of the client to the relay server 330.

The client management server (SSLS) 360 manages a service script required for scraping and scraping license information of the client. The client management server 360 verifies the scraping license of the client through the client company server 350 and delivers a service script to the scraping engine 320 when the client is a normal user.

The relay server 330, the plurality of security gateways 340 and 345, and the customer company server 350 may be installed in the customer IDC (Internet Data Center) 300.

On the other hand, the present invention by scraping (login, certificate) through a web browser installed on a mobile device, for example, a smartphone scrapes the server of the target organization and receives the result. Target institutions may be, for example, the IRS, the Health Insurance Corporation, cash receipts, real estate information, telecommunications companies, and SNS. The web browser used in the present invention may be an HTML5-based browser such as Chrome (Chrome, safari), and the language may be JavaScript at development time, and there is no need to install an app.

Hereinafter, the terms used in the present invention will be briefly described in Table 1.

In Table 1, WSGS is a communication relay station that initiates E2E (end-to-end) with the target organization, the target server.

4 and 5 is a first embodiment of the overall configuration diagram of a web browser-based scraping system according to the present invention, it is configured using a cloud (Cloud). 4 and 5, the finger is the name of the scraping service management company. The present invention protects personal information using SSL (Secure Socket Layer) which is a personal information maintenance protocol for protecting personal information. In addition, SG1 ~ SG n with different IPs are placed so that the IPs look different when viewed from the server of the target authority. SG (Secure Gateway) sends and receives HTTP Response Html File with target organization by HTTP Request Get / Post Method.

Referring to Figure 5, WSGS, SMS, SSLS, SG, SMDB is provided by the cloud (Cloud). When SG starts up, it calls SMS Update API that it is ready to use with its IP information. SMS updates SG information to SMDB. When the client requests communication, it checks with SG Search API through which SGS the WSGS should request communication. According to the client's communication request, WSGS configures the full text and communicates with the server of the target organization via the selected SG.

The IP allocation process of SMS is performed as follows. SMS manages the IP and status of each SG. SMS reallocates SG IP based on scraping request and instance usage (calls per SG)

The Client requests scraping from the Scraping Library on the service screen. Request a service script from the SSLS (service name, license key, and other information). SSLS verifies the license key via SMS. The SMS delivers the script to the client when the license verification result is a normal user. Scraping Labrary analyzes scripts to perform scraping services and receive scraping results. Configure scraping result and deliver it to service screen through web browser.

Client (Administrator) is a manager page that shows various information such as server information, customer information, success rate. Client (developer) develops script by scraping developer and uploads script to SSLS.

6 and 7 are second embodiments of the overall configuration diagram of the web browser-based scraping system according to the present invention, and are constructed using a cloud. 6 and 7 is the name of the scraping service management company. The present invention protects personal information using SSL (Secure Socket Layer) which is a personal information maintenance protocol for protecting personal information. In addition, SG1 ~ SG n with different IPs are placed so that the IPs look different when viewed from the server of the target authority. SG (Secure Gateway) sends and receives HTTP Response Html File with target organization by HTTP Request Get / Post Method.

Referring to FIG. 7, the SG in the Customer IDC calls the Update API of the customer server that it is ready for use with its IP information at startup. The client server updates the SG information with the client database. When the client requests communication, WSGS checks with the client server through the SG Search API which SG the communication request should be made. WSGS composes the full text according to the client's communication request and communicates with the target organization (server) via the selected SG.

The SG IP allocation process of the customer server is performed as follows. The customer server manages the IP and status of each SG. The customer server reassigns the SG IP based on scraping requests and instance usage (MAX 150 calls per SG). SSLS belonging to Finger server validates license and delivers script. SMS is responsible for log loading and status reports. Client requests scraping from Scraping Library on service screen. The Scraping Library requests the service script from the SSLS (service name, license key, and other information). The SSLS verifies the license key via SMS. If the license verification result in SMS is a normal user, SSLS sends the script to the client. Scraping Library (Scraping Engine) analyzes the script to perform the scraping service, configures the result of the scraping and delivers to the service screen of the web browser.

Client (administrator) manages the administrator page showing various information such as server information, customer information, success rate. In the client, the scraping developer develops the script and uploads it to SSLS.

8 is a block diagram illustrating a configuration of an HTML 5 based client. Referring to FIG. 8, a scraping request value is configured on a customer page and called through a standard API of the scraping library. The service type, customer license, etc. among the input values received through the standard API are transmitted to the client management server (SSLS). After verifying the license, the client management server receives the service script and executes it on the scraping engine. When executing service script, external library such as Crypto / PKI / Net / Common or Native function can be used as needed. The collected results can be sent directly to the browser according to the client's request or sent to the client's server through customizing I / F.

9 is a comparison between Secure Gateway (SG) and Proxy in terms of segment encryption (E2E, End-to-End). Secure Gateway provides an asynchronous connection so that the same security level is directly connected to the server when communicating between segments. To provide. In particular, SSL (TLS) communication provides full E2E connection without decrypting in the middle of the session. In other words, while the Proxy decrypts to a plain text state, the Secure Gateway maintains a ciphertext state, so that personal information can be more reliably protected.

Table 2 shows the advantages and disadvantages of comparing Proxy Server with Secure Gateway.

10 illustrates a communication procedure between the relay server WSGS, the security gateway SG, and the scraping management server SMS in the scraping process. Referring to FIG. 10, when the relay server WSGS 1010 requests the IP of the security gateway SG 1030 for communication relaying to the scraping management server SMS 1020, the scraping management server SMS 2020 relays the relay. Provide the server (WSGS, 1010) with the IP of the SG. The relay server 1010 transmits a scraping request to the SG 1030 using the assigned SG IP, and the SG 1030 receives the scraped data scraped from the target server (not shown). At this time, the SMS 1020 checks the usage amount of the security gateway 1030 and references it when assigning an IP next time.

The present invention can be embodied as code that can be read by a computer (including all devices having an information processing function) on a computer-readable recording medium. Computer-readable recording media include all kinds of recording devices that store data that can be read by a computer system. Examples of computer-readable recording devices include ROM, RAM, CD-ROM, magnetic tape, floppy disks, optical data storage devices, and the like. In addition, in this specification, “unit” may be a hardware component such as a processor or a circuit, and / or a software component executed by a hardware component such as a processor.

Although the present invention has been described with reference to the embodiments shown in the drawings, this is merely exemplary, and it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible. Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

100: cloud service 105: client
110: Web Browser 120: Scraping Engine
130: relay server (WSGS) 140: first security gateway (SG1)
145: second security gateway (SG2) 150: scraping management server (SMS)
160: client management server (SSLS) 170: target server 1
175: target server 2 310: web browser
320: scraping engine 330: relay server (WSGS)
340: first security gateway (SG1) 345: second security gateway (SG2)
350: client server 360: client management server (SSLS)
370: target server 1 375: target server 2
1010: relay server (WSGS) 1020: scraping management server (SMS)
1030: security gateway (SG)

Claims (10)

A web browser installed in the client device and having a user certificate for accessing the target server having the scraping target information, the web browser accepting the user's scraping request;
A scraping engine that receives a scraping request from the web browser;
A relay server (WSGS) for generating information (full text) satisfying the requirements of the target server for scraping; And
Receives a scraping request and a full message from the relay server, accesses and scrapes the target server with an IP different from that of the relay server, transfers the scraped data to the relay server, and each of the plurality of security gateways having different IPs. (SG);
And a scraping management server (SMS) that manages IPs and operating states of the plurality of security gateways,
The security gateway transmits its IP information and ready to use status to the scraping management server upon startup, and the relay server requests the scraping management server's IP from the scraping management server upon receiving a scraping request from the web browser. The scraping management server selects one of the plurality of security gateways to transfer the IP of the selected security gateway to the relay server, and the scraping engine sends the scraping request information including a service script and certificate information of the client to the relay server. Web browser based scraping system, characterized in that delivered to. The method of claim 1,
It further includes a client management server (SSLS) that manages scraping license information of clients and service scripts required for scraping.
When the web browser receives a scraping request from the client, the web browser requests a service script from the client management server (SSLS) through the scraping engine, and the client management server verifies the client's scraping license to the scraping engine if the user is a normal user. A web browser based scraping system, characterized in that it delivers a service script. The method of claim 2,
The relay server, the plurality of security gateways, the scraping management server and the client management server is provided in a cloud web services platform, the IP of the plurality of security gateways is provided by the cloud web services platform, web browser based Scraping system. The method of claim 1, wherein the plurality of security gateways
It provides asynchronous connection, provides the same security level as directly connected to the target server for end-to-end communication, and encrypts the ciphertext without decrypting it in the middle of the session during SSL (Secure Socket Layer) communication. A web browser based scraping system, characterized by maintaining state. When the web browser receives a scraping request from the client to the target server, transmitting the scraping request and the user certificate of the client to the scraping engine 120;
Sending a service script for the scraping and the user certificate to a relay server by a scraping engine;
The relay server constructs information (full text) that satisfies the requirements of the target server for scraping, and transmits the user certificate, service script, and the full text to a security gateway using a personal information maintaining protocol (SSL);
The security gateway accessing and scraping the target server using the full text and user certificate information and a service script through an IP different from the IP of the relay server; And
The web browser includes receiving the scraped information through the security gateway, the relay server and the scraping engine,
Each time the relay server accesses the target server, a security gateway having a different IP is assigned to access the target server through the assigned security gateway. The method of claim 5,
Requesting, by the scraping engine, a service script from a web browser to a client management server when the scraping request is received from the web browser; And
And the client management server verifies the scraping license of the client and provides a service script to the scraping server if the client is a legitimate client. The method of claim 5,
When the security gateway is activated, transmitting its IP and ready to use message to the scraping management server;
The scraping management server storing and managing an IP and a ready state of use of the security gateway;
When the relay server receives a client's scraping request from the scraping engine, requesting a security gateway IP from the scraping management server; And
The scraping management server further comprises the step of viewing the operating state of the security gateway and transmitting the IP of the security gateway ready for use to the relay server, Web browser based scraping method. The method of claim 5,
The secure gateway provides an asynchronous connection, provides the same level of security as directly connected to the target server for end-to-end communication, and in the middle of a session during SSL (Secure Socket Layer) communication. A web browser based scraping system, characterized in that it maintains a ciphertext state without decryption. The method according to any one of claims 5 to 8,
The relay server, a plurality of security gateways, scraping management server and client management server is provided by a cloud web service platform,
IP of the security gateway is characterized in that provided in the cloud web services platform, web browser based scraping method. A web browser installed in the client device and having a user certificate for accessing the target server having the scraping target information, the web browser accepting the user's scraping request;
A scraping engine that receives a scraping request from the web browser;
A relay server (WSGS) for generating information (full text) satisfying the requirements of the target server for scraping;
Receives a scraping request and a full message from the relay server, accesses and scrapes the target server with an IP different from that of the relay server, transfers the scraped data to the relay server, and each of the plurality of security gateways having different IPs. (SG); And
It includes a client company server for managing the IP and the operation state of the plurality of security gateway,
The security gateway transmits its IP information and ready-to-use state to the client company server upon activation, and the relay server requests the client server server for the security gateway's IP when receiving a scraping request from the web browser. The server selects one of the plurality of security gateways in a ready state to transfer the IP of the selected security gateway to the relay server, and the scraping engine relays the scraping request information including a service script and certificate information of the client. A web browser based scraping system, characterized in that it is delivered to the server.

Images