KR20240055224A - Information scraping method and system using hybrid scraping and computer program for the same - Google Patents

Abstract

The information scraping method includes the steps of a data scraping module requesting information scraping from the information providing server through a first communication session connected to the information providing server; determining, by the data scraping module, that transmission of web browser information or reception of information by the web browser is necessary for information scraping, based on a response received from the information provision server; transmitting, by the data scraping module, session information corresponding to the first communication session to a browser scraping module executable in a web browser; And the data scraping module may include receiving, from the browser scraping module, an information attribute value scraped by the web browser from the information provision server using the session information. According to the above method, data scraping based on an application programming interface (API) and scraping by a web browser are used in a hybrid method, thereby improving the performance of information collection and automation by using the browser only for a portion of the entire communication section. and stability can be secured at the same time.

Description

Information scraping method and system using hybrid scraping and computer program therefor {INFORMATION SCRAPING METHOD AND SYSTEM USING HYBRID SCRAPING AND COMPUTER PROGRAM FOR THE SAME}

Embodiments relate to an information scraping method and system, a computing device, and a computer program therefor. More specifically, embodiments use a web browser only for a portion of the entire communication section, so that scraping by a web browser and data scraping based on an application programming interface (API) are performed in a hybrid manner. It is about technology that automates information collection.

As information and communication technology develops, the need to collect data from various information sources is increasing. Scraping is a technology for easily obtaining information stored in external organizations while minimizing the user's manual work. Scraping technology includes screen scraping, which connects to the website of an information provider such as a financial institution and extracts only the necessary data from the data displayed on the screen through the website, and accesses the institution's server, etc. in accordance with the institution's policy and provides There is data scraping, which involves parsing and receiving the necessary information.

Screen scraping is performed using a web browser that can access the website of an information provider. This web browser-based scraping has the disadvantage of taking up a lot of system resources of the terminal and being slow. Meanwhile, data scraping utilizes data communication based on an application programming interface (API), so it has the advantage of being able to process information quickly with few resources. For example, Korean Patent No. 10-1815235 discloses a data scraping system and method for allocating scraping tasks according to a scheduling algorithm.

However, in order to acquire information through data scraping, there is a difficulty in that the developer of the scraping engine must directly implement the values generated by the web browser. Recently, the use of security systems such as Fraud Detection System (FDS) has increased, and in order to obtain information from financial institutions, it is often necessary to transmit valid security attribute values to the institution. API-based data scraping There is a problem in that it is difficult to generate these security attribute values. Additionally, in the case of organizations that use browser fingerprinting or dynamic web technology, there is a limitation in that it is impossible to obtain information without using a web browser.

Korean Patent No. 10-1815235

According to one aspect of the present specification, scraping by a web browser and data scraping based on an application programming interface (API) are hybridized by using a web browser only for a portion of the entire communication section. ) method and system for information scraping can be provided.

According to one aspect of the present specification, when web browser-based scraping is necessary in acquiring information through data scraping, the information attribute value is scraped by the web browser by running the web browser through a local device or remote server. It is possible to provide a user device that can acquire information and receive information from an information provision server based on this information.

According to one aspect of the present specification, in response to a request received from a computing device configured to collect information by a data scraping method, by executing a web browser, information attribute values are acquired from an information provision server by a scraping method by a web browser and computing them. A scraping service server that can be provided to the device can be provided.

Furthermore, according to one aspect of the present specification, a computer program may be provided in combination with hardware to execute the information scraping method by the above-described system, server, and device.

The problem to be solved by this specification is not limited to the above, and can be expanded to various matters that can be derived by the embodiments of the invention described below.

An information scraping method according to an embodiment of the present specification includes the steps of a data scraping module requesting information scraping from the information providing server through a first communication session connected to the information providing server; determining, by the data scraping module, that transmission of web browser information or reception of information by the web browser is necessary for information scraping, based on a response received from the information provision server; transmitting, by the data scraping module, session information corresponding to the first communication session to a browser scraping module executable in a web browser; and receiving, by the data scraping module, an information attribute value scraped by the web browser from the information provision server using the session information from the browser scraping module.

The information scraping method according to one embodiment further includes the step of the browser scraping module scraping the information attribute value from the information provision server by executing the web browser using the session information.

In one embodiment, the information attribute value includes one or more of cookie data, storage identification data, web user interface component data, and rendering result data.

In one embodiment, the step of scraping the information attribute value includes rendering, by the web browser, data received from the information provision server as the information attribute value.

In one embodiment, the step of scraping the information attribute value further includes the step of the web browser transmitting identification information of the web browser to the information provision server.

The information scraping method according to one embodiment includes the steps of the data scraping module creating a second communication session connected to the information providing server using the information attribute value; and the data scraping module receiving a response value to the information scraping request from the information providing server through the second communication session.

In one embodiment, the first communication session and the second communication session are created using an application programming interface (API) preset for the information providing server.

A computing device according to an embodiment of the present specification includes a memory storing program instructions; a communication module including a data scraping module; and a processor configured to control the memory and the communication module by executing the instructions.

The processor requests data scraping from the information provision server through a first communication session connected to the information provision server, and based on the response received from the information provision server, transmits web browser information or uses the web browser for information scraping. It is determined that information reception by a browser is necessary, and session information corresponding to the first communication session is transmitted to a browser scraping module executable in a web browser, and the web browser scrapes from the information provision server using the session information. and configured to control the data scraping module to receive an information attribute value from the browser scraping module.

In one embodiment, the processor creates a second communication session connected to the information provision server using the information attribute value, and sends a response value to the information scraping request from the information provision server through the second communication session. and control the data scraping module to receive data.

In one embodiment, the communication module further includes the browser scraping module, wherein the processor executes the web browser using the session information to scrape the information attribute value from the information provision server. It is further configured to control.

The information scraping system according to an embodiment of the present specification includes a data scraping module, requests data scraping to the information providing server through a first communication session connected to the information providing server, and receives a response from the information providing server. a user device configured to determine that transmission of information to or receipt of information by a web browser is necessary for information scraping; and a scraping service server including a browser scraping module capable of executing a web browser.

The user device is further configured to transmit session information corresponding to the first communication session to the scraping service server. In addition, the scraping service server is further configured to scrape information attribute values from the information provision server by executing the web browser using the session information and transmit the information attribute values to the user device.

In one embodiment, the user device creates a second communication session connected to the information provision server using the information attribute value, and receives a response value for an information scraping request from the information provision server through the second communication session. It is further configured to receive.

A computer program according to an embodiment of the present specification may be combined with hardware and stored in a computer-readable medium to execute the information scraping method according to the above-described embodiments.

According to the disclosure of this specification, scraping by a web browser and data scraping based on an application programming interface (API) are used in a hybrid manner by using a web browser only for a portion of the entire communication section. This has the advantage of automating information collection.

Through this hybrid scraping method, there is an advantage in easily obtaining information from information providing organizations using security attribute values that can only be processed by a web browser, browser fingerprinting, and the dynamic web. In addition, by using a web browser only for a portion of the entire communication section that requires information transmission and reception by a web browser, there is an advantage in reducing resource usage of the computing device performing scraping and ensuring performance and stability at the same time.

It should be understood that the effect of the present specification is not limited to the matters described above, and can be expanded to various contents that can be derived from the detailed description of the embodiments of the invention below.

FIG. 1 is a diagram illustrating an example of the operating environment of an information scraping system according to an embodiment of the present specification.
Figure 2 is a flowchart showing each step of the information scraping method according to an embodiment of the present specification.
Figure 3 is a block diagram for explaining the internal configuration of a computing device according to an embodiment of the present specification.
Figure 4 is a flowchart showing each step of the information scraping method according to another embodiment of the present specification.
Figure 5 is a flowchart showing each step of the information scraping method according to another embodiment of the present specification.

In describing the embodiments of the present specification, if it is determined that a detailed description of a known configuration or function may obscure the gist of the embodiments of the present specification, the detailed description thereof will be omitted. In addition, in the drawings, parts that are not related to the description of the embodiments of the present specification are omitted, and similar parts are given similar reference numerals.

In the embodiments of the present specification, when a component is said to be “connected,” “coupled,” or “connected” to another component, this refers not only to a direct connection relationship, but also to an indirect relationship where another component exists in between. Connection relationships may also be included. In addition, when a component is said to “include” or “have” another component, this does not mean excluding the other component, but may further include another component, unless specifically stated to the contrary. .

In the embodiments of this specification, terms such as first, second, etc. are used only for the purpose of distinguishing one component from other components, and do not limit the order or importance of components unless specifically mentioned. No. Therefore, within the scope of the embodiments of the present specification, the first component in an embodiment may be referred to as a second component in other embodiments, and similarly, the second component in the embodiment may be referred to as the first component in other embodiments. It may also be called.

In the embodiments of the present specification, distinct components are intended to clearly explain each feature, and do not necessarily mean that the components are separated. That is, a plurality of components may be integrated to form one hardware or software unit, or one component may be distributed to form a plurality of hardware or software units. Therefore, even if not specifically mentioned, such integrated or distributed embodiments are also included in the scope of the embodiments of the present specification.

In this specification, a network may be a concept that includes both wired and wireless networks. At this time, the network may refer to a communication network in which data exchange between devices, systems, and devices can be performed, and is not limited to a specific network.

Embodiments described herein may have aspects that are entirely hardware, partly hardware and partly software, or entirely software. In this specification, “unit,” “device,” or “system” refers to computer-related entities such as hardware, a combination of hardware and software, or software. For example, in this specification, a part, module, device, or system refers to a running process, processor, object, executable, thread of execution, program, and/or computer. It may be a (computer), but is not limited thereto. For example, both an application running on a computer and the computer may correspond to a part, module, device, or system in the present specification.

Additionally, in this specification, the computing device may be a mobile device such as a smartphone, tablet computer, wearable device, and HMD (Head Mounted Display), or a home appliance with a PC or display function. It may be a fixed device. Additionally, as an example, the computing device may be an in-vehicle cluster or an Internet of Things (IoT) device. That is, in this specification, computing devices may refer to devices capable of operating applications, and are not limited to a specific type. In the following, for convenience of explanation, the device on which the application runs is referred to as a computing device.

In this specification, the network communication method is not limited, and connections between each component may not be connected through the same network method. The network may include not only a communication method using a communication network (for example, a mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.), but also short-range wireless communication between devices. For example, a network can include objects and any communication method through which they can network, and is not limited to wired communication, wireless communication, 3G, 4G, 5G, or other methods. For example, wired and/or networks include Local Area Network (LAN), Metropolitan Area Network (MAN), Global System for Mobile Network (GSM), Enhanced Data GSM Environment (EDGE), High Speed Downlink Packet Access (HSDPA), W-CDMA (Wideband Code Division Multiple Access), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), Bluetooth, Zigbee, Wi-Fi, VoIP (Voice over Internet Protocol), LTE Advanced, IEEE802.16m, WirelessMAN-Advanced, HSPA+, 3GPP Long Term Evolution (LTE), Mobile WiMAX (IEEE 802.16e), UMB (formerly EV-DO Rev. C), Flash-OFDM, iBurst and It may refer to a communication network using one or more communication methods selected from the group consisting of MBWA (IEEE 802.20) systems, HIPERMAN, Beam-Division Multiple Access (BDMA), Wi-MAX (World Interoperability for Microwave Access), and ultrasonic communication. However, it is not limited to this.

Components described in various embodiments do not necessarily mean essential components, and some may be optional components. Accordingly, embodiments consisting of a subset of the elements described in the embodiments are also included in the scope of the embodiments of the present specification. In addition, embodiments that include other components in addition to the components described in the various embodiments are also included in the scope of the embodiments of the present specification.

Hereinafter, embodiments of the present specification will be examined in detail with reference to the drawings.

1 is a diagram illustrating an example of the operating environment of an information scraping system according to an embodiment of the present specification.

Referring to FIG. 1, an information scraping system according to an embodiment may include a computing device communicatively connected to the information providing server 3 through a wired and/or wireless network. At this time, the computing device may be one or more user devices (2), or may be a scraping service server (1). Figure 1 is an example for explaining the invention, and the number of devices or servers is not limited as in Figure 1.

One or more user devices 2 may be a stationary terminal implemented as a computer system or a mobile terminal. One or more user devices (2) include, for example, a smartphone, mobile phone, navigation, computer, laptop, digital broadcasting terminal, PDA (Personal Digital Assistants), PMP (Portable Multimedia Player), tablet PC, game console ( There are game consoles, wearable devices, IoT (internet of things) devices, VR (virtual reality) devices, and AR (augmented reality) devices. For example, in embodiments, the user device 2 is one of various physical computer systems capable of communicating with the scraping service server 1 and the information provision server 3 over a network, substantially using a wireless or wired communication method. It can mean one thing.

The scraping service server 1 may be implemented as a computing device or a plurality of computing devices that communicate with one or more user devices 2 through a wired and/or wireless network to provide commands, codes, files, content, services, etc. there is. For example, the scraping service server 1 may be a system that provides each service to one or more user devices 2 connected through a network. As a more specific example, the scraping service server (1) provides the service (i.e. information scraping) targeted by the application to one or more user devices (2) through an application as a computer program installed and running on one or more user devices (2). It can be provided as . As another example, the scraping service server 1 may distribute files for installing and running the above-described application to one or more user devices 2, receive user input information, and provide a corresponding service.

Figure 2 is a flowchart showing each step of the information scraping method according to an embodiment of the present specification. For convenience of explanation, information scraping methods according to embodiments will be described with reference to FIGS. 1 and 2.

First, the user of the user device 2 can request data from the information providing server 3 by scraping (S101). For example, the user device 2 may be a computing device, such as a smartphone, laptop computer, or PC, on which a scraping module is installed to scrape information from the information provision server 3 without user intervention. In addition, the information provision server 3 is a server of any institution that holds information that the user wishes to obtain, and may be a server of a financial institution such as a bank or credit card company, but is not limited thereto. For example, the information provision server 3 may be a web server that provides a website opened by a financial institution such as a bank to provide services to consumers.

The scraping module installed in the user device (2) receives information from the information provision server (3) according to the transmission and reception data and communication method previously stipulated in the information provision server (3), such as an application programming interface (API). It is configured to automatically collect data, and the hardware and/or software means for performing this operation are referred to herein as “data scraping modules.” For example, the data scraping module may be implemented in the form of a software application stored on or running on the user device 2.

However, in order to collect information from the information provision server 3, there are cases where reception or transmission of web browser information is necessary, and in this specification, the user device 2 uses a web browser in the corresponding section. It is configured to scrape information. First, the data scraping module running on the user device 2 may determine whether reception or transmission of web browser information is necessary when performing data communication with the information provision server 3 (S102).

In this specification, a case in which reception of web browser information is necessary refers to a case in which information must be confirmed through execution of a web browser because it is difficult to predict information to be rendered in the browser using only information in communication through the HTTP protocol. For example, when it is necessary to receive web browser information, when the information provision server 3 provides information through a dynamic website, and when screen capture of the user device 2 is required to collect information, information is provided. There are cases where the information provided by the server 3 is obfuscated by known obfuscation algorithms such as base 64 and XOR encoding.

In addition, in this specification, the case where transmission of web browser information is necessary means when the information provision server 3 requires identification information of the user's web browser prior to providing information or as part of the process of providing information. means. For example, in cases where transmission of web browser information is necessary, the information providing server 3 may request a security attribute value, such as a security token, that can only be created through a web browser. Financial institutions often operate a Fraud Detection System (FDS) to block abnormal access to information, and FDS is used to prepare for network attacks such as Distributed Denial of Service attacks (DDoS). For this purpose, security attribute values generated by browser fingerprinting can be requested.

If it is determined that reception or transmission of information by a web browser is necessary for information scraping, the data scraping module of the user device 2 may transmit session information for communication with the information provision server 3 to the web browser (S103 ). In one embodiment, the web browser, like the data scraping module described above, may be stored or implemented within the user device 2 and executed through the user device 2. However, in another embodiment, the user device 2 is configured to collect information only by data scraping, and the web browser may be configured to run through the scraping service server 1 rather than the user device 2. .

The browser scraping module, which is provided in the user device (2) or the scraping service server (1) and capable of executing a web browser, executes the web browser using the session information received from the data scraping module, so that the web browser acts as an information provision server. (3) Connection and user authentication are performed automatically (S104). Through this, the web browser running on the user device (2) or the scraping service server (1) automatically connects to the website provided by the information provision server (3) without any separate operation by the user and provides information requested by the user or the user. The information attribute value required to obtain the requested information can be received.

The above operation can be performed by executing a web driver, a web automation framework, on the user device (2) or the scraping service server (1). At this time, the web driver uses Java script, session storage, cookie data, and Document Object Model (DOM) to share session information with the data scraping module of the user device (2). ) It is configured to exchange data, etc. with the data scraping module.

The web driver functions to configure a virtualized environment corresponding to one or more operating systems and execute a web browser running on the operating systems. For example, a plurality of web browsers operating on the same or different operating systems can be executed on one computing device (user device 2 or scraping service server 1) through a web driver. Therefore, without the need to develop a separate scraping module for each operating system of the device on which scraping is performed, the web driver can be driven and by developing a platform-independent script, the advantage is that information can be scraped by implementing an arbitrary web browser environment. There is.

In one embodiment, the web driver is executed by the user device 2, and session information is passed from the data scraping module within the user device 2 to the virtualized web environment created by the web driver, thereby preventing scraping by the web browser. This comes true.

However, in another embodiment, the web driver is executed by the scraping service server 1, and the scraping service server 1 can scrape information using a web browser within a virtualized environment. Alternatively, screen data corresponding to the virtual web environment executed by the scraping service server 1 is transmitted to the user device 2, so that the user can use the user device 2 even if the web driver is not directly installed on the user device 2. It is also possible to monitor or remotely control the web environment within the scraping service server (1) through ).

In one embodiment, information collection using a web browser may be included as part of the process of collecting information through data scraping. For example, in the case of reCAPTCHA, which requires security attribute values corresponding to media content such as images and audio, when a web browser performs communication of media content and receives media content for one-time authentication (S105), the web browser obtains One media content can be passed to the data scraping module as an information attribute value. The data scraping module that acquires the media content uses it to extract and transmit the security token value, thereby creating a data scraping session connected to the information provision server (3) (S106) and collecting the necessary information from the information provision server (3). By receiving it, a response value for the user's information scraping request can be generated.

According to the embodiment described above, depending on the nature of communication between the data scraping module running on the user device 2 and the information providing organization 3, in some sections where the function of a web browser is required, such as generating a security token value. Scraping is performed by a web browser, and in the remaining sections, the function is implemented by a data scraping module, thereby ensuring the performance and stability of information scraping at the same time.

Meanwhile, in another embodiment, the data scraping module may determine that reception or transmission of information by the web browser is necessary in response to separately receiving a web browser-based task request from the user (S102). For example, if the data is not reflected in chronological order, the number of views is limited, or information is provided through a user-responsive interface, the information is provided in response to a user requesting to view such information. By running a web environment for inquiry in the browser, you can help users do additional work using the web browser. For example, additional tasks performed by users include, but are not limited to, comparing data scraping details with actual screen details, and processing security-sensitive information such as financial certificates.

Figure 3 is a block diagram for explaining the internal configuration of a computing device according to an embodiment of the present specification. The computing device 200 shown in FIG. 3 may be applied to any of the one or more user devices 2 or the scraping service server 1 described above with reference to FIG. 1, and each device and server may include some components. It can have the same or similar internal configuration by adding or excluding it.

Referring to FIGS. 1 and 3 , the computing device 200 may include a memory 210, a processor 220, a communication module 230, and an input/output unit 240. The memory 210 is a non-transitory computer-readable recording medium, and is a non-perishable large capacity device such as random access memory (RAM), read only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. It may include a permanent mass storage device. Here, non-perishable mass storage devices such as ROM, SSD, flash memory, disk drive, etc. may be included in the above-mentioned device or server as a separate persistent storage device that is distinct from the memory 210.

In addition, the memory 210 includes an operating system and at least one program code (for example, a scraping module installed and driven on the user device 2, etc., or code for an application installed on the user device 2, etc. to provide a specific service). It can be saved. These software components may be loaded from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, and memory cards.

In another embodiment, software components may be loaded into the memory 210 through the communication module 230 rather than a computer-readable recording medium. For example, at least one program is based on a computer program installed by files provided over a network by developers or a file distribution system that distributes the installation file of the application (e.g., scraping service server 1). It may be loaded into memory 210.

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to the processor 220 by the memory 210 or the communication module 230. For example, processor 220 may be configured to execute received instructions according to program code stored in a recording device such as memory 210.

The communication module 230 may provide a function for the user device 2 to communicate with the scraping service server 1 and/or the information provision server 3 through a network. In addition, when the scraping service server 1 is implemented in the form of the computing device 200 shown in FIG. 3, the communication module 230 has a function for the scraping service server 1 to communicate with the information providing server 3. can be provided. That is, the communication module 230 is a part that realizes the data scraping module and/or browser scraping module described above with reference to FIG. 2 by having its function controlled by the processor 220 that refers to the memory 210.

The input/output unit 240 may be a means for interfacing with an external input/output device (not shown). For example, external input devices may include devices such as a keyboard, mouse, microphone, and camera, and external output devices may include devices such as a display, speaker, and haptic feedback device. As another example, the input/output unit 240 may be a means for interfacing with a device that integrates input and output functions into one, such as a touch screen.

Additionally, in other embodiments, computing device 200 may include more components than those of FIG. 3 depending on the nature of the device to which it is applied. For example, when the computing device 200 is applied to the user device 2, it is implemented to include at least some of the above-described input/output devices, or a transceiver, a Global Positioning System (GPS) module, a camera, various sensors, It may further include other components such as a database, etc. As a more specific example, if the user device is a smartphone, the smartphone generally includes various devices such as acceleration sensors, gyro sensors, camera modules, various physical buttons, buttons using a touch panel, input/output ports, and vibrators for vibration. It can be implemented to include more components.

The information scraping method described below may be performed by the user device 2 and/or the scraping service server 1 implemented in the form of the computing device described above with reference to FIG. 3. As an example, the information scraping service may be provided to the user based on at least one of an application, software, and other programs that operate based on the user device 2 and/or the scraping service server 1. As an example, the user may request information scraping from the user device 2. The user device 2 operates the data scraping module and the browser scraping module mounted on the user device 2 in a hybrid method, or performs data scraping, but scrapes only the sections requiring scraping by a web browser. Hybrid data scraping can be performed by calling the service server (1). In this way, information obtained through hybrid data scraping can be provided to the user as a response value corresponding to the user's scraping request. For example, the response value may be displayed on a display of the user device 2 or transmitted from the user device 2 to another device.

Figure 4 is a flowchart showing each step of the information scraping method according to another embodiment of the present specification.

Referring to FIG. 4, in this embodiment, the user device 2 includes both a data scraping module 231 and a browser scraping module 232. The data scraping module 231 is configured to receive information from the information provision server 3 by communicating with the information provision server 3 in a pre-defined manner, such as an API. In this specification, the communication session between the data scraping module 231 and the information providing server 3 at this time is also referred to as a first communication session.

Meanwhile, the browser scraping module 232 is configured to receive information from the information provision server 3 by executing a web browser and accessing a website provided by the information provision server 3. The user device 2 may be implemented as a computing device 200 as described above with reference to FIG. 3, and the data scraping module 231 and the browser scraping module 232 may form part of the communication module 230. You can.

In this embodiment, the data scraping module 231 may request data scraping based on API or the like from the information providing server 3 through the first communication session (S201). Next, the data scraping module 231 receives the response of the information provision server 3 to the request (S202), and the response of the information provision server 3 is a request for identification information of the web browser or the web browser. If it is difficult to confirm the information without receiving it through , session information for data scraping can be shared with the browser scraping module 232.

At this time, the shared session information refers to information that can be read or loaded by the web browser so that the web browser can immediately move to the required work screen without separate user input. For example, the information provision server (3) It may be access information such as URL (Uniform Resource Locator) and/or cookie data, but is not limited to this.

The browser scraping module 232 executes a website using session information and connects to the information provision server 3, thereby receiving information provided by the information provision server 3 through the website (S204). The received information attribute value may be transmitted to the data scraping module 231 (S205). For example, the browser scraping module 232 may receive information from the information provision server 3 using a screen scraping method.

In one embodiment, the scraping process performed by the browser scraping module 232 executing a web browser may be performed while being displayed on the display of the user device 2 so that the user of the user device 2 can check it. However, this is an example, and in another embodiment, the browser scraping module 232 may scrape information by running a web browser in the background without exposing the execution state of the web browser to the user interface.

Information attribute values received through scraping by a web browser include cookie data required to obtain the information requested by the user from the information provision server (3), a specific key and/or value of local storage, and the web user interface. It may be the value of a component or media (image, sound, etc.) content, or data (e.g., screen data) generated as a result of rendering data provided by the information provision server 3 in a web browser.

The data scraping module 231 transmits the information attribute value received from the web browser to the information providing server 3 (S206), and can receive the information requested by the user from the information providing server 3 through data scraping. (S207). In this specification, the communication session between the data scraping module 231 and the information providing server 3 created using the information attribute value in this way is also referred to as a second communication session. At this time, the information attribute value transmitted to the information providing server 3 may be a security token value for FDS authentication, but is not limited thereto.

Figure 5 is a flowchart showing each step of the information scraping method according to another embodiment of the present specification. Unlike the embodiment described above with reference to FIG. 4, in the embodiment shown in FIG. 5, only the data scraping module 231 is mounted on the user device 2. Meanwhile, a scraping service server 1 equipped with a browser scraping module 100 capable of executing a web browser is configured to communicate with the user device 2.

Referring to FIG. 5, the user device 2 requests information from the information provision server 3 through the data scraping module 231 (S301), and the response received from the information provision server 3 (S302) is web If identification information of the browser is requested or the information is difficult to confirm without receiving it through a web browser, session information for data scraping can be transmitted to the scraping service server 1 (S303). The scraping service server 1 executes a web browser to acquire information attribute values from the information provision server 3 (S304), and transmits the acquired information attribute values to the user device 2 (S305).

In one embodiment, the user device 2 may expose a user interface for controlling the web browser of the scraping service server 1 within the user interface of the data scraping module 231. For example, the user can check the scraping status through a web browser through a control interface provided in the form of a separate window or pop-up within the software application for data scraping, or directly through the web browser. can be controlled.

Next, in one embodiment, the user device 2 transmits the information attribute value received from the scraping service server 1 to the information provision server 3 to obtain the information requested by the user from the information provision server 3 through a data scraping method. It can be received (S306, S307).

In the embodiment shown in FIG. 5, the reception of information by the data scraping module 231 and the reception of information by the browser scraping module 100 called by the data scraping module 231 have been described as being sequentially performed. However, this is an example, and in other embodiments, data scraping through an API, etc. and scraping through a web browser can be performed simultaneously.

For example, in this embodiment, the data scraping module 231 classifies the data requested by the user into first data that can be collected through data scraping such as API and second data that can be collected through a web browser, and for the first data The data scraping module 231 may receive information from the information providing server 3, but may request to receive information by calling the scraping service server 1 for the second data. At this time, the scraping service server 1 can collect the second data from the information providing server 3 and transmit it to the user device 2 by running a web browser independently of the operating state of the user device 2. The user device 2 may generate a response value to be provided to the user by combining the first data directly received by the user device 2 and the second data received from the scraping service server 1.

Additionally, in one embodiment, the scraping service server 1 may store information attribute values, such as cookie data, generated through information scraping by a web browser for a preset time. This is a method of transmitting previously created information attribute values back to the user device (2) without the need to re-run the web browser when information collection is requested from the same information provision server (3) from the user device (2) in the near future. This is to enable a quick response. For example, the scraping service server (1) stores cookie data generated through user authentication for one or more websites for each website, and the user device (2) requests information collection from a specific website. If possible, pre-stored cookie data can be transmitted as an information attribute value.

The embodiments described above can be implemented at least partially as a computer program and recorded on a computer-readable recording medium. Computer-readable recording media on which programs for implementing embodiments are recorded include all types of recording devices that store data that can be read by a computer. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, and optical data storage devices. Additionally, computer-readable recording media may be distributed across computer systems connected to a network, and computer-readable codes may be stored and executed in a distributed manner. Additionally, functional programs, codes, and code segments for implementing this embodiment can be easily understood by those skilled in the art to which this embodiment belongs.

Although the above-described specification has been described with reference to the embodiments shown in the drawings, these are merely illustrative examples, and those skilled in the art will understand that various modifications and modifications of the embodiments are possible therefrom. However, such modifications should be considered within the scope of technical protection of this specification. Therefore, the true technical protection scope of the present specification should be determined to include other implementations, other embodiments, and those equivalent to the claims according to the technical spirit of the appended claims.

Claims (15)

A data scraping module requesting information scraping from the information provision server through a first communication session connected to the information provision server;
determining, by the data scraping module, that transmission of web browser information or reception of information by the web browser is necessary for information scraping, based on a response received from the information provision server;
transmitting, by the data scraping module, session information corresponding to the first communication session to a browser scraping module executable in a web browser; and
An information scraping method comprising receiving, by the data scraping module, an information attribute value scraped by the web browser from the information providing server using the session information from the browser scraping module.
According to paragraph 1,
The information scraping method further includes the step of the browser scraping module scraping the information attribute value from the information provision server by executing the web browser using the session information.
According to paragraph 2,
The information attribute value includes one or more of cookie data, storage identification data, web user interface component data, and rendering result data.
According to paragraph 2,
The step of scraping the information attribute value includes rendering data received from the information provision server by the web browser as the information attribute value.
According to paragraph 2,
The scraping of the information attribute value further includes the step of the web browser transmitting identification information of the web browser to the information provision server.
According to paragraph 1,
generating, by the data scraping module, a second communication session connected to the information provision server using the information attribute value; and
The information scraping method further comprising the step of the data scraping module receiving a response value to the information scraping request from the information providing server through the second communication session.
According to clause 6,
The first communication session and the second communication session are created using an application programming interface preset for the information providing server.
A computer program stored on a computer-readable medium coupled with hardware to execute the information scraping method according to any one of claims 1 to 7.
Memory where program instructions are stored;
a communication module including a data scraping module; and
A processor configured to control the memory and the communication module by executing the command,
The processor,
Requesting data scraping from the information provision server through a first communication session connected to the information provision server,
Based on the response received from the information provision server, determining that transmission of web browser information or reception of information by the web browser is necessary for information scraping,
transmitting session information corresponding to the first communication session to a browser scraping module executable in a web browser;
A computing device configured to control the data scraping module so that the web browser receives information attribute values scraped from the information provision server using the session information from the browser scraping module.
According to clause 9,
The processor,
Creating a second communication session connected to the information provision server using the information attribute value,
The computing device is further configured to control the data scraping module to receive a response value for an information scraping request from the information providing server through the second communication session.
According to clause 9,
The communication module further includes the browser scraping module,
The computing device is further configured to control the browser scraping module to scrape the information attribute value from the information provision server by executing the web browser using the session information.
Comprising a data scraping module, requesting data scraping from the information providing server through a first communication session connected to the information providing server, and transmitting web browser information for information scraping based on a response received from the information providing server. or a user device configured to determine that receipt of information by the web browser is necessary; and
Includes a scraping service server that includes a browser scraping module capable of executing a web browser,
the user device is further configured to transmit session information corresponding to the first communication session to the scraping service server,
The scraping service server is further configured to scrape information attribute values from the information provision server by executing the web browser using the session information, and transmit the information attribute values to the user device.
According to clause 12,
The information attribute value includes one or more of cookie data, storage identification data, web user interface component data, and rendering result data.
According to clause 12,
The user device is further configured to create a second communication session connected to the information provision server using the information attribute value, and to receive a response value for an information scraping request from the information provision server through the second communication session. Information scraping system.
According to clause 14,
An information scraping system in which the first communication session and the second communication session are created using an application programming interface preset for the information providing server.

Images