KR20240094218A - System and method for page scraping service and computer program for the same - Google Patents

Abstract

A page scraping service system may include a scraping server configured to collect information by scraping from an information provider server. The scraping server includes a scraping engine configured to receive image information as part or all of a screen for displaying information provided by the information provision agency server on one or more devices connected to the information provision agency server; and an image processing unit configured to extract one or more data records from the image information. According to the page scraping service system, the type of information to be collected is determined by collecting each page as an image for one or more pages provided by an information provider and operating in a distributed search method to extract information from the collected images ex post. In many cases, inquiry errors due to process load can be prevented and processing time can be shortened.

Description

Page scraping service system and method and computer program therefor {SYSTEM AND METHOD FOR PAGE SCRAPING SERVICE AND COMPUTER PROGRAM FOR THE SAME}

Embodiments relate to a page scraping service system and method and a computer program therefor. More specifically, in the embodiments, when scraping information such as financial transaction details from the information provider server, images are collected for each page and data records are extracted from the images ex post to prevent and process inquiry errors due to process load. It's about technology that saves time.

Companies often build an Enterprise Resource Planning (ERP) system to enable account inquiries within the company's business solutions even without access to internet banking web pages, etc. In addition, recently, the Fraud Detection System (FDS), which detects suspicious transactions and maintains financial security by comprehensively analyzing various information collected from electronic financial transactions, has been attracting attention.

However, in the case of companies holding a large number of financial institution accounts, errors may occur in the inquiry due to process load in the process of checking accounts with a large number of transaction details online and linking them to the ERP system or FDS. Additionally, when opening a new account for each business of a company, difficulties arise in managing multiple accounts in the company's operating department.

Registered Patent Publication No. 10-1675416

According to one aspect of the present invention, when a user of a user organization such as a company scrapes information such as financial transaction details of his or her account from the server of an information provider, the transaction details are distributed through images for each page on which the information is displayed. A page scraping service system and method that prevents inquiry errors due to process load, shortens processing time, and enables the collection of large amounts of information, such as large amounts of deposit/withdrawal details for multiple accounts, and a computer program for the same. can be provided.

A page scraping service system according to one aspect of the present invention includes a scraping server configured to collect information by scraping from an information provider server. At this time, the scraping server includes a scraping engine configured to receive image information as part or all of a screen for displaying information provided by the information provision agency server on one or more devices connected to the information provision agency server; and an image processing unit configured to extract one or more data records from the image information.

In one embodiment, the image processing unit is further configured to extract the one or more data records from the image information while the communication connection between the scraping server and the information providing agency server is terminated.

In one embodiment, the scraping engine is further configured to receive the image information from one or more pages that configure a screen for displaying the information and are each configured to display a preset number of data records.

The page scraping service system according to one embodiment further includes a batch management server configured to transmit page index data of pre-collected information to the scraping server. At this time, the scraping server compares the number of the one or more pages constituting the screen for displaying the information with the page index data to determine a new page containing a data record not included in the pre-collected information. It further includes a configured index processing unit. Additionally, the scraping engine is further configured to receive the new page as the image information.

The page scraping service system according to one embodiment further includes a database server configured to store information collected from the information provision agency server by the scraping server and provide the stored information to the user agency.

In one embodiment, the scraping engine is further configured to transmit virtual input information defining a query range of information provided by the information providing agency server to the information providing agency server. At this time, the batch management server is further configured to generate the virtual input information to change the search range based on a change in the number of data records of information stored in the server in the database and to transmit the virtual input information to the scraping engine. do.

In one embodiment, the batch management server generates the virtual input information so that the inquiry period for information on the user organization decreases as the change in the number of data records corresponding to the user organization during a preset unit period increases. It is more structured.

A page scraping service method according to an aspect of the present invention includes the steps of a scraping server of a page scraping service system connecting to an information provider server through a scraping method; Receiving, by the scraping server, part or all of a screen for displaying information provided by the information provision agency server on one or more devices connected to the information provision agency server as image information; and extracting, by the scraping server, one or more data records from the image information.

The page scraping service method according to one embodiment further includes terminating a communication connection between the scraping server and the information provider server before extracting the one or more data records.

In one embodiment, the step of receiving the image information includes the scraping server receiving the image information from one or more pages each configured to configure a screen for displaying the information and displaying a preset number of data records. It includes steps to:

A page scraping service method according to an embodiment includes the steps of a batch management server of the page scraping service system transmitting page index data of pre-collected information to the scraping server; And the scraping server compares the number of the one or more pages constituting the screen for displaying the information with the page index data to determine a new page containing a data record not included in the pre-collected information. Includes more steps. At this time, the step of receiving the image information includes receiving the new page as the image information.

A page scraping service method according to an embodiment includes the steps of: storing information collected from the information provider server by the scraping server in a database server of the page scraping service system; And the page scraping service system further includes providing information stored in the database server to a user organization.

In the page scraping service method according to an embodiment, before the step of receiving the image information, the placement management server provides information provided by the information provider server in the database based on a change in the number of data records of the information stored in the server. generating virtual input information that defines a search range of information; The batch server transmitting the virtual input information to the scraping server; And it further includes the step of the scraping server transmitting the virtual input information to the information provider server.

In one embodiment, the step of generating the virtual input information includes the batch management server adjusting the inquiry period for information on the user organization as the change in the number of data records corresponding to the user organization during a preset unit period increases. and generating the virtual input information to reduce the virtual input information.

The computer program according to one embodiment is combined with hardware to execute the page scraping service method according to the above-described embodiments, and may be stored in a computer-readable medium.

A page scraping service system and method according to an aspect of the present invention is a distributed query method that collects images for each page for each information display page provided by an information provider and extracts information from the collected images. By operating as , there is an advantage in preventing inquiry errors due to process load and shortening processing time even when there are many types of information to be collected.

By using the page scraping service system and method according to one aspect of the present invention, it is easy to collect transaction details that meet specific inquiry conditions in a batch method among the transaction details of many accounts held by user organizations such as companies, so the collection The transaction details can be used for fund exchange through the Enterprise Resource Planning (ERP) system or the Fraud Detection System (FDS), and have the advantage of automated fund management.

For example, by realizing mass inquiry of real-time transaction details using the page scraping service system and method according to one aspect of the present invention, account management by financial institution and certificate is possible, thereby automating the management of newly opened and canceled accounts. , it is possible to prevent financial accidents by updating daily balance information of deposits or savings accounts, and has the advantage of being able to verify missing data through self-representation of transaction details.

In the past, each company had to implement its own fund management system, making development and maintenance difficult. However, according to the page scraping service system and method according to one aspect of the present invention, it is possible to transmit a large amount of real-time transaction details, so scraping engine and batch The system including the management server, DB server, etc. and the ERP system or FDS linked thereto can be configured on-premise and have the advantage of protecting corporate information at a high level of security.

1 is a schematic block diagram showing the configuration of a page scraping service system according to an embodiment.
Figure 2 is a schematic block diagram showing the hardware configuration applied to each server of the page scraping service system according to an embodiment.
Figure 3 is a flowchart showing each step of the page scraping service method according to an embodiment.
Figure 4 is a flowchart showing each step of a page scraping service method according to another embodiment.
5 and 6 are exemplary screen images showing a process of acquiring data by a page scraping service method according to an embodiment.
Figure 7 is a flowchart showing a process of dynamically determining a viewing period of information by a page scraping service method according to an embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1 is a schematic block diagram showing the configuration of a page scraping service system according to an embodiment.

Referring to FIG. 1, the page scraping service system 2 according to this embodiment is configured to communicate and operate with one or more user devices 1 and one or more information provider servers 4. The user device 1 refers to a computing device used by a user of any user organization that uses the page scraping service system 2 according to embodiments, such as a company, sole proprietor, or individual.

Additionally, the information provider server 4 refers to a server of any organization that provides information related to users. For example, the information provider server 4 may be a financial institution such as a bank, credit card company, securities company, or insurance company, or a public institution such as the National Tax Service, government office, government, or local government. In the following specification, the information providing organization server 4 is explained by taking as an example a bank server where an account of the user organization is opened. However, the types of organizations from which the page scraping service system 2 according to embodiments can receive information are not limited to this.

In addition, in one embodiment, the page scraping service system (2) is connected to one or more external servers (3) such as a Fraud Detection System (FDS) or an Enterprise Resource Planning (ERP) system associated with the user organization. It can also operate by further communicating with. For example, the page scraping service system 2 receives transaction details from multiple accounts of users opened across one or more information provider servers 4 by scraping, and detects abnormal transactions or detects abnormal transactions for the received transaction details. The scraped information can also be transmitted to an external server (3) for other types of use.

For the above operation, the page scraping service system 2 may be communicatively connected to the user device 1, one or more information provider servers 4, and an external server 3 through a wired and/or wireless network.

For example, a wired and/or wireless network that communicatively connects one or more of the page scraping service system (2), the user device (1), the information provider server (4), and the external server (3) is called a LAN (Local Area Network). Area Network), MAN (Metropolitan Area Network), GSM (Global System for Mobile Network), EDGE (Enhanced Data GSM Environment), HSDPA (High Speed Downlink Packet Access), W-CDMA (Wideband Code Division Multiple Access), CDMA ( Code Division Multiple Access, TDMA (Time Division Multiple Access), Bluetooth, Zigbee, Wi-Fi, VoIP (Voice over Internet Protocol), LTE Advanced, IEEE802.16m, WirelessMAN- Advanced, HSPA+, 3GPP Long Term Evolution (LTE), Mobile WiMAX (IEEE 802.16e), UMB (formerly EV-DO Rev. C), Flash-OFDM, iBurst and MBWA (IEEE 802.20) systems, HIPERMAN, Beam-Division Multiple It may refer to a communication network using one or more communication methods selected from the group consisting of Access (BDMA), Wi-MAX (World Interoperability for Microwave Access), and ultrasonic communication, but is not limited thereto.

The user device 1 is a device used by a user of a user organization who wishes to check information that can be collected through scraping, for example, account transaction details. For example, if the user organization is a company, the user device 1 may be a personal computer, a laptop computer, etc. used by the company's accounting manager or any executive or employee.

In this specification, scraping refers to a technology in which a computer system connects to a specific server on behalf of a user, performs user authentication, and collects necessary information. Scraping technologies include screen scraping, which collects information on the screen after the computer system accesses a web page, etc. to display the necessary screen, and an application programming interface (API) that is defined in advance by the server. There is a data scraping technology that receives information from a server through, and scraping in this specification may mean using any of these methods.

Additionally, the number and shape of user devices 1 shown in FIG. 1 are merely exemplary. For example, in FIG. 1, the user device 1 is shown in the form of a laptop computer, but in other embodiments, the user device 1 may be a mobile communication terminal such as a smartphone, a personal digital assistant (PDA), or a tablet. It may be implemented in the form of any computing device, such as a tablet or a set-top box for IPTV (Internet Protocol Television).

The page scraping service system 2 includes a scraping server 21 equipped with a scraping engine. In one embodiment, the page scraping service system 2 further includes a batch management server 22 for managing the scraping operation of the scraping server 21. Also, in one embodiment, the page scraping service system 2 includes a database (DB) server 23 in which scraped information and/or user authentication information is stored. Furthermore, in one embodiment, the page scraping service system 2 further includes a relay server 24 configured to transmit the scraped information to an external server 3, such as an FDS or ERP system.

The scraping server 21 collects information by scraping from one or more information providing agency servers 4, and converts part or all of the screen into an image for displaying information on a device connected to the information providing agency server 4. It is configured to collect and extract data records, such as transaction details, from the collected images after the fact. For the above operation, the scraping server 21 includes a scraping engine 211 for connecting to the information provider server 4 by scraping to collect the necessary screens as images, and a scraping engine 211 for processing the image information to extract data records. It may include an image processing unit 213.

In one embodiment, the batch management server 22 may transmit page index data that can identify information collected in advance to the scraping server 21. In this case, the scraping server 21, based on the page index data transmitted from the batch management server 22, receives only new pages that do not contain previously collected information as images, excluding pages for which data has already been collected. You can. Data records extracted from the received images may be transferred from the scraping server 21 to the batch management server 22.

For the above operation, in one embodiment, the scraping server 21 may further include an index processing unit 212. The index processing unit 212 compares each page of the screen provided from the information provision agency server 4 connected to the scraping engine 211 with the page index data of the batch management server 22, so that the specific page has previously been deleted. You can determine whether it is a page that collected data or a page that contains new data.

For example, the page index data provided by the batch management server 22 may include the number of pages previously collected as images, for example, the number of the final page. The index processing unit 212 can specify new pages that have not been previously scraped by comparing the number of pages on the screen provided by the information provision agency server 4 with page index data. When new data is specified, the scraping engine 211 can capture the new page as image information and extract data records from the captured image.

In one embodiment, the process of the scraping server 21 extracting individual data records from the screen image of the information provider server 4 is independent of the communication connection between the scraping server 21 and the information provider server 4. It can be done. For example, the scraping server 21 connects to the information provider server 4 by scraping, loads a screen on which information such as transaction details is displayed, collects one or more pages of the screen as an image, and converts the transaction from the collected images. The process of extracting details may be performed after the communication connection between the scraping server 21 and the information providing agency server 4 is terminated. While the communication connection is maintained, each page is collected as an image, and the process of extracting transaction details from the image is performed after the communication connection is terminated, preventing inquiry errors due to process load and shortening processing time. You can.

The operation of the scraping server 21 including the image processing unit 213 will be described in detail later with reference to FIGS. 3 to 6.

The batch management server 22 transmits page index data to the scraping server 21 to request information, and when the scraping server 21 collects images of new pages, the page index data can be updated based on them.

Additionally, the deployment management server 22 may receive authentication information for operation of the scraping server 21 from the user device 1 and transmit it to the scraping server 21. At this time, authentication information is information for verifying the identity of the user organization with respect to the information provision agency server 4 in order to receive information from the information provision agency server 4, such as joint certificate, financial certificate, ID and password. It may be, but is not limited to this.

The database server 23 serves to store information collected from the information provider server 4 by the scraping server 21. For example, the database server 23 includes an information storage unit 231 in which scraped information is stored, and the information stored in the information storage unit 231 can be stored in the information storage unit 231 by users of the user organization who access the page scraping service system 2. It can be provided to device (1).

Additionally, in one embodiment, the database server 23 further includes a ledger storage unit 232 in which ledger information of user organizations is stored. In this specification, ledger information may refer to one or more company account information (including account number and financial institution information) for which the user organization wishes to collect transaction details. Additionally, the ledger information may further include authentication information of the user organization corresponding to each account.

The page scraping service system 2 according to embodiments and each server or unit included therein may be entirely hardware, or may have aspects that are partly hardware and partly software. For example, each server or unit of the page scraping service system 2 may collectively refer to hardware and related software for processing or/or exchanging data of a specific format and content through electronic communication. In this specification, terms such as “unit,” “module,” “device,” “terminal,” “server,” or “system” are intended to refer to a combination of hardware and software driven by the hardware. For example, hardware may be a data processing device that includes a CPU or other processor. Additionally, software driven by hardware may refer to a running process, object, executable, thread of execution, program, etc.

Additionally, each element constituting the page scraping service system 2 is not necessarily intended to refer to a separate device that is physically separate from each other. That is, each server 21 to 24 of the page scraping service system 2 shown in FIG. 1 and each part constituting these servers use the hardware constituting the page scraping service system 2 for the operations performed by the hardware. They are only divided functionally, and each part does not necessarily have to be provided independently from each other. Of course, depending on the embodiment, one or more of the servers 21 to 24 described above may be implemented as separate devices that are physically separate from each other.

Figure 2 is a schematic block diagram showing the hardware configuration applied to each server of the page scraping service system according to an embodiment.

Referring to FIG. 2, one or more servers among each server of the page scraping service system according to embodiments are each implemented in the form of a computing device including hardware 200, a memory 210, and a processor 220. , may include a communication module 230 and an input/output unit 240.

The memory 210 is a non-transitory computer-readable recording medium, and is a non-perishable large capacity device such as random access memory (RAM), read only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. It may include a permanent mass storage device. Here, non-perishable mass storage devices such as ROM, SSD, flash memory, disk drive, etc. may be included in the above-mentioned device or server as a separate persistent storage device that is distinct from the memory 210.

Additionally, the memory 210 may store an operating system and at least one program code (for example, a code for a security module or an application installed to provide a specific service). These software components may be loaded from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, and memory cards.

In another embodiment, software components may be loaded into the memory 210 through the communication module 230 rather than a computer-readable recording medium. For example, at least one program is based on a computer program installed in memory ( 210).

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to the processor 220 by the memory 210 or the communication module 230. For example, processor 220 may be configured to execute received instructions according to program code stored in a recording device such as memory 210.

The communication module 230 may provide a function for the page scraping service system 2 to communicate with the user device 1, the information provider server 4, and/or the external server 3 through a network. Additionally, the communication module 230 may provide a function for the page scraping service system 2 to communicate with one or more other devices through a wired and/or wireless network. That is, the communication module 230 is a part that realizes each of the function modules described above with reference to FIG. 1 by having its functions controlled by the processor 220 that refers to the memory 210.

The input/output unit 240 may be a means for interfacing with an external input/output device (not shown). For example, external input devices may include devices such as a keyboard, mouse, microphone, and camera, and external output devices may include devices such as a display, speaker, and haptic feedback device. As another example, the input/output unit 240 may be a means for interfacing with a device that integrates input and output functions into one, such as a touch screen.

Additionally, in other embodiments, each server of the page scraping service system 2 may include more hardware components than those shown in FIG. 2 depending on the nature of the applied device. For example, when the page scraping service system 2 is applied to a terminal device used by a user, it is implemented to include at least some of the above-described input and output devices, or a transceiver, a Global Positioning System (GPS) module, a camera, It may further include other components such as various sensors, databases, etc. As a more specific example, when the terminal device is a smartphone, the smartphone generally includes an acceleration sensor, a gyro sensor, a camera module, various physical buttons, buttons using a touch panel, input/output ports, and a vibrator for vibration. It can be implemented to further include various components such as:

The page scraping service method described below may be performed at least partially involving information transmission and reception with a server implemented in the form of a computing device including the hardware 200 configuration described above with reference to FIG. 2. As an example, the page scraping service method may be provided to the user in the form of a service based on at least one of applications, software, and other programs that operate on the user device and/or server.

Figure 3 is a flowchart showing each step of the page scraping service method according to an embodiment. Hereinafter, for convenience of explanation, the page scraping service method according to this embodiment will be described with reference to FIGS. 1 and 3.

First, the scraping server 21 connects to the information provider server 4 by scraping using the authentication information of the user organization in accordance with a request received from one or more devices (e.g., user device 1) related to the user organization. (S11).

The scraping server 21 may receive data corresponding to a screen for the information provision agency server 4 to provide information to devices connected thereto, and may receive this screen in the form of an image (S13). At this time, the scraping server 21 transmits an input defining inquiry conditions, such as a inquiry period, to the information provider server 4, thereby creating one or more pages on which data records (e.g., transaction details) that meet the conditions are displayed. can be loaded (S12), and each loaded page can be collected in the form of one or multiple image files.

After each page of the information provision screen is collected in image form, the scraping server 21 can extract individual data records (e.g., transaction details) from the collected image information using a method such as character recognition (S15). . Since the above operation is performed by processing images already collected from the information providing agency server 4, there is no need for the scraping server 21 to remain connected to the information providing agency server 4 during extraction of data records. That is, the scraping server 21 may be configured to extract transaction details from the image after terminating the communication connection with the information provider server 4 (S14).

The process of the scraping server 21 according to embodiments collecting screen images and extracting transaction details from the images will be described in detail later with reference to FIGS. 5 and 6.

Figure 4 is a flowchart showing each step of a page scraping service method according to another embodiment.

Referring to FIG. 4, a user organization that wishes to use the page scraping service method according to this embodiment can register ledger information in the DB server 23 through the user device 1 (S21). This may mean that a user institution, such as a company, stores its account information opened across one or multiple financial institutions and authentication information for user authentication for the account in the DB server 23.

If there is information that the user of the user organization wants to inquire, such as account transaction details, the information request can be sent to the DB server 23 and/or the batch management server 22 (S22), and the batch management server 22 In response to this, a page scraping inquiry can be requested from the scraping server 21. At this time, the batch management server 22 may transmit page index data that allows identification of previously scraped pages to the scraping server 21 (S23).

In the embodiment of FIG. 3, the batch management server 22 is shown to perform scraping when a separate information request is made. However, in another embodiment, the batch management server 22 is provided with a predetermined scheduler to run a preset schedule. It is also possible to control the scraping server 21 to periodically perform scraping based on the scraping server 21.

The scraping server 21 can scrape information from the information provider server 4 according to the request of the batch management server 22. In order to collect information from the information provider server (4) by scraping on behalf of the user organization, the scraping server (21) uses the user organization's authentication information such as joint certificate, public certificate, and simple authentication to collect information from the information provider server (4). User authentication can be performed for 4).

In one embodiment, when there are multiple information providing agency servers 4 for receiving information, the page scraping service system 2 sequentially performs authentication for the user for each information providing agency server 4. Alternatively, batch authentication may be supported so that it can be processed at once for multiple information provision agency servers (4). For example, the batch management server 22 receives the user's authentication key (e.g., certificate and certificate password) from the user device 1, and sends this authentication key to each of the plurality of information provider servers 4. By processing each of the plurality of authentication methods, it is possible to simultaneously obtain authentication data for performing user authentication on several different information provider servers 4. Authentication data may be electronic signature data encrypted by combining the certificate and certificate password entered by the user.

The scraping server 21 receives authentication information of the user organization from the batch management server 22, and receives information such as financial transaction details from the information provision organization server 4 through scraping using this, and multiple information as described above. When authentication to the information provision agency server 4 is performed at once, connection to a plurality of information provision agency servers 4 can be performed simultaneously using a plurality of authentication information generated by the batch management server 22. It may be possible.

In the conventional scraping method, the scraping engine collects all information provided by the information provider at each inquiry by scraping, and if there is information collected and stored in advance, data duplicated with existing data is collected after scraping is performed. It is designed to identify and discard. However, receiving all data every time like this has the problem of increasing the process load and increasing the possibility of errors when there are multiple objects (eg, accounts) to collect data from.

On the other hand, in the embodiments of the present invention, the scraping server 21 is configured to load each page of the screen on which the information provider server 4 provides data (S24) and receive the loaded page in the form of an image (S24) S25). Since the process of extracting data records from the received images may be performed while the communication connection between the scraping server 21 and the information provider server 4 is terminated (S26), a large amount of transaction details for multiple accounts This can prevent increased process load or error potential.

Additionally, in one embodiment, the batch management server 22 transmits page index data for identifying previously collected information pages to the scraping server 21, thereby eliminating the need for the scraping server 21 to receive all information pages. Only pages containing new data can be searched distributedly. If there is no pre-built page index data, the batch management server 22 may construct page index data by querying the ledger and other information stored in the DB server 23.

5 and 6 are exemplary screen images showing a process of acquiring data by a page scraping service method according to an embodiment.

Referring to FIGS. 4 to 6, in the page scraping service system 2 according to an embodiment, the scraping server 21 performs an authentication process with the information provider server 4 for scraping using the authentication information of the user organization. It can be done. For example, if a company that is a user organization wishes to conduct a batch inquiry for a specific transaction period, the scraping server 21 automatically logs in to the financial institution's corporate banking web page using the user organization's certificate and , you can load the screen by selecting a menu to load a screen that displays transaction details such as pre-registration account inquiry, pre-registration account balance inquiry, etc.

Figure 5 shows an example of such a transaction details display screen, and the transaction details display screen may include an information area 430 where one or more data records (i.e., individual transaction details) that are the target of scraping are displayed. In addition, the transaction history display screen includes an information area 410 where account information such as account number is displayed, and/or transaction history search classification (deposit and/or withdrawal, etc.) 421, and a sorting method (e.g., most recent or past). It may further include a condition area 420 that defines search conditions such as date (day, etc.) (422) and inquiry period (423).

At this time, the scraping server 21 according to embodiments may receive at least the information area 430 on the screen where transaction details are displayed in the form of an image. Since the information area 430 includes one or more individual transaction details 431 to 440, the scraping server 21 captures the information area 430 as an image file and then performs image processing on the generated image file. Individual transaction details (431 to 440) can be extracted ex post through .

Meanwhile, depending on the search conditions defined through the condition area 420, if the number of transaction details that meet the search conditions is large, the transaction details may not be displayed on one page. In the example shown in FIG. 5, the transaction details displayed in the information area 430 are changed by moving the handle 511 down on the scroll bar 510 on the right side of the information area 430, and the scroll bar Each screen with a different position of the handle 511 on 510 is defined as a page in this specification.

In Figure 5, page movement using the scroll bar 510 is explained, but the information screen can be configured to switch between transaction history pages by clicking the button for page movement (e.g., forward/backward) or the button corresponding to the page number. The type of page that displays information and the method of moving between pages are not limited to the examples described in this specification.

FIG. 6 shows the information area 450 and the transaction details 451 to 460 included in the information area 450 being changed by lowering the handle 511 of the scroll bar 510 on the screen shown in FIG. 5. indicates. That is, the movement from the page shown in FIG. 5 to the page shown in FIG. 6 is achieved by moving the handle 511, and at this time, the scraping server 4 according to the embodiments creates at least the information area 450 of the new page. It is possible to receive images as images and extract individual transaction details (451 to 460) through image processing on the received images.

Depending on the operation method of user interface (UI) elements such as the scroll bar 510, each page may include one or more overlapping transaction details. In this case, the scraping server 21 extracts transaction details from the images of each collected page, compares the transaction details extracted from each image, discards duplicate transaction details, and provides only the newly collected transaction details to the user agency. or/or may be stored in the DB server 23.

In one embodiment, the batch management server 22 transmits the number of pages on which transaction details were previously collected (or the number of the final page) as page index data to the scraping server 21, and the scraping server 21 Using this, only new pages among the information screens of the information provider server 4 can be collected as images. For example, assume that the page index data delivered by the batch management server 22 is 5. At this time, the scraping server 21 moves each displayed page by moving the handle 511 in minimum units through the scroll bar 510, but since the first to fifth pages are pages for which data was previously collected, they are converted into images. Without collecting, you can collect images from the 6th or lower pages, which are new pages.

In another embodiment, even if the number of previously collected pages (or final page number) is a random natural number n, the final nth page may not have been completely filled with transaction details during previous data collection. In this case, even if the nth page has the same page number, if the page is not the final page, there is a possibility that new transaction details have been added to the page, so if the page index data received from the batch management server 22 is n, scraping is performed. The server 21 may be configured to recognize pages starting from the nth page as new pages and collect them as images.

In the embodiment shown in FIGS. 5 and 6, the transaction details of the information areas 430 and 450 are arranged in such a way that past transaction details are displayed at the top and recent transaction details are displayed toward the bottom, so the scroll bar 510 ), the lower the handle 511 is, the more a new page containing new transaction details is displayed. However, this is an example, and in other embodiments, information such as transaction details may be sorted in a different way, and the scraping server 21 creates a new page based on the sorting form of the data records provided by the information provider server 4. You can also judge the part where is displayed.

The scraping server 21 can collect images from pages determined to be new pages through the above-described process (S25) and extract data records such as transaction details from the collected images (S26). The information extracted by the scraping server 21 is transmitted to the batch management server 22 (S27), and the batch management server 22 can notify the user device 1 of the user organization of the scraped information. In one embodiment, the batch management server 22 stores the scraped information in the DB server 23 (S28) and uses it to allow the user device 1 to access the DB server 23 to check the information. Information can also be transmitted to organizations (S29).

In one embodiment, the scraping server 21 may extract the transaction details of each page and then use the total area 445, which is part of the transaction details display screen, to verify whether the extracted data is missing. At this time, the total area 445 refers to an area where the sum of all deposited amounts and/or all withdrawn amounts of the transaction details displayed on the relevant page are displayed. For example, the scraping server 21 collects the information areas 430 and 450 and the total area 445 of each page together or separately as images, and each transaction history extracted from the images of the information areas 430 and 450. By comparing the sum of the deposit and/or withdrawal amounts (431 to 440, 451 to 460) with the contents of the total area 445, it is possible to verify whether all transaction details have been extracted from the page without missing data.

Additionally, in one embodiment, the page scraping service system 2 may further transmit the scraped information to an external server 3, such as a company's FDS or ERP system, through the relay server 24 (FIG. 1). For example, the relay server may be configured to communicate with the external server 3 through a preset application programming interface (API). In the user organization's FDS or ERP system, transaction details searched by the page scraping service system (2) in a batch manner can be easily confirmed through API, enabling real-time transaction history bulk inquiry, so that all accounts held by the company can be checked. There is an advantage in being able to easily manage corporate funds by utilizing transaction history information.

In the page scraping service method according to embodiments of the present invention, each page of the screen on which information such as transaction details is displayed is first collected as an image, and then data is extracted from the image afterwards. At this time, for user organizations whose transaction details do not change significantly, it may be unnecessary to re-collect the entire page of transaction details each time. Therefore, in one embodiment, the search conditions for information to be scraped may be dynamically changed by determining how frequently information is needed for a specific user organization based on the update trend of information scraped for a specific user organization.

Figure 7 is a flowchart showing the process of dynamically determining the information viewing period by the page scraping service method according to this embodiment.

Referring to FIG. 7, the batch management server 22 stores information such as the transaction details of the user organization scraped by the above-described process in the DB server 23 (S31), and stores the DB server 23 in relation to a specific user organization. ) can be determined at what level the information stored in is updated (S32). At this time, the degree to which information is updated means a change in the number of data records stored in the DB server 23 during a preset unit period. For example, if the change in the number of data records during a unit period is more than a certain number, the degree of information update is large, and if the change in the number of data records during a unit period is less than that number, the degree of information update is small, and the standard for the degree of update can be determined. there is.

At this time, the batch management server 22 can dynamically change the search range for scraping information of the corresponding user organization according to the degree of update of the data record of each user organization (S33). For example, the batch management server 22 reduces the inquiry period and/or the number of inquiries and inquiry cycle of transaction details when the degree of update of the data record of the user organization is large, and when the degree of update of the data record is small, the transaction details are reduced. It may be configured to increase the inquiry period and/or the number of inquiries and inquiry cycle.

Since a large change in the number of transaction details means that new transaction details are frequently generated, the batch management server 22 sets the inquiry period for the corresponding user organization to a relatively short period (e.g., 1 day) and Information scraping by the scraping server 21 can be performed periodically. On the other hand, since the fact that data records are updated less frequently means that new transaction details are generated less frequently, the batch management server 22 sets the inquiry period for the corresponding user organization for a relatively long period (e.g., 3 days, 1 week, etc.). By setting it to , information scraping by the scraping server 21 can be performed at regular intervals.

The batch management server 22 specifies the dynamically determined inquiry range as the inquiry condition, generates a virtual input for retrieving transaction details from the information provider server 4 (S34), and scrapes this virtual input. It can be delivered to the server 21 (S35). For example, in this specification, virtual input refers to the search range defined through the area 423 on the transaction history display screen described above with reference to FIGS. 5 and 6, such as the most recent 10, the most recent 30, today, the previous day, 3 days, It refers to a data packet transmitted to the information provider server (4) by scraping to change the period to 7 days, 10 days, 15 days, 1 month, 3 months, 6 months, 1 year, etc.

The scraping server 21 loads the page corresponding to the corresponding inquiry range (e.g., inquiry period and/or number of inquiries) from the information provider server 4 according to the virtual input transmitted from the batch management server 22, By receiving the page as an image (S37), transaction details on the image can be extracted.

Operations by the page scraping service method according to the embodiments described above may be at least partially implemented as a computer program and recorded on a computer-readable recording medium. Computer-readable recording media on which programs for implementing operations by the page scraping service method according to embodiments are recorded include all types of recording devices that store data that can be read by a computer. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. Additionally, computer-readable recording media may be distributed across computer systems connected to a network, and computer-readable codes may be stored and executed in a distributed manner. Additionally, functional programs, codes, and code segments for implementing this embodiment can be easily understood by those skilled in the art to which this embodiment belongs.

The present invention discussed above has been described with reference to the embodiments shown in the drawings, but these are merely illustrative examples, and those skilled in the art will understand that various modifications and modifications of the embodiments are possible therefrom. However, such modifications should be considered within the technical protection scope of the present invention. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the attached claims.

Claims (15)

Including a scraping server configured to collect information by scraping from the information provider server,
The scraping server is,
a scraping engine configured to receive image information as part or all of a screen for displaying information provided by the information provision agency server on one or more devices connected to the information provision agency server; and
A page scraping service system comprising an image processing unit configured to extract one or more data records from the image information.
According to paragraph 1,
The image processing unit is further configured to extract the one or more data records from the image information in a state in which the communication connection between the scraping server and the information providing agency server is terminated.
According to paragraph 1,
The scraping engine is further configured to receive the image information from one or more pages that configure a screen for displaying the information and are each configured to display a preset number of data records.
According to paragraph 3,
It further includes a batch management server configured to transmit page index data of pre-collected information to the scraping server,
The scraping server is configured to compare the number of the one or more pages constituting the screen for displaying the information with the page index data to determine a new page containing a data record not included in the pre-collected information. It further includes an index processing unit,
The scraping engine is further configured to receive the new page as the image information.
According to clause 4,
A page scraping service system further comprising a database server configured to store information collected from the information provider server by the scraping server and provide the stored information to a user organization.
According to clause 5,
The scraping engine is further configured to transmit virtual input information defining a query range of information provided by the information providing agency server to the information providing agency server,
The batch management server is further configured to generate the virtual input information to change the search range based on a change in the number of data records of information stored in the server in the database, and to transmit the virtual input information to the scraping engine. Scraping service system.
According to clause 6,
The batch management server is further configured to generate the virtual input information so that the inquiry period for information on the user organization decreases as the change in the number of data records corresponding to the user organization during a preset unit period increases. system.
A step where the scraping server of the page scraping service system connects to the information provider server through scraping;
Receiving, by the scraping server, part or all of a screen for displaying information provided by the information provision agency server on one or more devices connected to the information provision agency server as image information; and
A page scraping service method comprising the step of extracting, by the scraping server, one or more data records from the image information.
According to clause 8,
Before extracting the one or more data records, the page scraping service method further includes terminating a communication connection between the scraping server and the information provider server.
According to clause 8,
The step of receiving the image information is,
A page scraping service method comprising the step of receiving, by the scraping server, the image information from one or more pages each configured to configure a screen for displaying the information and displaying a preset number of data records.
According to clause 10,
Transmitting, by the batch management server of the page scraping service system, page index data of pre-collected information to the scraping server; and
The scraping server comparing the number of the one or more pages constituting the screen for displaying the information with the page index data to determine a new page containing a data record not included in the pre-collected information. It further includes,
The step of receiving the image information includes receiving the new page as the image information.
According to clause 11,
storing information collected from the information provider server by the scraping server in a database server of the page scraping service system; and
A page scraping service method further comprising providing, by the page scraping service system, information stored in the database server to a user organization.
According to clause 12,
Before receiving the image information,
generating, by the batch management server, virtual input information that defines a search range of information provided by the information provider server, based on a change in the number of data records of information stored in the server in the database;
The batch server transmitting the virtual input information to the scraping server; and
A page scraping service method further comprising the step of the scraping server transmitting the virtual input information to the information provider server.
According to clause 13,
The step of generating the virtual input information is such that the placement management server reduces the inquiry period for information on the user organization as the change in the number of data records corresponding to the user organization during a preset unit period increases. A page scraping service method comprising generating input information.
A computer program stored in a computer-readable recording medium coupled with hardware to execute the page scraping service method according to any one of claims 8 to 14.

Images