KR102602178B1 - Web contents crawling management server which supports efficient distributed crawling for a pluraliy of web sites and the operating method thereof - Google Patents

Abstract

A web content crawling management server that supports efficient distributed crawling of multiple web sites and a method of operating the same are disclosed. The present invention collects all URLs of one or more web pages constituting each website from a plurality of web sites, assigns the URLs of the collected web pages to a plurality of data collection servers in a round-robin manner, and The present invention relates to a web crawling management server that transmits a work command instructing to collect web content data by accessing a web page of a URL assigned to each data collection server for a plurality of data collection servers, and a method of operating the same.

Description

Web content crawling management server that supports efficient distributed crawling of multiple websites and its operating method {WEB CONTENTS CRAWLING MANAGEMENT SERVER WHICH SUPPORTS EFFICIENT DISTRIBUTED CRAWLING FOR A PLURALIY OF WEB SITES AND THE OPERATING METHOD THEREOF}

The present invention relates to a web content crawling management server that supports efficient distributed crawling of multiple web sites and a method of operating the same.

Web crawling refers to a technology for extracting data from web pages, and is used to create copies of web pages or keep data on web pages up to date.

In relation to this, the existing data collection server for collecting web content data from web pages using web crawling technology was configured to perform web content data collection for each website.

For this reason, if you want to simultaneously perform web content data collection for multiple websites through existing data collection servers, the time required to collect web content data varies depending on the website, so first collect web content data. In many cases, it was difficult to efficiently collect web content data due to idle time occurring on the data collection server that had completed the collection task.

If a technology that appropriately allocates web content data collection tasks to each of the data collection servers is introduced when performing web content data collection tasks for multiple websites simultaneously, idle time that may occur in the data collection servers can be eliminated. Time can be minimized.

Therefore, in order to support data collection servers to more efficiently perform web content data collection tasks, research is needed on technologies that support efficient distributed crawling of multiple websites.

The present invention collects all URLs of one or more web pages constituting each website from a plurality of web sites, assigns the URLs of the web pages to a plurality of data collection servers in a round-robin manner, and then For data collection servers, by presenting a web crawling management server and an operation method for transmitting a work command instructing to collect web content data by accessing a web page of a URL assigned to each data collection server, We aim to support data collection servers to perform web content data collection tasks more efficiently.

A web content crawling management server that supports efficient distributed crawling of a plurality of web sites according to an embodiment of the present invention collects the URL (Uniform Resource) of one or more web pages constituting each web site from the plurality of web sites. A URL collection unit that generates a collection data set consisting of URLs of a plurality of web pages by collecting all locators, and a URL queue of each URL of the plurality of web pages included in the collection data set. ), a URL storage processing unit that stores the URLs of each of the plurality of web pages stored in the URL queue, sequentially extracting the URLs according to the storage order in the URL queue, and producing k (k is a natural number of 2 or more) pieces of data. A URL allocation unit that allocates to collection servers in a round robin manner, and for each of the k data collection servers, collects web content data by accessing a web page with a URL assigned to each data collection server. It includes a command transmission unit that transmits an instructing work command.

In addition, a method of operating a web content crawling management server that supports efficient distributed crawling of a plurality of web sites according to an embodiment of the present invention includes selecting one or more web pages constituting each web site from the plurality of web sites. generating a collection data set consisting of the URLs of a plurality of web pages by collecting all the URLs of the plurality of web pages, storing the URLs of each of the plurality of web pages included in the collection data set in a URL queue, the URL The URL of each of the plurality of web pages stored in the queue is sequentially extracted according to the storage order in the URL queue and assigned to k (k is a natural number of 2 or more) data collection servers in a round-robin manner. and transmitting, to each of the k data collection servers, a work command instructing to collect web content data by accessing a web page of a URL assigned to each data collection server.

The present invention collects all URLs of one or more web pages constituting each website from a plurality of web sites, assigns the URLs of the web pages to a plurality of data collection servers in a round-robin manner, and then For data collection servers, by presenting a web crawling management server and an operation method for transmitting a work command instructing to collect web content data by accessing a web page of a URL assigned to each data collection server, It can support data collection servers to perform web content data collection tasks more efficiently.

Figure 1 is a diagram illustrating the structure of a web content crawling management server that supports efficient distributed crawling of a plurality of web sites according to an embodiment of the present invention.
2 and 3 are diagrams for explaining the operation of a web content crawling management server that supports efficient distributed crawling of a plurality of web sites according to an embodiment of the present invention.
Figure 4 is a flow chart illustrating a method of operating a web content crawling management server that supports efficient distributed crawling of a plurality of web sites according to an embodiment of the present invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the attached drawings. This description is not intended to limit the present invention to specific embodiments, but should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention. In describing each drawing, similar reference numerals are used for similar components, and unless otherwise defined, all terms used in this specification, including technical or scientific terms, are within the scope of common knowledge in the technical field to which the present invention pertains. It has the same meaning as generally understood by those who have it.

In this document, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary. Additionally, in various embodiments of the present invention, each component, functional block, or means may be composed of one or more subcomponents, and the electrical, electronic, and mechanical functions performed by each component may be electronic. It may be implemented with various known elements or mechanical elements such as circuits, integrated circuits, and ASICs (Application Specific Integrated Circuits), and may be implemented separately or by integrating two or more into one.

Meanwhile, the blocks in the attached block diagram or the steps in the flow chart are computer program instructions that are mounted on the processor or memory of equipment capable of data processing, such as general-purpose computers, special-purpose computers, portable laptop computers, and network computers, and perform designated functions. It can be interpreted to mean. Because these computer program instructions can be stored in a memory provided in a computer device or in a computer-readable memory, the functions described in the blocks of a block diagram or the steps of a flow diagram can be produced as a manufactured product containing instruction means to perform them. It could be. In addition, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical function(s). Additionally, it should be noted that in some alternative embodiments, it is possible for functions mentioned in blocks or steps to be executed in a different order. For example, two blocks or steps shown in succession may be performed substantially simultaneously or in reverse order, and in some cases, some blocks or steps may be omitted.

Figure 1 is a diagram illustrating the structure of a web content crawling management server that supports efficient distributed crawling of a plurality of web sites according to an embodiment of the present invention.

Referring to FIG. 1, a web content crawling management server that supports efficient distributed crawling of a plurality of web sites according to an embodiment of the present invention includes a URL collection unit 111, a URL storage processing unit 112, and a URL allocation unit. It includes (113) and a command transmission unit (114).

The URL collection unit 111 collects all URLs (Uniform Resource Locators) of one or more web pages constituting each website from the plurality of web sites, thereby creating a collection data set consisting of the URLs of a plurality of web pages. ) is created.

For example, the plurality of websites are referred to as ‘Website 1, Website 2, and Website 3’, and at this time, ‘Website 1’ is composed of ‘Webpage 1, Webpage 2, and Webpage 3’. , 'Website 2' is said to be composed of 'Webpage A, Webpage B, Webpage C, Webpage D, Webpage E', and 'Website 3' is composed of 'Webpage A, Webpage B'. Let's assume it is configured.

Then, the URL collection unit 111 collects information from ‘Website 1, Website 2, and Website 3’ to ‘Webpage 1, Webpage 2, Webpage 3, Webpage A, Webpage B, Webpage C, Web By collecting the URLs of page D, web page E, web page A, and web page B, a collection data set consisting of the URLs of a plurality of web pages as shown in Table 1 below can be created.

Website web page URL of the web page website 1 web page 1 URL 1 web page 2 URL 2 web page 3 URL 3 website 2 Web page A URL A web page B URL B web page C URL C web page D URL D Web page E URL E website 3 web page URL is web page URL me

The URL storage processing unit 112 stores the URL of each of the plurality of web pages included in the collection data set in a URL queue.

At this time, according to one embodiment of the present invention, the URL storage processing unit 112 is a specific configuration for storing the URL of each of the plurality of web pages in the URL queue, and includes a URL group creation unit 115 and a group storage order number. It may include an allocation unit 116 and a storage processing unit 117.

The URL group generator 115 groups the URLs of each of the plurality of web pages according to the website from which each URL was collected, thereby creating a URL group corresponding to each of the plurality of web sites.

The group storage sequence number allocation unit 116 allocates a group storage sequence number with the earliest sequence to the URL group corresponding to each of the plurality of web sites in descending order of the number of URLs included in each URL group.

The storage processing unit 117 divides the URLs of each of the plurality of web pages into URL groups and sequentially stores them in the URL queue according to the group storage order assigned to each URL group, but is included in the same URL group. For existing URLs, the URLs of each of the plurality of web pages are stored in the URL queue by storing them in the URL queue in a random storage order.

Hereinafter, the operations of the URL group creation unit 115, the group storage order allocation unit 116, and the storage processing unit 117 will be described in detail using an example.

First, let us assume that, as in the above-described example, a collection data set as shown in Table 1 above is created by the URL collection unit 111.

Then, the URL group creation unit 115 groups the URLs of each of the plurality of web pages included in the collected data set as shown in Table 1 by the website from which each URL was collected, thereby grouping them into 'Web site 1, Web URL groups corresponding to 'Site 2' and 'Website 3' can be created as shown in Table 2 below.

URL group Website URL of the web page URL group 1 website 1 URL 1 URL 2 URL 3 URL group 2 website 2 URL A URL B URL C URL D URL E URL group 3 website 3 URL is URL me

Then, the group storage order allocation unit 116 assigns group storage order numbers to 'URL group 1, URL group 2, and URL group 3' in descending order of the number of URLs included in each URL group. can do.

In relation to this, the number of URLs included in each of 'URL group 1, URL group 2, and URL group 3' is '3, 5, and 2', so the group storage number allocation unit 116 is 'URL group 1'. , URL Group 2, and URL Group 3', the group storage order can be assigned as 'No. 2, No. 3, No. 1'.

Then, the storage processing unit 117 divides the URLs of each of the plurality of web pages as shown in Table 1 into URL groups and sequentially stores them in the URL queue according to the group storage order assigned to each URL group. However, for URLs included in the same URL group, the URLs of each of the plurality of web pages can be stored in the URL queue as shown in Table 3 below by storing them in the URL queue in a random storage order. .

Group save order number URL group URL of the web page One URL group 3 URL me URL is 2 URL group 1 URL 3 URL 1 URL 2 3 URL group 2 URL C URL E URL A URL D URL B

In this way, when the URL of each of the plurality of web pages is stored in the URL queue by the URL storage processing unit 112, the URL allocation unit 113 stores each of the plurality of web pages stored in the URL queue. URLs are sequentially extracted according to the storage order in the URL queue and allocated to k (k is a natural number of 2 or more) data collection servers 131, 132, and 133 in a round robin manner.

The command transmission unit 114 provides a work command instructing each of the k data collection servers 131, 132, and 133 to collect web content data by accessing the web page of the URL assigned to each data collection server. send.

Hereinafter, the operations of the URL allocation unit 113 and the command transmission unit 114 will be described in detail using an example.

First, let k be '3', and the '3' data collection servers (131, 132, 133) are 'data collection server 1 (131), data collection server 2 (132), and data collection server 3 (133). ', and let us assume that, as in the above-described example, the URL of each of the plurality of web pages is stored in the URL queue as shown in Table 3 above by the storage processing unit 117.

Then, the URL allocation unit 113 sequentially extracts the URLs of each of the plurality of web pages stored in the URL queue as shown in Table 3 according to the storage order in the URL queue, as shown in Figure 2. As shown, it can be assigned to 'data collection server 1 (131), data collection server 2 (132), and data collection server 3 (133)' in a round robin manner.

Then, the command transmission unit 114 sends a web page of the URL assigned to each data collection server for each of 'data collection server 1 (131), data collection server 2 (132), and data collection server 3 (133)'. You can connect to and send a work command instructing to collect web content data.

Relatedly, in the picture shown in FIG. 2, 'URL B, URL 1, URL E, URL B' is assigned to 'data collection server 1 (131)', and 'URL B' is assigned to 'data collection server 2 (132)'. A, URL 2, URL A' are assigned, and 'URL 3, URL C, URL D' is assigned to 'data collection server 3 (133)', so the command transmission unit 114 is assigned to 'data collection server 1 (131). )' to 'Web Page I, Web Page 1, Web Page E, Web Page B', and to 'Data Collection Server 2 (132)' to 'Web Page A, Web Page 2, Web Page B'. A work order for 'A' can be transmitted, and a work order for 'web page 3, web page C, web page D' can be sent to 'data collection server 3 (133)'.

That is, the web content crawling management server 110 collects all URLs of one or more web pages constituting each web site from the plurality of web sites, and collects k data for each URL of the plurality of web pages. After allocating to the servers 131, 132, and 133 in a round-robin manner, for the k data collection servers 131, 132, and 133, access the web page of the URL assigned to each data collection server to view web content. A task command can be sent that instructs data to be collected.

According to one embodiment of the present invention, the web content crawling management server 110 may further include a request receiving unit 118 and an information transmitting unit 119.

After the work command is transmitted to each of the k data collection servers 131, 132, and 133 by the command transmission unit 114, the request receiving unit 118 operates the k data collection servers 131, 132, 133. ), whenever a web content data collection task for any one of the plurality of web pages is started, a web page for web content data collection is generated from the k data collection servers 131, 132, 133. Receives a request to transmit information about the body area.

For example, as in the above-described example, the work command is transmitted to each of 'data collection server 1 (131), data collection server 2 (132), and data collection server 3 (133)' by the command transmission unit 114. And after that, when it is said that the web content data collection for 'web page E' has started in 'data collection server 1 (131)', the request receiver 118 receives 'data collection server 1 (131)'. A request for information about web page E' may be received.

In this way, in the first data collection server, which is one of the k data collection servers 131, 132, and 133, the web content data collection operation for the first web page, which is one of the plurality of web pages, is started. Accordingly, when a request to transmit information about the body area of the first web page is received from the first data collection server, the information transmission unit 119 checks the information about the body area of the first web page and then, Information about the body area of the first web page is transmitted to the first data collection server.

At this time, according to one embodiment of the present invention, the information transmission unit 119 checks the information about the body area of the first web page and then collects the first data by collecting information about the body area of the first web page. The specific configuration for transmission to the server includes an event generation unit 120, an area confirmation unit 121, a location confirmation unit 122, an additional area confirmation unit 123, an additional location confirmation unit 124, and a body area determination unit. It may include (125) and a transmission processing unit (126).

As the first data collection server starts collecting web content data for the first web page, the event generator 120 receives information about the body area of the first web page from the first data collection server. When a transmission request is received, a body area confirmation event is generated to check the body area of the first web page.

When the body area confirmation event is generated by the event generation unit 120, the area confirmation unit 121 renders the first web page through a web browser and then prints the markup that constitutes the rendered first web page. By parsing the markup language code, the sizes of a plurality of partial areas constituting the first web page are confirmed.

Hereinafter, the operations of the event generation unit 120 and the area confirmation unit 121 will be described in detail using an example.

First, as in the above-mentioned example, as the web content data collection task for 'web page E' begins in 'data collection server 1 (131)', 'data collection server 1 (131)' Let's assume that a request for information about 'web page E' is received from ')'.

Then, the event generator 120 may generate a body area confirmation event to check the body area of ‘web page E’.

In this way, when a body area confirmation event for 'web page E' is generated by the event generation unit 120, the area confirmation unit 121 renders 'web page E' through a web browser and then renders the 'web page E'. By parsing the markup language code constituting 'web page E', the sizes of a plurality of partial areas constituting 'web page E' can be confirmed.

In this regard, if 'web page E' is the same as the picture indicated by reference numeral 200 in FIG. 3, 'web page E (200)' is 'partial area 1 (201), partial area 2 (202), partial area. Since it is composed of ‘partial area 3 (203)’ and partial area 4 (204), the area confirmation unit 121 includes ‘partial area 1 (201), partial area 2 (202), partial area 3 (203), and partial area 4 ( You can check the size of '204)'.

In this way, when the size of the plurality of partial areas constituting the first web page is confirmed by the area confirmation unit 121, the position confirmation unit 122 determines the size of the plurality of partial areas constituting the first web page. After designating the partial area with the maximum size as the candidate body area, the first position, which is the placement position of the candidate body area on the first web page, is confirmed.

At this time, according to one embodiment of the present invention, the area confirmation unit 121 determines the location of a reference vertex, which is one of the vertices constituting the candidate body area, on the first web page, It can be confirmed by the first location, which is the placement location on the first web page.

For example, according to the above-described example, the sizes of ‘partial area 1 (201), partial area 2 (202), partial area 3 (203), and partial area 4 (204)’ are confirmed by the area confirmation unit 121. It is said that among 'partial area 1 (201), partial area 2 (202), partial area 3 (203), and partial area 4 (204)', the partial area with the largest size is called 'partial area 2 (202)'. Let's assume.

Then, the location confirmation unit 122 can designate ‘partial area 2 202’ as the candidate body area.

At this time, when the vertex 211 at the upper left point among the vertices 211, 212, 213, and 214 constituting 'partial area 2 (202)' is called the reference vertex, the location confirmation unit 122 determines the reference vertex ( The position 211 on the 'web page E (200)' can be confirmed as the first position, which is the placement position on the 'web page E (200)' of 'partial area 2 (202)'.

In this way, when the first location is confirmed by the location confirmation unit 122, the additional area confirmation unit 123 selects a plurality of second web pages constituting the first website, which is a website including the first web page. , randomly select n web pages (n is a natural number of 2 or more), render each of the n web pages through the web browser, and then markup each of the n rendered web pages. By parsing the language code, the size of a plurality of partial areas constituting each of the n web pages is confirmed.

For example, let n be ‘3’ and assume that the first position 211 is confirmed by the position confirmation unit 122, as in the above-described example.

At this time, according to the above-mentioned example, the website including 'webpage E (200)' is 'website 2', and 'website 2' is 'webpage A, webpage B, webpage C, webpage D, web page E', the additional area confirmation unit 123 selects 'three' web pages from 'web page A, web page B, web page C, web page D, web page E'. You can choose randomly.

In this regard, if the 'three' web pages randomly selected by the additional area confirmation unit 123 are 'web page A, web page B, and web page D', the additional area confirmation unit 123 After rendering each of 'Web Page A, Web Page B, and Web Page D' through a web browser, the markup language code of each rendered 'Web Page A, Web Page B, and Web Page D' is parsed, You can check the sizes of the plurality of partial areas that make up each of page A, web page B, and web page D'.

In this way, when the size of the plurality of partial areas constituting each of the n web pages is confirmed by the additional area confirmation unit 123, the additional location confirmation unit 124 for each of the n web pages, By designating the partial area with the largest size among the plurality of partial areas constituting each web page as the reference area, a reference area corresponding to each of the n web pages is designated, and then The arrangement position on each web page of the corresponding reference area is confirmed as the reference position corresponding to each of the n web pages.

At this time, according to one embodiment of the present invention, the additional location confirmation unit 124 determines each web page of the vertex of the point corresponding to the reference vertex among the vertices constituting the reference area corresponding to each of the n web pages. The position on the screen can be confirmed as a reference position corresponding to each of the n web pages.

For example, according to the above-described example, assume that the sizes of a plurality of partial regions constituting each of ‘web page A, web page B, and web page D’ have been confirmed by the additional region confirmation unit 123.

Then, the additional location confirmation unit 124 uses the partial area with the largest size among the plurality of partial areas constituting each web page for each of 'web page A, web page B, and web page D' as the reference area. By designating, the reference areas corresponding to each of 'web page A, web page B, and web page D' can be designated.

In relation to this, when the partial area with the largest size among the plurality of partial areas constituting 'web page A' is called 'partial area 5', the additional location confirmation unit 124 selects 'partial area 5' as 'web page A'. It can be designated as a reference area corresponding to page A'.

In this way, the additional location confirmation unit 124 can designate reference areas corresponding to each of 'web page A, web page B, and web page D', such as 'partial area 5, partial area 6, partial area 7'. .

At this time, according to the above-described example, the reference vertex 211 is the vertex of the upper left point among the vertices 211, 212, 213, and 214 constituting 'partial area 2 202', so an additional location confirmation unit ( 124) shows the location of the vertex of the upper left point among the vertices constituting 'partial area 5, partial area 6, and partial area 7' on each web page, 'webpage A, webpage B, and webpage D' respectively. It can be confirmed by the reference position corresponding to .

In relation to this, the reference area corresponding to 'web page A' is 'partial area 5', so the additional location confirmation unit 124 detects 'web page A' of the vertex at the upper left point among the vertices constituting 'partial area 5'. 'The position on the screen can be confirmed as the reference position corresponding to 'Web Page A'.

In this way, the additional location confirmation unit 124 can confirm the reference positions corresponding to each of ‘web page A, web page B, and web page D’.

In this way, when the reference position corresponding to each of the n web pages is confirmed by the additional location confirmation unit 124, the body area determination unit 125 determines the reference position corresponding to each of the n web pages. By checking whether it matches the first location, if it is confirmed that the reference location corresponding to each of the third web pages of the preset reference number or more among the n web pages matches the first location, the candidate body area is determined to be the body area of the first web page.

At this time, according to one embodiment of the present invention, the body area determination unit 125 compares the reference position corresponding to each of the n web pages with the first location to correspond to each of the n web pages. If the separation distance between the reference location and the first location is confirmed to be within a preset reference distance, it can be confirmed that the reference location corresponding to each of the n web pages matches the first location.

For example, as in the above-mentioned example, it is assumed that the first location 211 is confirmed by the position confirmation unit 122, and the additional position confirmation unit 124 determines 'web page A, web page B, web page D ' Let's assume that the reference positions corresponding to each have been confirmed.

Then, the body area determination unit 125 compares the reference position and the first position 211 corresponding to each of 'web page A, web page B, and web page D' to 'web page A, web page B, and web page D'. It is possible to check whether the reference position corresponding to each page D' matches the first position 211.

In relation to this, the reference position corresponding to 'web page A' and the first position 211 are compared to each other, and the separation distance between the reference position corresponding to 'web page A' and the first position 211 is a preset reference distance. If it is confirmed that it is within the range, the body area determination unit 125 can confirm that the reference position corresponding to 'web page A' matches the first position 211.

In this way, the body area determination unit 125 can check whether the reference position corresponding to each of ‘web page A, web page B, and web page D’ matches the first position 211.

At this time, the preset reference number is '2', and the reference position corresponding to each of 'web page A, web page B' among 'web page A, web page B, and web page D' is the first position (211). If it is confirmed that it matches, the body area determination unit 125 may determine that 'partial area 2 (202)' is the body area of 'web page E (200)'.

In this way, if the candidate body area is determined by the body area determination unit 125 to be the body area of the first web page, the transmission processing unit 126 sends information about the candidate body area to the body area of the first web page. It is designated as information about the body area and transmitted to the first data collection server.

For example, as in the above-described example, if the body area determination unit 125 determines that 'partial area 2 (202)' is the body area of 'web page E (200)', the transmission processing unit 126 The information on 'partial area 2 (202)' can be designated as information on the body area of 'web page E (200)' and transmitted to 'data collection server 1 (131)'.

Then, when 'data collection server 1 (131)' collects web content data from 'web page E (200)', 'web page E (200)' based on information about the body area of 'web page E (200)' Web content data can be collected from 'partial area 2 (202)', which is the body area of '(200)'. In this way, 'data collection server 1 (131)' collects web content data from the body area of 'web page E (200)', thereby preventing unnecessary data existing in other areas of 'web page E (200)' from being collected. It may not be possible.

Ultimately, the web content crawling management server 110 according to the present invention receives a request to transmit information in the body area from the data collection server each time a predetermined data collection server collects content data of a specific web page. In this way, when a request to transmit information about the body area is received, the body area of the web page is checked in response to the request and the information about it is transmitted to the data collection server, so that the data collection servers transmit the information to the body area of the web page. It can support collecting only more refined web content data from .

According to one embodiment of the present invention, the information transmission unit 119 may further include a table creation unit 127 and an information rapid transmission unit 128.

When the table creation unit 127 determines that the candidate body area is the body area of the first web page by the body area determination unit 125, the table creation unit 127 selects a reference area corresponding to each of the third web pages to the first web page. After designating each of the three web pages as a body area, a body area table recording information on the body area of each of the third web pages is created.

After the body area table is created by the table creation unit 127, the information quick transmission unit 128 transmits the data to the second data collection server, which is another one of the k data collection servers 131, 132, and 133. As the web content data collection operation for the fourth web page, which is one of the third web pages, begins, a request to transmit information about the body area of the fourth web page is received from the second data collection server. , After extracting information about the body area of the fourth web page recorded in the body area table from the body area table, without performing a process for determining the body area of the fourth web page, 4 Information about the body area of the web page is transmitted to the second data collection server.

Hereinafter, the operations of the table creation unit 127 and the information rapid transmission unit 128 will be described in detail using an example.

First, as in the above-mentioned example, as the reference position corresponding to each of 'web page A and web page B' is confirmed by the body area determination unit 125 to match the first position 211, the 'part If it is determined that 'area 2 (202)' is the body area of 'web page E (200)', the table creation unit 127 generates 'part' which is the reference area corresponding to each of 'web page A and web page B'. A body area table recording information about area 5 and partial area 6' can be created as shown in Table 4 below.

web page body area Web page A partial area 5 web page B Partial area 6

In this way, after the table creation unit 127 creates the text area table as shown in Table 4 above, the web content data collection for 'web page A' begins in 'data collection server 2 (132)'. , Let us assume that a request to transmit information about the body area of 'web page A' is received by the web content crawling management server 110 from 'data collection server 2 (132)'.

Then, the information quick transmission unit 128 uses the area confirmation unit 121, the location confirmation unit 122, the additional area confirmation unit 123, the additional location confirmation unit 124, and the body area determination unit 125. Without performing a process to determine the body area of 'Web Page A', information about 'Partial Area 5', which is the body area of 'Web Page A', is extracted from the Body Area Table as shown in Table 4 above, and then 'Partial Area 5' is Information about 'Area 5' can be transmitted to 'Data Collection Server 2 (132)'.

Meanwhile, as the content data collection for 'Web Page C' begins in 'Data Collection Server 3 (133)', information about the body area of 'Web Page C' is collected from 'Data Collection Server 3 (133)' In the case where the transmission request is received by the web content crawling management server 110, since information about the body area of 'web page C' is not recorded in the body area table as shown in Table 4 above, the web content crawling management server (110) is the body of 'web page C' through the area confirmation unit 121, location confirmation unit 122, additional area confirmation unit 123, additional location confirmation unit 124, and body area determination unit 125. After determining the area, information about the body area of 'Web Page C' can be transmitted to 'Data Collection Server 3 (133)'.

Figure 4 is a flow chart illustrating a method of operating a web content crawling management server that supports efficient distributed crawling of a plurality of web sites according to an embodiment of the present invention.

In step S410, the URL of one or more web pages constituting each website is collected from the plurality of web sites, thereby generating a collection data set consisting of the URLs of the plurality of web pages.

In step S420, the URL of each of the plurality of web pages included in the collected data set is stored in a URL queue.

In step S430, the URLs of each of the plurality of web pages stored in the URL queue are sequentially extracted according to the storage order in the URL queue, and k (k is a natural number of 2 or more) data collection servers. Allocate in a round robin manner.

In step S440, a work command instructing to collect web content data by accessing a web page of a URL assigned to each data collection server is transmitted to each of the k data collection servers.

At this time, according to one embodiment of the present invention, in step S420, the URLs of each of the plurality of web pages are grouped by the website from which each URL is collected, thereby forming a URL group corresponding to each of the plurality of web sites. generating a URL group corresponding to each of the plurality of web sites, allocating a group storage sequence number in descending order in descending order of the number of URLs included in each URL group; and assigning a group storage sequence number to the plurality of web sites in descending order, and The URLs of each page are divided into URL groups and sequentially stored in the URL queue according to the group storage order assigned to each URL group. However, for URLs included in the same URL group, the storage order is random. It may include storing the URL of each of the plurality of web pages in the URL queue by storing the URL in the URL queue.

In addition, according to an embodiment of the present invention, a method of operating the web content crawling management server includes, after the work command is transmitted to each of the k data collection servers, in the k data collection servers, the plurality of Whenever a web content data collection operation for one of the web pages is started, a request to transmit information about the body area of the web page for web content data collection is received from the k data collection servers. step, and in a first data collection server that is one of the k data collection servers, as a web content data collection operation for a first web page that is one of the plurality of web pages is started, the first data When a request to transmit information about the body area of the first web page is received from the collection server, the information about the body area of the first web page is checked, and then the information about the body area of the first web page is sent to the above. It may further include transmitting to the first data collection server.

At this time, according to one embodiment of the present invention, the step of transmitting information about the body area of the first web page to the first data collection server includes web content for the first web page from the first data collection server. As the data collection operation begins, when a request to transmit information about the body area of the first web page is received from the first data collection server, a body area confirmation event to check the body area of the first web page Generating a step, when the body area confirmation event occurs, rendering the first web page through a web browser, parsing the markup language code constituting the rendered first web page, and Checking the sizes of a plurality of sub-areas constituting a page, designating a sub-area with the largest size among the plurality of sub-areas constituting the first web page as a candidate body area, Confirming a first position, which is an arrangement position on a first web page, from a plurality of second web pages constituting a first website, which is a website including the first web page, n (n is 2 or more) After randomly selecting (a natural number) web pages, rendering each of the n web pages through the web browser, parsing the markup language code of each of the n rendered web pages, Checking the size of a plurality of sub-areas constituting each web page, for each of the n web pages, a sub-area with the largest size among the plurality of sub-areas constituting each web page is selected as a reference area. After designating the reference area corresponding to each of the n web pages, the placement position of the reference area corresponding to each of the n web pages on each web page is determined by specifying the reference area corresponding to each of the n web pages. Checking with the corresponding reference location, checking whether the reference location corresponding to each of the n web pages matches the first location, and selecting a third web page that is more than a preset reference number among the n web pages. If it is confirmed that the reference position corresponding to each of the first positions matches the first position, determining the candidate body area to be the body area of the first web page, and determining that the candidate body area is the body area of the first web page If it is determined to be an area, it may include the step of designating the information on the candidate body area as information on the body area of the first web page and transmitting it to the first data collection server.

In addition, according to one embodiment of the present invention, the step of confirming the first location includes determining the location of a reference vertex, which is one of the vertices constituting the candidate body area, on the first web page. It can be confirmed as the first position, which is the arrangement position on the first web page, and in the step of checking as the reference position corresponding to each of the n web pages, a reference area corresponding to each of the n web pages is selected. Among the constituting vertices, the position of the vertex corresponding to the reference vertex on each web page can be confirmed as the reference position corresponding to each of the n web pages, and is assumed to be the body area of the first web page. In the determining step, the reference location corresponding to each of the n web pages is compared with the first location, and the separation distance between the reference location corresponding to each of the n web pages and the first location is set to a preset standard. If it is confirmed that it is within the distance, it can be confirmed that the reference location corresponding to each of the n web pages matches the first location.

Additionally, according to one embodiment of the present invention, the step of transmitting information about the body area of the first web page to the first data collection server determines that the candidate body area is the body area of the first web page. Then, designating the reference area corresponding to each of the third web pages as the body area of each of the third web pages, and then creating a body area table recording information about the body area of each of the third web pages. After the step and the body area table is created, a web content data collection operation for a fourth web page, which is one of the third web pages, is performed in a second data collection server, which is another one of the k data collection servers. As it starts, when a request to transmit information about the body area of the fourth web page is received from the second data collection server, the body area is Extracting information about the body area of the fourth web page recorded on the body area table from the table, and then transmitting the information about the body area of the fourth web page to the second data collection server. More may be included.

Above, with reference to FIG. 4, a method of operating a web content crawling management server that supports efficient distributed crawling of a plurality of web sites according to an embodiment of the present invention has been described. Here, the operating method of the web content crawling management server that supports efficient distributed crawling of a plurality of web sites according to an embodiment of the present invention is a configuration of the operation of the web content crawling management server 110 described using FIG. 1. Since it may correspond to , a more detailed description thereof will be omitted.

The method of operating a web content crawling management server that supports efficient distributed crawling of a plurality of web sites according to an embodiment of the present invention can be implemented as a computer program stored in a storage medium to be executed through combination with a computer. .

In addition, the method of operating a web content crawling management server that supports efficient distributed crawling of a plurality of web sites according to an embodiment of the present invention is implemented in the form of program instructions that can be executed through various computer means and is computer readable. Can be recorded on media. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and constructed for the present invention or may be known and usable by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, the present invention has been described with specific details such as specific components and limited embodiments and drawings, but this is only provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , those skilled in the art can make various modifications and variations from this description.

Accordingly, the spirit of the present invention should not be limited to the described embodiments, and the scope of the patent claims described below as well as all modifications that are equivalent or equivalent to the scope of this patent claim shall fall within the scope of the spirit of the present invention. .

110: Web content crawling management server that supports efficient distributed crawling of multiple websites
111: URL collection unit 112: URL storage processing unit
113: URL allocation unit 114: Command transmission unit
115: URL group creation unit 116: Group storage sequence allocation unit
117: storage processing unit 118: request receiving unit
119: Information transmission unit 120: Event generation unit
121: Area confirmation unit 122: Location confirmation unit
123: Additional area confirmation unit 124: Additional location confirmation unit
125: Body area decision unit 126: Transmission processing unit
127: Table creation unit 128: Information rapid transmission unit
131, 132, 133: k data collection servers

Claims (14)

In the web content crawling management server that supports efficient distributed crawling of multiple websites,
A URL collection unit that generates a collection data set consisting of the URLs of a plurality of web pages by collecting all URLs (Uniform Resource Locators) of one or more web pages constituting each website from the plurality of web sites. ;
a URL storage processor that stores URLs of each of the plurality of web pages included in the collection data set in a URL queue;
The URLs of each of the plurality of web pages stored in the URL queue are sequentially extracted according to the storage order in the URL queue and sent to k (k is a natural number of 2 or more) data collection servers in a round robin. A URL allocation unit that allocates using the robin) method; and
For each of the k data collection servers, a command transmission unit that transmits a work command instructing to collect web content data by accessing the web page of the URL assigned to each data collection server.
Including,
The URL storage processing unit
a URL group generator that generates a URL group corresponding to each of the plurality of web sites by grouping the URLs of each of the plurality of web pages according to the web site from which each URL is collected;
a group storage sequence number allocator for assigning a group storage sequence number to URL groups corresponding to each of the plurality of web sites in descending order of the number of URLs included in each URL group; and
The URLs of each of the plurality of web pages are divided into URL groups and sequentially stored in the URL queue according to the group storage order assigned to each URL group. For URLs included in the same URL group, A storage processing unit that stores URLs of each of the plurality of web pages in the URL queue by storing them in the URL queue in a random storage order.
A web content crawling management server that includes. delete According to paragraph 1,
After the work command is transmitted to each of the k data collection servers, each time a web content data collection operation for any one of the plurality of web pages is started in the k data collection servers, a request receiver that receives a request for transmission of information about the body area of a web page for web content data collection from the k data collection servers; and
In a first data collection server, which is one of the k data collection servers, as a web content data collection operation for a first web page, which is one of the plurality of web pages, is started, the first data collection server When a request to transmit information about the body area of the first web page is received, the information about the body area of the first web page is checked, and then the information about the body area of the first web page is sent to the first web page. Information transmission unit that transmits to the data collection server
A web content crawling management server further comprising: According to paragraph 3,
The information transmission unit
As the first data collection server starts collecting web content data for the first web page, when a request to transmit information about the body area of the first web page is received from the first data collection server, an event generator that generates a body area confirmation event to check the body area of the first web page;
When the body area confirmation event occurs, the first web page is rendered through a web browser, then the markup language code constituting the rendered first web page is parsed, and the first web page is rendered. 1 an area confirmation unit that checks the sizes of a plurality of partial areas constituting a web page;
After designating a partial area with the largest size among a plurality of partial areas constituting the first web page as a candidate body area, confirming the first position, which is the placement position of the candidate body area, on the first web page. Location confirmation unit;
From a plurality of second web pages constituting the first website, which is a website including the first web page, n (n is a natural number of 2 or more) web pages are randomly selected and displayed through the web browser. After rendering each of the n web pages, parsing the markup language code of each of the rendered n web pages to check the size of a plurality of partial regions constituting each of the n web pages. Area confirmation unit;
For each of the n web pages, a reference area corresponding to each of the n web pages is designated by designating a partial area with the largest size among a plurality of partial areas constituting each web page as a reference area. Afterwards, an additional position confirmation unit that checks the arrangement position on each web page of the reference area corresponding to each of the n web pages as a reference position corresponding to each of the n web pages;
By checking whether the reference position corresponding to each of the n web pages matches the first location, the reference position corresponding to each of the third web pages of a preset reference number or more among the n web pages is determined by the first location. a body area determination unit that determines that the candidate body area is a body area of the first web page when it is confirmed that it matches the first location; and
When the candidate body area is determined to be the body area of the first web page, information on the candidate body area is designated as information on the body area of the first web page and transmitted to the first data collection server. processing department
A web content crawling management server that includes. According to paragraph 4,
The location confirmation unit
Confirming the position of a reference vertex, which is one of the vertices constituting the candidate body area, on the first web page as the first position, which is the placement position of the candidate body area on the first web page,
The additional location confirmation unit
Among the vertices constituting the reference area corresponding to each of the n web pages, the position of the vertex corresponding to the reference vertex on each web page is confirmed as the reference position corresponding to each of the n web pages. And
The body area determination unit
By comparing the reference position corresponding to each of the n web pages with the first location, it is determined that the separation distance between the reference position corresponding to each of the n web pages and the first location is within a preset reference distance. When confirmed, a web content crawling management server characterized in that it is confirmed that the reference location corresponding to each of the n web pages matches the first location. According to paragraph 4,
The information transmission unit
If the candidate body area is determined to be the body area of the first web page, the reference area corresponding to each of the third web pages is designated as the body area of each of the third web pages, and then the third web page a table creation unit that generates a text area table recording information on each text area; and
After the body area table is created, the second data collection server, which is another one of the k data collection servers, starts collecting web content data for the fourth web page, which is one of the third web pages. Accordingly, when a request to transmit information about the body area of the fourth web page is received from the second data collection server, the body area of the fourth web page is selected from the body area table without performing a process for determining the body area of the fourth web page. , an information rapid transmission unit that extracts information about the body area of the fourth web page recorded on the body area table and then transmits the information about the body area of the fourth web page to the second data collection server.
A web content crawling management server further comprising: In a method of operating a web content crawling management server that supports efficient distributed crawling of a plurality of websites,
Collecting all URLs (Uniform Resource Locators) of one or more web pages constituting each web site from the plurality of web sites, thereby generating a collection data set consisting of URLs of a plurality of web pages;
storing the URL of each of the plurality of web pages included in the collected data set in a URL queue;
The URLs of each of the plurality of web pages stored in the URL queue are sequentially extracted according to the storage order in the URL queue and sent to k (k is a natural number of 2 or more) data collection servers in a round robin. robin) method; and
For each of the k data collection servers, transmitting a work command instructing to collect web content data by accessing a web page of a URL assigned to each data collection server.
Including,
The step of storing the URL of each of the plurality of web pages in the URL queue is
grouping the URLs of each of the plurality of web pages according to the website from which each URL was collected, thereby creating a URL group corresponding to each of the plurality of web sites;
assigning group storage order numbers in descending order to the URL groups corresponding to each of the plurality of web sites in descending order of the number of URLs included in each URL group; and
The URLs of each of the plurality of web pages are divided into URL groups and sequentially stored in the URL queue according to the group storage order assigned to each URL group. For URLs included in the same URL group, Storing the URL of each of the plurality of web pages in the URL queue by storing them in the URL queue in a random storage order.
A method of operation of a web content crawling management server including. delete In clause 7,
After the work command is transmitted to each of the k data collection servers, each time a web content data collection operation for any one of the plurality of web pages is started in the k data collection servers, Receiving, from the k data collection servers, a request to transmit information on the body area of a web page for web content data collection; and
In a first data collection server, which is one of the k data collection servers, as a web content data collection operation for a first web page, which is one of the plurality of web pages, is started, the first data collection server When a request to transmit information about the body area of the first web page is received, the information about the body area of the first web page is checked, and then the information about the body area of the first web page is sent to the first web page. Transfer steps to data collection server
A method of operating a web content crawling management server further comprising: According to clause 9,
The step of transmitting information about the body area of the first web page to the first data collection server
As the first data collection server starts collecting web content data for the first web page, when a request to transmit information about the body area of the first web page is received from the first data collection server, generating a body area confirmation event to check the body area of the first web page;
When the body area confirmation event occurs, the first web page is rendered through a web browser, then the markup language code constituting the rendered first web page is parsed, and the first web page is rendered. 1 Checking the sizes of a plurality of partial areas constituting a web page;
After designating a partial area with the largest size among a plurality of partial areas constituting the first web page as a candidate body area, confirming the first position, which is the placement position of the candidate body area, on the first web page. step;
From a plurality of second web pages constituting the first website, which is a website including the first web page, n (n is a natural number of 2 or more) web pages are randomly selected and displayed through the web browser. After rendering each of the n web pages, parsing the markup language code of each of the rendered n web pages to check the size of a plurality of partial regions constituting each of the n web pages. ;
For each of the n web pages, a reference area corresponding to each of the n web pages is designated by designating a partial area with the largest size among a plurality of partial areas constituting each web page as a reference area. Afterwards, confirming the arrangement position of the reference area corresponding to each of the n web pages on each web page as the reference position corresponding to each of the n web pages;
By checking whether the reference position corresponding to each of the n web pages matches the first location, the reference position corresponding to each of the third web pages of a preset reference number or more among the n web pages is determined by the first location. If confirmed to match the first location, determining the candidate body region to be a body region of the first web page; and
When the candidate body area is determined to be the body area of the first web page, designating the information about the candidate body area as information about the body area of the first web page and transmitting it to the first data collection server.
A method of operation of a web content crawling management server including. According to clause 10,
The step of confirming the first location is
Confirming the position of a reference vertex, which is one of the vertices constituting the candidate body area, on the first web page as the first position, which is the placement position of the candidate body area on the first web page,
The step of checking the reference location corresponding to each of the n web pages is
Among the vertices constituting the reference area corresponding to each of the n web pages, the position of the vertex corresponding to the reference vertex on each web page is confirmed as the reference position corresponding to each of the n web pages. And
The step of determining that it is the body area of the first web page is
By comparing the reference position corresponding to each of the n web pages with the first location, it is determined that the separation distance between the reference position corresponding to each of the n web pages and the first location is within a preset reference distance. When confirmed, a method of operating a web content crawling management server, characterized in that it is confirmed that the reference location corresponding to each of the n web pages matches the first location. According to clause 10,
The step of transmitting information about the body area of the first web page to the first data collection server
If the candidate body area is determined to be the body area of the first web page, the reference area corresponding to each of the third web pages is designated as the body area of each of the third web pages, and then the third web page generating a text area table recording information on each text area; and
After the body area table is created, the second data collection server, which is another one of the k data collection servers, starts collecting web content data for the fourth web page, which is one of the third web pages. Accordingly, when a request to transmit information about the body area of the fourth web page is received from the second data collection server, the body area of the fourth web page is selected from the body area table without performing a process for determining the body area of the fourth web page. , extracting information about the body area of the fourth web page recorded in the body area table, and then transmitting the information about the body area of the fourth web page to the second data collection server.
A method of operating a web content crawling management server further comprising: A computer-readable recording medium recording a computer program for executing the method of any one of claims 7, 9, 10, 11, or 12 through combination with a computer. A computer program stored in a storage medium for executing the method of any one of claims 7, 9, 10, 11, or 12 through combination with a computer.

Images