KR20130017620A - Apparatus and method for archiving web resource - Google Patents

Abstract

PURPOSE: A multipurpose device for web resource archiving and a method thereof are provided to perform archiving for a dynamic webpage with a script and a plug-in as well as a static web page without the omission of web resources. CONSTITUTION: A web collection crawler(2010) accesses a URL(Uniform Resource Locator) to collect web document data and link file data. A web snap shot robot(2030) accesses the URL accessed by the web collection crawler to capture a web page screen as snap shot data. A database(2040) stores the web document data, the link file data, and the snap shot data. A web archiving mapper(2060) maps data corresponding to the same URL among the web document data, the link file data, and the snap shot data in order to output the same to the database. A web archiving managing unit(2050) hierarchically connects the mapped data to provide the connected web data. [Reference numerals] (2010) Web collection crawler; (2020) URL manager; (2030) Web snap shot robot; (2040) Database; (2050) Web archiving managing unit; (2060) Web archiving mapper; (2070) Web archiving browser; (2080) Web archiving update unit

Description

Apparatus and method for archiving web resource}

The present invention relates to a method and apparatus for archiving web resources. More particularly, the present invention relates to a method and apparatus for collecting and storing web resources using a snapshot robot to preserve web resources having various formats in a nearly circular form.

Web resources are an important medium that information users can access quickly and easily in the information age. Web resources are important resources used in various fields such as personal communication, publishing, academics, and e-commerce as well as science and technology communication.

However, these web resources are updated or destroyed periodically or aperiodically. As a result, it is not possible for users to obtain previously existing web resources from updated web pages. Therefore, the importance of web archiving to collect and preserve web resources with weak data persistence is emphasized. As the related research of web archiving increases, the development of a web collection crawler for collecting web resources is required.

The tool for archiving web resources is called a crawler or havester. Early crawlers were designed based on general indexing robots, but later, a dedicated crawler for archiving web resources, NEDLIB harvesters developed by the Networked European Deposit Library (NEDLIB). It has been used in many national libraries, mainly in Europe. Later developments include Heritrix from the Internet Archive and HTTrack, which can store websites all at once. Since then, web collection crawlers such as DeepArc, PageVault, and WGet have been developed for various purposes.

Collecting web resources consisting of text-based static web pages linked by standard HTML links is easily implemented by existing web collection crawlers. However, collecting web resources consisting of dynamic web pages using a steadily increasing number of scripts or plug-in techniques such as JavaScript or Flash is often not possible with traditional web collection crawlers. Because the results of script execution depend on the type of web browser, web pages that employ JavaScript are difficult to collect successfully. Because Flash is not only a plug-in but also a commercial format, collecting web resources is difficult.

1 is a diagram illustrating components of a general web page screen.

The general web page screen includes not only simple text 1010 but also a dynamic image 1030 using a static image 1020 and a script.

Simple text 1010 and static image 1020 are collected and stored intact without damage by the existing web collection crawler. Even if a user retrieves stored web archive data and restores the web page, simple text and static images can be restored as they were collected.

However, the dynamic image 1030 using a script or a plug-in is not stored as it is when the existing web collection crawler collects and stores it. This is because a dynamic image can only be completely restored when its associated script or plug-in is saved and executed. Therefore, even if an existing web collection crawler that does not support it collects and saves the web page, when the user visits the web page using the web archive material, the dynamic image area is not restored and is represented as a blank space on the screen of the web page. do.

In addition, the Web collection crawler's collection of web resources for web pages based on internal databases is often impossible. This is because, in the case of web pages using an internal database, even if the web resources are collected and restored, some of the resources are not collected and thus the contents are not known.

In addition, there is not only a surface web that can be freely accessed through a search engine, but also a deep web that is not indexed by a general search engine. Deep webs are often caused by the technical limitations of the web-gathering crawler itself, or even when a web page authentication process is required.

Even if a web site operator places a robot.txt file on a web site using the robot exclusion protocol, the web page crawler cannot access the web page.

As such, there are various tools for archiving web resources, but it is difficult to collect web resources perfectly because of resources in the form of scripts and plug-ins, and other reasons. In addition, collecting deep web resources takes more time and difficulty than surface webs. As a result, even if the web resource archiving is performed, there is a problem that the user cannot restore and present the perfect web page.

The present invention is to solve the above problems, the technical problem to be achieved by the present invention to the web archiving apparatus and method, it is possible to effectively recover the web page by collecting the web resources without suffering the above-mentioned problems. .

In order to solve the above technical problem, the web resource archiving apparatus according to an embodiment of the present invention is a web collection crawler, a web collection crawler that accesses a URL (Uniform Resource Locator) to collect web document data and link file data Web snapshot robot that captures web page screen as snapshot data by accessing the same URL, database storing web document data, link file data and snapshot data, web document data stored in database, link file data and snap Web archiving mapper which maps the data corresponding to the same URL among the shot materials and outputs it to the database, instructs mapping to web archiving mapper, and provides web data connected by hierarchically linking the mapped data. It includes an archiving management unit.

In addition, the web resource archiving apparatus according to an embodiment of the present invention stores a URL that the web collection crawler accesses, and the URL manager for managing in a FIFO (First In First Out) method so that the URL to be accessed by the web snapshot robot is not duplicated It further includes.

In addition, the web resource archiving apparatus according to an embodiment of the present invention further includes a web archiving browser for browsing the web resources by using a web hierarchical structure.

In addition, the web resource archiving apparatus according to an embodiment of the present invention further includes a web archiving update unit for requesting an update by determining whether the collected and stored data are updated.

In addition, the web resource archiving apparatus according to an embodiment of the present invention repeatedly captures the web page screen so that the web snapshot robot generates snapshot data for all the deep web resources belonging to the corresponding URL.

In addition, the web resource archiving apparatus according to an embodiment of the present invention converts the captured data into a specific image format and stores the captured data in a database storing the captured data.

In order to solve the above technical problem, the web resource archiving method according to an embodiment of the present invention, accessing the Uniform Resource Locator (URL) to collect and store web document data and link file data, access to the above URL Capturing and storing the web page screen as snapshot data, and mapping and storing data corresponding to the same URL among the stored web document data, link file data, and snapshot data, and connecting the mapped data hierarchically. linking to provide linked web material.

In addition, the web resource archiving method according to an embodiment of the present invention further comprises the step of managing in a FIFO (First In First Out) scheme so that the URL to be accessed is not duplicated in order to store the accessed URL and capture it as snapshot data. .

In addition, the web resource archiving method according to an embodiment of the present invention further includes the step of browsing the stored web resources using a hierarchical structure.

In addition, the web resource archiving method according to an embodiment of the present invention further comprises the step of requesting an update by determining whether to update the collected and stored data.

In addition, the web resource archiving method according to an embodiment of the present invention repeats the step of capturing and storing the web page screen as snapshot data so that the capturing and storing of all the deep web resources belonging to the URL are completed.

In addition, in the web resource archiving method according to an embodiment of the present invention, storing the captured data in the database converts the captured data into a specific image format and stores the captured data.

Method and apparatus for archiving web resources according to the present invention has the following effects.

According to an embodiment of the present invention, an archiving without a web resource is effective for not only text-based static web pages but also dynamic web pages using scripts and plug-ins.

According to an embodiment of the present invention, there is an effect of searching the snapshot data of the relevant web page by text search.

According to an embodiment of the present invention, there is an effect of hierarchically browsing snapshot data for a web page.

1 is a diagram illustrating components of a general web page screen.
2 is a block diagram of an apparatus for archiving web resources according to an embodiment of the present invention.
3 is a diagram illustrating a mapping operation for collected data of an apparatus for archiving web resources according to an embodiment of the present invention.
4 is a diagram illustrating web hierarchical structured data by an apparatus for archiving web resources according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating a first browsing screen for web archive material provided by an apparatus for archiving a web resource according to an embodiment of the present invention.
FIG. 6 is a diagram illustrating a second browsing screen for web archive material provided by an apparatus for archiving web resources according to an embodiment of the present invention.
7 is a flowchart illustrating a method of archiving a web resource according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and accompanying drawings, but the present invention is not limited to or limited by the embodiments.

As used herein, terms used in the present invention are selected from general terms that are widely used in the present invention while taking into account the functions of the present invention, but these may vary depending on the intention or custom of a person skilled in the art or the emergence of new technologies. In addition, in certain cases, there may be a term arbitrarily selected by the applicant, in which case the meaning thereof will be described in the description of the corresponding invention. Therefore, it is intended that the terminology used herein should be interpreted based on the meaning of the term rather than on the name of the term, and on the entire contents of the specification.

The web resource archiving apparatus according to the present invention visits a web page and stores web document data, link file data, and snapshot data of a web page, which are components included in the web page. It also provides users with web archive data by mapping the stored data, structuring them in a web hierarchical structure and browsing them.

2 is a block diagram of an apparatus for archiving web resources according to an embodiment of the present invention.

The web resource archiving apparatus of FIG. 2 includes a web collection crawler 2010, a URL manager 2020, a web snapshot robot 2030, a database 2040, a web archiving manager 2050, a web archiving mapper 2060, and a web. An archiving browser 2070 and a web archiving updater 2080 may be included.

Here, the URL manager 2020, the web archiving browser 2070, and the web archiving update unit 2080 are not essential components and may be excluded according to embodiments.

The web collection crawler 2010 may access web pages and collect web document data and link file data composed of simple text, images, and the like of the web pages. At this time, the web page includes a plurality of sub-web pages connected by a link, and the web collection crawler 2010 may collect web document data and link file data included in the sub-web page.

The URL manager 2020 may manage URLs of web pages visited by the web collection crawler 2010. The URL manager 2020 manages the URL of the web page visited by the web collection crawler 2010 to prevent the web snapshot robot 2030 from repeatedly visiting the same web page. In addition, web pages that are subject to snapshot data collection can be selected in a first-in-first-out (FIFO) manner for registered URLs.

The web snapshot robot 2030 may revisit the web page that the web collection crawler has accessed to capture a screen of the web page as snapshot data. Or you can refer to the URL manager to access the web page and capture the web page screen as snapshot data. The capture of the web page screen is performed on a plurality of sub web pages connected to the main page as in the data collection of the web collection crawler 2010, and the snapshot operation is generated by capturing all the sub web pages.

Since the snapshot data generation of the web snapshot robot 2030 captures the screen of each web page as it is, all the components of the web page can be collected. In other words, since there is no need to collect additional scripts or plug-in programs, even if the components are JavaScript or Flash, they can be reflected in the snapshot data.

In addition, even when the web resource archiving is prohibited by the robot exclusion protocol, the web snapshot robot does not collect web resources, but only captures them by accessing the web page. Do.

The snapshot data generated by the web snapshot robot 2030 may be converted into a specific image format and stored in the database 2040 to be described below. The image format may be, for example, JPG, GIF, PNG, BMP, TIFF, or the like.

The database 2040 may store web document data and link file data collected by the web collection crawler 2010, and snapshot data captured by the web snapshot robot 2030. In addition, the results output from the web archiving management unit 2050 or the web archiving mapper 2060 to be described below may be received and stored.

The web archiving manager 2050 may manage materials stored in the database 2040. You can modify or delete the stored web pages, derive statistics on the stored data, or monitor the web pages on which the data is collected. In addition, a search service for data stored in the database 2040 is provided to general users, and for this purpose, indexing of data may be performed.

In addition, the web archiving management unit 2050 may perform a structured work on the data mapped by the web archiving mapper 2060 below. In general, a web page has a hierarchical structure because it includes a plurality of sub-web pages linked by a link. Accordingly, the web archiving management unit 2050 may provide the linked web material by hierarchically linking the stored web materials in order to preserve the web page similar to the original.

The web archiving mapper 2060 may perform mapping for data stored in the database 2040. The web archiving mapper 2060 may mutually map the web document data and link file data collected and stored by the web collection crawler 2010 and the snapshot data collected by the web snapshot robot 2030 based on the URL. The data collected or generated from the same URL by this operation can be derived on the result screen together in browsing the data search results by the user.

Although snapshot data may contain text in the image, it is only an image, not text. Therefore, even if the user searches by the keyword in the image, it is impossible to derive the search result. Accordingly, the web archiving mapper 2060 enables keyword search for snapshot data by mapping with text and web document data collected at the same URL.

The web archiving browser 2070 may browse the web archive material to the user. That is, when a user inputs a desired search word, the web archiving manager 2050 searches for web archive data stored in the database 2040 and displays the result to the user. The web archiving browser 2070 displays web materials hierarchically linked by the web archiving management unit 2050 and displays sub-web pages of the selected web page. This allows the user to navigate from the parent web page to the child web page and retrieve the data.

The web archiving update unit 2080 receives a web page monitoring command of the web archiving management unit 2050 to check whether the stored web archive data is updated, and if the update of the data is confirmed, sends a web resource collection command to the web collection crawler. Can be directed. Checking whether the stored web archive data is updated periodically or aperiodically.

3 is a diagram illustrating a mapping operation for collected data of an apparatus for archiving web resources according to an embodiment of the present invention.

The web archiving mapper 3010 and the database 3020 may be involved in the mapping operation. The data stored by the web collection crawler are URLs, content data, and attachment file data of a web page. The data stored by the web snapshot robot are URL and snapshot data. The mapping operation is performed by the web archiving mapper 3010 on the stored data. Since the data collected by the web collection crawler and the data collected by the web snapshot robot are stored together with the URL information as described above, the mapping is performed based on the URL information. This mapping operation defines the associations between the data collected or created at the same URL and stored.

After the web archiving mapper 3010 performs the mapping operation, the web archiving mapper 3010 stores the mutually mapped data in the database 3020 again. Thus, when searching the web archive data by the user, even if the user searches the text belonging to the web page, it is possible to output the snapshot data of the web page to which the text belongs as a search result.

4 is a diagram illustrating web hierarchical structured data by an apparatus for archiving web resources according to an embodiment of the present invention.

The data stored in the database and mutually mapped by the above-mentioned web archiving mapper are hierarchically linked by the web archiving manager. In FIG. 4, the site depth means the level of the web layer. Level 1 means the upper web page, and the lower web page as the level increases from level 2 to level n.

In each level, there are html materials 4010, 4020, and 4030 included in the same web layer, and they are not interconnected because they belong to the same web layer. The html data 4010 is connected to the snapshot data 4011, which means that the data are mutually mapped by the web archiving mapper. The level 1 html data 4010 is connected to two level 2 html materials 4040 and 4050, and the level 2 html data is linked to the next level of html data 4050 and 4060.

Thus, web hierarchical structured material allows a user to continue a web page from html material 4010 belonging to level 1 corresponding to the parent web page to level 2 or higher html material 4040 and 4050 corresponding to the child web page. It is possible to navigate. This web hierarchical structure allows users to search web archive data as if they were accessing a real web page.

FIG. 5 is a diagram illustrating a browsing screen for web tier 1 for web archive material provided by an apparatus for archiving web resources according to an embodiment of the present invention.

At the top of the browsing screen, there is a blank 5010 for searching and a search button. Below the search field is text 5020 indicating the level of the current web tier and the search result format. At the bottom of the browsing screen, the search results for the above search terms are expressed in a snapshot-thumbnail format 5030.

For example, when a user inputs KoreanScience as a search word as shown in the drawing, the search result 5010 indicates that the search result 5020 of the snapshot-thumbnail format at the web depth level 1 is displayed below the search box 5010. The web resource archiving device searches for snapshot data cross-mapped with KoreanScience. When the search is completed, the web resource archiving device displays the search results by listing the retrieved snapshot data as a snapshot-thumbnail image 5030 at the bottom of the browsing screen.

The user can click on the search result image to view the original snapshot data of the web page.

6 is a diagram illustrating a browsing screen in web layer 2 for web archive material provided by an apparatus for archiving web resources according to an embodiment of the present invention.

At the top of the browsing screen, there is a blank 6010 for searching and a search button. Below the search box is the original snapshot data for web pages one level above the current web tier. That is, the snapshot image 6020 selected by clicking among the snapshot images of the layer higher than the current search layer is enlarged and displayed. Below that is text 6030 that indicates the level of the current web tier and the search result format. At the bottom of the browsing screen, the search results for the above search terms are expressed in a snapshot-thumbnail format (6040).

For example, when a user clicks on one of the search result images of FIG. 5, the original snapshot data 6020 of the clicked image is displayed below the search field. In addition, the web hierarchy level is changed to level 2, which is the level of subweb pages of the web page displayed by the original snapshot data (6030). At the bottom of the browsing screen, subweb pages for the web page of the displayed original snapshot material are displayed as an image 6040 in snapshot-thumbnail format.

In this way, the user using the web resource archiving device can continuously search the sub web page by clicking the sub snapshot-thumbnail format.

7 is a flowchart illustrating a method of archiving a web resource according to an embodiment of the present invention.

The web collection crawler collects the web document data and the link file data by accessing the URL and stores the data in the database (S7010).

At this time, the web collection crawler may also collect and store web resources for sub-web pages linked to the web pages of the URL. When saving data, the above mentioned data and the URL of the web page can be saved together.

The web snapshot robot accesses the URL that the web collection crawler accesses, captures the web page screen as snapshot data, and stores the data in the database (S7020).

The URL accessed by the web snapshot robot can be accessed by following the same URL that the web collection crawler accessed. The web snapshot robot accesses a web page of a URL where a web collection crawler accesses and collects web resources, that is, web document data and link file data, and captures and stores the web page as snapshot data. When saving snapshot data, save the URL of the web page.

In an embodiment different from the above description, the URL manager may register the access URL of the web collection crawler and inform the web snapshot robot of the URL to access the web snapshot robot in a FIFO manner so as not to be duplicated. This is not an essential step of the web resource archiving method.

The web archiving mapper receives the instructions of the web archiving manager to map the web document data, the link file data, and the snapshot data stored in the database based on the URL (S7030).

When the web resource is saved, the URL that is the source is stored together, so that the mapping between data is based on the URL. Web document data, link file data, and snapshot data stored from the same URL are mapped to each other. These mapped materials are displayed as search results when some of them are included in the search results, and are provided to users using the web archive data.

Snapshot data can not be searched even if the keyword included in the image is searched by keyword, so it is possible to search by keyword by mapping with other data collected from the web page.

The web archiving management unit structures the data mapped to each other in a web hierarchical structure and stores them in a database (S7040).

Since web pages have a web hierarchical structure, it is necessary to structure the web archive data in a web hierarchical structure. Web hierarchical structuring simply links the stored and distributed data to parent web pages. This provides web hierarchical connection material to users of web archives, similar to visiting real web pages.

The web archiving browser displays the web hierarchical structure data stored in the database on the screen (S7050).

The web archiving browser displays the search results as a plurality of snapshot-thumbnail images along with the level information, which is web layer information. This allows the user to check the web hierarchy level to which the search data for the keyword they searched belongs.

The user can click on multiple snapshot-thumbnail images to view the original snapshot data. The web archiving browser also displays a snapshot-thumbnail image for the subweb page associated with the web page corresponding to the clicked snapshot-thumbnail image. This allows the user to continuously view web archive material for linked web pages.

The web archiving update unit checks whether the stored data is updated and updates the data (S7060).

Since the web resources are updated periodically or aperiodically, the web archiving update unit checks this and updates the stored data. The web archiving update unit instructs the web collection crawler to collect and store the above web document data, link file data, and snapshot data when the update of the stored data is confirmed. Thereafter, the web resource archiving apparatus repeats the steps of S7010 to S7040 to update the stored data.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible. Therefore, the scope of the present invention should not be limited by the described embodiments, but should be determined by the equivalents of the appended claims, as well as the appended claims.

Description of the symbols for the main parts of the drawings is as follows.
2010: Web Collection Crawler 2020: URL Manager
2030: Web Snapshot Robot 2040: Database
2050: Web archiving management unit 2060: Web archiving mapper
2070: Web archiving browser 2080: Web archiving update unit

Claims (12)

A web collection crawler that accesses a Uniform Resource Locator (URL) and collects web document data and link file data;
A web snapshot robot accessing a URL accessed by the web collection crawler and capturing a web page screen as snapshot data;
A database for storing the web document data, link file data, and snapshot data;
A web archiving mapper for mapping data corresponding to the same URL among the web document data, link file data, and snapshot data stored in the database and outputting the data to the database; And
And a web archiving management unit for instructing the web archiving mapper to map and hierarchically linking the mapped data to provide the linked web material. The method of claim 1,
The web resource archiving apparatus further includes a URL manager that stores the URL accessed by the web collection crawler and manages the URL to be accessed by the web snapshot robot in a first-in-first-out (FIFO) manner. Web resource archiving device. The method of claim 1,
The web resource archiving apparatus further comprises a web archiving browser for browsing the structured data in the web hierarchical structure with hierarchical information. The method of claim 1,
The web resource archiving apparatus further includes a web archiving updating unit which determines whether to update the stored data and requests an update. The method of claim 1,
The web snapshot robot is a web resource archiving device for capturing a web page screen repeatedly to generate a snapshot data for all the deep web resources belonging to the URL. The method of claim 1,
The web snapshot robot, web resource archiving device for converting the snapshot data into a specific image format and stored in the database. Accessing a Uniform Resource Locator (URL) to collect and store web document data and link file data;
Accessing the URL and capturing and storing a web page screen as snapshot data;
Mapping and storing data corresponding to the same URL among the stored web document data, link file data, and snapshot data; And
And hierarchically linking the mapped materials to provide the linked web materials. 8. The method of claim 7,
The web resource archiving method further includes the step of managing the web resource archiving in a first-in-first-out (FIFO) manner so that the URL to be accessed is not duplicated to store the accessed URL and capture the snapshot data. . 8. The method of claim 7,
The web resource archiving method further comprises the step of browsing the data structured in the web hierarchical structure together with hierarchical information. 8. The method of claim 7,
The web resource archiving method may further include determining and updating the stored data. 8. The method of claim 7,
The capturing and storing the web page screen as snapshot data is repeated until the capturing and storing of all the deep web resources belonging to the URL are completed. 8. The method of claim 7,
The capturing and storing the snapshot data may include converting and storing the snapshot data into a specific image format.

Images