KR20100008466A - Apparatus and method for eliminating double webpage - Google Patents

Abstract

PURPOSE: A device and a method for removing duplicated web pages are provided to eliminate the duplicated with the same contents in the search results. CONSTITUTION: An address converting unit(120) changes the relative address included in the content of the inputted web page into the absolute address. A hash-value calculating unit(130) calculates the hash-value of the absolute address web page transformed to the absolute address. A controller(160) decides whether the calculated hash-value as described above exists in the hash-value list of the hash-value list storage or not. When the corresponding hash-value does not exist, the controller adds hash-value in the hash-value list. The controller stores the collected web page as described above in the web page storage unit.

Description

Apparatus and method for eliminating double webpage}

The present invention relates to an apparatus and method for removing duplicate web pages, and more particularly, to an apparatus and method for removing duplicate web pages for removing duplicate web pages having the same contents in a web page search service.

Recently, with the rapid development of wireless communication technology, mobile communication terminals are becoming an indispensable personal necessity. In addition, mobile communication terminals are expanding from a simple voice call use to chatting or text messages, storing and transmitting image images using a camera phone, real-time video viewing, and wireless Internet search. In addition, with the development of memory-related technology, the mobile communication terminal is also expanding its function as a storage medium capable of storing a large amount of data.

According to the related art, the wireless webpage provided through the wireless Internet has a shorter content than the wired webpage. Thus, the wireless web page can be stored by copying the same content to another URL.

Since wireless webpages are often stored separately in different URLs according to the characteristics of the terminal, duplicate webpages containing the same contents may be frequently generated.

Therefore, a web search service for searching a wireless web page has a problem in that a web page having the same content is repeatedly displayed as a search result.

In addition, this has a problem that results in a decrease in the reliability of the search results.

An object of the present invention is to provide an apparatus and method for removing duplicate web pages that can remove duplicate web pages having the same contents in a wireless website search service.

According to an aspect of the present invention, there is disclosed an apparatus for removing duplicate webpages that removes duplicate webpages when a webpage search server providing a webpage search service crawls.

The apparatus for removing duplicate web pages according to an embodiment of the present invention includes a web page input unit for receiving a web page collected when the crawl is performed, and an address conversion unit for converting a relative address included in the content of the input web page into an absolute address. And a hash value calculated by the hash value calculator for calculating a hash value of the absolute address web page converted to the absolute address, a web page storage unit, a hash value list storage unit, and the hash value calculator. A control unit for determining whether or not the negative hash value list exists, and if the hash value does not exist, adds the hash value to the hash value list, and stores the collected web page in the web page storage unit. .

According to another aspect of the present invention, a duplicate webpage removal method is disclosed in which a duplicate webpage removal apparatus removes a duplicate webpage when a webpage search server providing a webpage search service crawls.

Duplicate web page removal method according to an embodiment of the present invention comprises the steps of receiving a web page found when performing the crawl, converting a relative address included in the content of the web page to an absolute address, the web including the absolute address Calculating a hash value of the page and determining whether the hash value exists in the hash value list, and if not present, not storing the web page, and if not, adding the hash value to the hash value list And storing the web page.

According to another aspect of the present invention, a duplicate webpage removal method is disclosed in which a duplicate webpage removal apparatus removes a duplicate webpage when a webpage search server providing a webpage search service crawls.

According to another embodiment of the present invention, a method of removing a duplicate webpage includes receiving a webpage found when the crawl is performed, extracting an address of the received webpage, and merging the contents of the webpage with the address. Calculating a hash value of the merged result; and determining whether the hash value exists in a hash value list, and if not present, without storing a web page, and if not present, recalling the hash value. Adding to the hash value list and storing the web page.

According to another aspect of the present invention, a recording medium having recorded thereon a program for performing a method for removing a duplicate webpage readable by an electronic device performing the method for removing a duplicate webpage is disclosed.

Receiving a recording medium recording a program for performing a method for removing a duplicate web page according to an embodiment of the present invention receives a web page found when performing the crawl, converting a relative address included in the content of the web page to an absolute address Calculating a hash value of the web page including the absolute address; and determining whether the hash value exists in the hash value list, and if not present, without storing the web page and, if not, the hash value. And adding a value to the hash value list and storing the web page.

According to another aspect of the present invention, a recording medium having recorded thereon a program for performing a method for removing a duplicate webpage readable by an electronic device performing the method for removing a duplicate webpage is disclosed.

A recording medium recording a program for performing a method of removing a duplicate web page according to another embodiment of the present invention comprises the steps of receiving a web page found when performing the crawl, extracting the address of the received web page, the address and the Merging the contents of the web page, calculating a hash value of the merged result, and determining whether the hash value exists in the hash value list, and if present, without storing the web page and not present If the hash value is added to the hash value list, the web page removal method comprising the step of storing the web page is performed.

The present invention has the effect of eliminating duplicate web pages having the same contents in the wireless website search service.

In addition, the present invention has the effect of improving the quality of the web page search service by removing the duplicate of the search results in the wireless web page search service.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all transformations, equivalents, and substitutes included in the spirit and scope of the present invention. In the following description of the present invention, when it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, the term "comprises" or "having" is intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and that one or more other features It is to be understood that the present invention does not exclude the possibility of adding or presenting numbers, steps, operations, components, parts, or a combination thereof.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

For better understanding of the present invention, a search process of a search site that provides a search service in a wired Internet network will be briefly described as an example.

A search site provides a search service by finding a web page that matches a search word and determining the order of the search results.

A web search finds an index with the exact location of all the information. When you do a search, the program checks the index and finds the most relevant search results.

Search results are output through the process of crawling, indexing, and outputting results.

Crawl is the process of finding and adding new and updated pages to the search site's index. Here, search sites bring billions of web pages using various kinds of computers. This task is called crawling.

In this case, crawling is performed by a program, and a program that brings a web page is commonly known as a robot, a bot, a spider, and the like. These programs determine which sites to crawl, how often to crawl them, and how many pages to fetch from each site.

In addition, the crawl process begins with a list of web page URLs that were created in the previous crawl process or added with sitemap data provided by the webmaster. When a web page fetcher visits such a website, it detects a link on each page and adds it to the list of crawled pages. Thus, new sites, changes to existing sites, and inactive links are identified and used to update the index.

Indexing then processes each site crawled by the program to produce a huge index of all words identified on each page and their location. The program also processes information contained in key content tags and attributes, such as title tags and alternate attributes.

Subsequently, when the user inputs a search word, the result output retrieves the index of the matching page and outputs the result determined to be most relevant to the user. Here, the relevance may be determined by a number of factors including the page rank for a particular page.

In the following description, the wireless internet search service performed by the mobile communication terminal will be described as an example. However, the apparatus and method for acquiring a web page title according to the present invention includes a notebook, a desktop, a PDA, and the like. Obviously, any electronic device capable of servicing can be applied without any limitation.

The apparatus for removing duplicate web pages according to an embodiment of the present invention may be included in a web page search server that provides a web page search service, and may be implemented as a device connected to a web page search server. In addition, it is apparent that the apparatus and method for obtaining a webpage title according to an embodiment of the present invention may be implemented by software that can be read by a webpage search server.

Before describing the embodiments of the present invention in detail, a brief description will be given of a method for removing duplicate web pages in order to clarify the gist of the present invention.

For example, suppose you have two web pages, A and B, where the contents of A and B match, and the number of links to other web pages in the body of A and B is the same as the absolute address of each link. In this case, A and B may be determined as duplicate web pages. Thus, either A or B can be removed from the search results for searching the web page.

In addition, after changing all relative addresses included in the body content of two web pages A and B into absolute addresses, A and B may be compared, and in the same case, A and B may be determined to be duplicate web pages. Here, the relative address is an address of a link included in the main content of the web page.

Such a method can prevent duplicate search results by comparing each other web page accumulated in the meantime before the new web page is included in the search target and discarding the new web page without putting it in the search target.

However, this method is practically difficult due to the limitation of performance for the computer to compare the contents of all web pages when the number of accumulated web pages is large.

Thus, the determination of the duplicate web page may be performed by calculating a hash value of the web page using a hash function and comparing the hash values.

1 is a block diagram of an apparatus for removing duplicate web pages according to an embodiment of the present invention.

Referring to FIG. 1, the apparatus for removing duplicate web pages 100 may include a web page collecting unit 110, an address converting unit 120, a hash value calculating unit 130, a web page storing unit 140, and a hash value list storage. The unit 150 and the controller 160 may be included.

Here, it will be apparent to those skilled in the art in view of the technical spirit of the present invention that the apparatus for removing duplicate web pages 100 may be implemented as a program module or a device in a web page search server that provides a web page search service. When the webpage search server receives a wireless webpage search request and a search word through the mobile communication terminal, the webpage search server may find an index suitable for the input search word and transmit the search result to the mobile communication terminal. Here, the index may include the title of the web page and the content of the web page.

In addition, the webpage search server determines that both the new webpage and the updated webpage are new webpages and collects new webpages when the crawl is performed.

In this case, the webpage search server may find a webpage according to a preset webpage to be collected, the number of times, the number of pages, and the like.

In addition, when the webpage search server collects new and updated new webpages when the webpage search server crawls, the duplicated webpage removing apparatus 100 removes the duplicated webpages from the new webpage. Here, the duplicate web page is a web page in which the contents of the new web page and the contents of the previously stored web page are the same.

Hereinafter, the configuration of the duplicate web page removal apparatus 100 will be described.

The webpage input unit 110 receives information of a webpage collected when a crawl is performed by a webpage search server. In this case, the crawling process is a process of updating a previously generated index by collecting a new web page. The crawling may be performed by tracking a link of the web page to obtain information about the web pages.

Here, the web page to be crawled is a web page for wireless Internet, and may be a general web page.

Here, the web page input unit 110 has been described as a component of the redundant web page removing apparatus 100, but this is merely an example. It is apparent to those skilled in the art in view of the technical concept of the present invention that the apparatus for removing a duplicate webpage 100 may be implemented without a separate input unit in a webpage search server.

The address conversion unit 120 converts a relative address included in the content of the web page input by the web page input unit 110 into an absolute address. Here, converting a relative address into an absolute address is intended to unify the address because the content of the web page can be determined to be different due to the difference in the link address included in the content of the web page even though the content of the web page is the same.

Here, the absolute address may be generated by adding a relative address to the base address.

The absolute address is a complete address including the server protocol, which is most often "http: //" for web pages.

For example, "http://www.macromedia.com/en/support/" is the absolute address of the Macromedia Fireworks technical support web page. Absolute addresses remain accurate regardless of the location of the source document, but do not link properly as the target document moves.

Relative addresses are also relative to the directory containing the source document.

In addition, the reference address is the actual address of the web page currently input.

That is, the relative address is an address of a subdirectory based on the address of the current webpage, and the address of the current webpage is omitted and only the addresses included in the lower part are omitted.

In other words, a web page contains an address and body content. Here, the main content may include a link to a reference web page referring to another web page. Here, the link includes address information of the reference web page. The address information may be an absolute address, but when the reference web page is a web page located in a subdirectory of the current web page, the address information may be a relative address based on the address of the current web page. Here, the relative address is an address indicating the location of a subdirectory, omitting the address of the current web page. Therefore, the relative address may be used as the address information of the link included in the content of the web page only when the web page of the subdirectory based on the current web page is linked.

For example, suppose the absolute address is "http://www.macromedia.com/us/support/file.htm". Here, "http://www.macromedia.com/kr/support/" may be a reference address and "file.htm" may be a relative address.

That is, "http://www.macromedia.com/en/support/" is the address of the current webpage displayed in the address bar of the webpage, and "file.htm" is the address information of the link included in the body content of the webpage. Becomes

For example, suppose a link included in a web page refers to a document called "http://krhost.net/url/url_html/html/02". At this time, if you want to refer to the document "http://krhost.net/url/url_html/html/01", the relative address corresponding to this document is "01". The reference address is "http://krhost.net/url/url_html/html/". Therefore, the absolute address may be "http://krhost.net/url/url_html/html/01" by adding the relative address "01".

The hash value calculator 130 calculates a hash value of a web page converted into an absolute address by the address converter 120 using a hash function.

Here, calculating a hash value converts a string into a shorter value or key that symbolizes the original. Hash values are used to index and search items in the database because finding them using a short hash key is faster than finding them using the original values. Here, the string used for calculating the hash value may use any part or all of all contents in the web page.

In addition, the hash function is an algorithm for calculating a hash value.

In addition to fast data retrieval, hash values are used to encrypt and decrypt digital signatures.

For example, the digital signature is converted using a hash function, and then the hash value and the digital signature are transmitted separately. The receiver uses the same hash function used by the sender to extract the hash value from the signature and compare it with the hash value already received. The digital signature is valid only if the comparison is the same.

In addition, commonly used hash functions include message digest 5 (MD5) and secure hash algorithm 1 (SHA). Such a hash function can convert a document composed of a string into a short string representing the content of the document, and the hash value is different if the content of the document is slightly different. Thus, comparing documents with a hash value calculated with a hash function has an advantage of comparing documents more efficiently than storing and comparing the entire document.

For example, hash functions are used to compare human fingerprints. That is, fingerprint comparison is performed by storing and comparing hash values of fingerprint data to which a hash function is applied instead of storing and comparing a fingerprint original.

The webpage storage unit 140 stores webpages that do not overlap with the contents of the existing webpages among the webpages input by the webpage input unit 110.

The hash value list storage unit 150 stores the hash value calculated by the hash value calculation unit 130.

The controller 160 reads the contents of all the web pages input by the web page input unit 110, and converts the relative addresses included in the web pages into absolute addresses through the address conversion unit 120.

In addition, the controller 160 receives the hash value of the web page including the converted address from the hash value calculator 130 and compares the hash values stored in the hash value list storage unit 150. When the hash value is present in the hash value list storage unit 150, the controller 160 eliminates duplicate web pages by not storing the corresponding web page in the web page storage unit 140.

When the hash value list storage unit 150 does not have a hash value, the controller 160 stores the hash value in the hash value list storage unit 150 and stores the web page in the web page storage unit. Save to 140.

Here, the web pages stored in the web page storage unit 150 are web pages deduplicated, and the web page search server may index the web pages using the deduplicated web pages when crawling.

2 is a flowchart illustrating a method of removing duplicate web pages according to an embodiment of the present invention.

Hereinafter, in order to facilitate the understanding and explanation of the present invention, a description will be given mainly of the apparatus for removing duplicate web pages.

Referring to FIG. 2, first, an apparatus 100 for removing a duplicate webpage receives a webpage collected when a crawl is performed (S210).

In this case, the crawling process is a process of updating a previously generated index by collecting a new web page. The crawling may be performed by tracking a link of the web page to obtain information about the web pages.

Subsequently, the apparatus for removing duplicate web pages 100 converts an absolute address from a relative address included in the content of the input web page (S220). Here, the relative address included in the content of the web page is a linked address referring to another web page, and the current address of the web page is used as a reference address.

Also, an absolute address is generated by adding a relative address to a base address.

Since the description of the absolute address, the relative address and the reference address has been described above, the description thereof will be omitted.

Subsequently, the duplicate webpage removing apparatus 100 calculates a hash value of the webpage including the converted address (S230). Here, the hash value is calculated using a hash function, and can be calculated using all contents of the web page as variables.

Since the description of the hash value and the hash function has been described above, the description thereof will be omitted.

Subsequently, the duplicate webpage removing apparatus 100 determines whether the calculated hash value exists in the hash value list stored in the duplicate webpage removing apparatus (S240), and if the hash value exists in the hash value list, as a result of the determination. Do not save the web page (S250).

If the hash value does not exist in the hash value list, the hash value is added to the hash value list (S260), and the web page is stored (S270).

3 is a flowchart illustrating a method of removing a duplicate webpage according to another exemplary embodiment of the present invention. More specifically, FIG. 3 is a method for removing duplicate web pages that does not require a process of converting a relative address included in a link to an absolute address among links included in the content of the web page.

Referring to FIG. 3, the apparatus 100 for removing a duplicate webpage receives a webpage collected when performing a crawl (S310).

Subsequently, the apparatus for removing duplicate web pages 100 extracts a reference address from the received web page (S320). Here, the reference address is the address of the web page currently input.

Subsequently, the apparatus for removing duplicate web pages 100 merges the extracted reference addresses with the contents of the web pages (S330).

For example, the base address is "http: // abc / x / y /" and the content of the web page is "<html> <body> this is test <a href=z.html> </ body> </ html > "Then the merged result is http: // abc / x / y / <html> <body> this is test <a href=z.html> </ body> </ html>".

Subsequently, the apparatus 100 for removing duplicate web pages calculates a hash value using a hash function of the merged result (S340).

Subsequently, the duplicate webpage removing apparatus 100 determines whether the calculated hash value exists in the hash value list stored in the duplicate webpage removing apparatus (S350), and when the hash value exists in the hash value list, as a result of the determination. Do not save the web page (S360).

If the hash value does not exist in the hash value list as a result of the determination, the hash value is added to the hash value list (S370), and the corresponding web page is stored (S380).

The above-described method of removing duplicate web pages can be removed only when duplicate web pages are stored in the same directory.

Duplicate web page removal method according to an embodiment of the present invention is implemented in the form of program instructions that can be executed by various computer means may be recorded on a computer readable medium. Computer-readable media may include, alone or in combination with the program instructions, data files, data structures, and the like.

The program instructions recorded on the computer readable medium may be those specially designed and constructed for the present invention, or may be known and available to those skilled in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Hardware devices specially configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory and the like. In addition, the above-described medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although the above has been described with reference to a preferred embodiment of the present invention, those skilled in the art to which the present invention pertains without departing from the spirit and scope of the invention as set forth in the claims below It will be appreciated that modifications and variations can be made.

1 is a block diagram of an apparatus for removing duplicate web pages according to an embodiment of the present invention.

2 is a flowchart illustrating a method of removing duplicate web pages according to an embodiment of the present invention.

3 is a flowchart illustrating a method of removing a duplicate webpage according to another exemplary embodiment of the present invention.

<Description of the symbols for the main parts of the drawings>

110: web page input unit 120: address conversion unit

130: hash value calculator 140: web page storage unit

150: hash value list storage unit 160: control unit

Claims (10)

In the apparatus for removing duplicate web pages that removes duplicate web pages when a web page search server that provides a web page search service crawls, A web page input unit for receiving a web page collected when the crawl is performed; An address conversion unit for converting a relative address included in the content of the input web page into an absolute address; A hash value calculator configured to calculate a hash value of the absolute address web page converted to the absolute address; Web page storage; A hash value list storage unit; And It is determined whether the hash value calculated by the hash value calculator is present in the hash value list of the hash value list storage unit. If the hash value does not exist, the hash value is added to the hash value list, and And a control unit for storing the collected web page in the web page storage unit. The method of claim 1, The absolute address is generated by adding the relative address to the reference address, And the reference address is an address of an input web page, and the relative address is an address included in a link among contents of the web page. The method of claim 1, The hash value is calculated using a hash function, The hash function is a duplicate web page removal apparatus, characterized in that any one of MD5 (message digest 5), SHA1 (secure hash algorithm 1). The method of claim 1, Device for removing a duplicate web page, characterized in that the web page is a web page for a wired or wireless Internet. In the method of removing duplicate web pages when the web page search server that provides the web page search service crawls the duplicate web page removal device, the duplicate web page is removed. Receiving a web page found when the crawl is performed; Converting a relative address included in content of the web page into an absolute address; Calculating a hash value of the web page including the absolute address; And Determining whether the hash value exists in the hash value list, and if not present, not storing the web page, and if not, adding the hash value to the hash value list and storing the web page. How to remove duplicate webpages that you include. In the method of removing duplicate web pages when the web page search server that provides the web page search service crawls the duplicate web page removal device, the duplicate web page is removed. Receiving a web page found when the crawl is performed; Extracting the address of the received web page; Merging the contents of the web page with the address; Calculating a hash value of the merged result; And Determining whether the hash value exists in the hash value list, and if not present, not storing the web page, and if not, adding the hash value to the hash value list and storing the web page. How to remove duplicate webpages that you include. The method according to claim 5, wherein The hash value is calculated using a hash function, The hash function is a method of removing a duplicate web page, characterized in that any one of the message digest 5 (MD5), secure hash algorithm 1 (SHA1). The method according to claim 5, wherein The web page is a duplicate web page removal method, characterized in that the web page for a wired or wireless Internet. Claims [1] A recording medium having recorded thereon a program for performing a method for removing duplicate web pages, which is readable by an electronic device performing the method for removing duplicate web pages. Receiving a web page found when the crawl is performed; Converting a relative address included in content of the web page into an absolute address; Calculating a hash value of the web page including the absolute address; Determining whether the hash value exists in the hash value list, and if not present, not storing the web page, and if not, adding the hash value to the hash value list and storing the web page. A recording medium readable by an electronic device in which a program for performing a method of removing duplicate web pages is included. Claims [1] A recording medium having recorded thereon a program for performing a method for removing duplicate web pages, which is readable by an electronic device performing the method for removing duplicate web pages. Receiving a web page found when the crawl is performed; Extracting the address of the received web page; Merging the contents of the web page with the address; Calculating a hash value of the merged result; And Determining whether the hash value exists in the hash value list, and if not present, not storing the web page, and if not, adding the hash value to the hash value list and storing the web page. A recording medium readable by an electronic device in which a program for performing a method of removing duplicate web pages is included.

Images