JP2010231761A - Multi-language object hierarchy extraction method and system from multi-language web site - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and a system for extracting a multi-language object hierarchy from a multi-language web site. <P>SOLUTION: This method includes steps for: inputting a web page into the multi-language web site; decomposing the web site into a lower-rank web site assembly classified by language so that web pages of each lower-rank web site has the same language; extracting a single language object hierarchy from each lower-rank web site, and recording a correspondence relation between each object on the hierarchy and a web page corresponding thereto; determining a parallel relation between different language web pages in different lower-rank web sites; and generating the multi-language object hierarchy of the multi-language web site following the extracted single language object hierarchy of each lower-rank web site, the correspondence relation between the recorded object and the web page corresponding thereto, and the determined parallel relation between the different language web pages. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates generally to information extraction, and more particularly to web mining and multilingual object hierarchy extraction methods and systems from multilingual web sites.

  Now that we have entered the Internet era, a great deal of information is accumulated on the Web. Computers are now an indispensable tool for finding the information you want to know in the lives of modern people. Computers can perform information processing such as calculation, storage, and retrieval at extremely high speeds, but lack the ability to understand information, which is an obstacle to intelligent information processing. In recent years, research on semantic processing for intelligent information processing has become active as a measure for dealing with this problem. Such techniques are described, for example, in T.W. Non-Patent Document 1 (Berners-Lee, J. Hendler, O. Lassila "The Semantic Web", Scientific American, May 2001, pp. 28-37), Non-Patent Document 2 (Nigel Shadbolt, Tim Berners-Lee, and Wendy Hall “The Semantic Web Revisited” (revisited the Semantic Web), IEEE Intelligent Systems 21 (3) pp. 96-101, May / June 2006, Non-Patent Document 3 (E. Hyvonen (editor) “Semantic Web Kick-Off in Finland-Vision, Technologies, Research. and Applications "(Start of the Semantic Web in Finland-Vision, Technology, Research, Applications), HIIT Publications, 2002-001, Helsinki Institute for Information Technology (HIIT), Finland, Helsinki, pp. 304), etc. . The entire disclosures of these are hereby incorporated by reference for all purposes. The above documents focus on formats and techniques for supporting computer understanding of information. Standardization mechanisms such as the World Wide Web Consortium (W3C) are based on artificial intelligence (AI) and widely used web information processing technologies that have been widely used to promote the adoption of semantic technologies. Based on mathematical logic for knowledge representation (eg, description logic, frame logic), XML, Resource Description Framework (RDF), Web Information Language (OWL), rule language (eg, Web Rule Language, Rule Markup Language) It actively defines various standards represented by. Many developers, entrepreneurs, and practitioners are already in the process of creating and deploying tool sets, products, and case studies for the realization of a semantic-based intelligent information utilization concept. There are even things that have been put to practical use. However, domain knowledge (currently ontology is the mainstream of knowledge representation on the web) in adopting the computer's powerful computing power and semantic standards in providing various intelligent information usage services to web users. Plays a major role internally. For this reason, the construction of domain knowledge has become an important issue that must be resolved. At the same time, with the spread of the Web, the entire world has become more interconnected. In order to establish a seamless and smooth communication channel between people who use different languages, include various language versions in the constructed domain knowledge and set the exact correspondence between these multilingual versions Is required. Then, in the domain knowledge construction problem, a new difficulty arises as to how to establish multilingual domain knowledge with correspondence between multiple languages.

  An ontology is used as a means for formalizing a document expressing domain knowledge. Ontology defines the concepts / objects in the domain and the relationships between these concepts / objects. The relationships between concepts / objects defined in the ontology include a wide variety of relationships such as “belonging to” and “located in”. The most common conceptual relationship in practical applications is hierarchical, such as “A belongs to B” (A being to B) or “A is a subordinate concept of B” (A is a sub-concept of B). It is a relationship. For example, the concept of “personal computer” is a subordinate concept of the concept of “computer”. A lightweight ontology that defines only hierarchical relationships is called a “hierarchy” and is typically embodied as a classification system or directory structure in practical applications.

  Several techniques related to the hierarchy extraction method have already been proposed in existing papers and patents. However, most of them only extract monolingual hierarchies from monolingual data sources, and very few techniques address the multilingual hierarchy extraction problem. “Multilingual hierarchy” means a hierarchy in which concepts / objects are defined or described in a plurality of languages. In the following, we first introduce existing papers or patents related to the multilingual hierarchy.

  Non-Patent Document 4 (2008 ICICIC Meeting Minutes, H.-C. Yang, D.-W. Chen, C.-H Lee's “A multi-level hierarchical mapping method on GHSOM” (Multilingual Hierarchical Mapping Based on GSOM) Method)) (hereinafter referred to as “reference document 1”) is a multilingual hierarchy construction method as described below. That is, first collect parallel multilingual documents ("Parallel multilingual" means that there are many different language versions in one document), and then manually mark the parallel relationship of these documents (I.e., marking documents that are separate languages but are actually different language versions of the same document). Then, a hierarchy is extracted for each single language subset of the document set. Finally, a correspondence between these single language hierarchies is established according to the parallel relationship of the pre-marked documents.

  Non-Patent Document 5 (1999 EMNLP / VLC Meeting Minutes, J. Daud, L. Padro, G. Rigau “Mapping Multiple Hierarchies Using Relaxation Labeling” (mapping of multilingual hierarchy using relaxation labeling method)) (Referred to as “reference document 2”) proposes a method of establishing correspondence between extracted single language hierarchies. In this method, a correspondence relationship between concept names / object names in different languages is determined by a language analysis technique using an external multilingual dictionary, and a correspondence relationship between hierarchies is established based thereon.

  Further, in Patent Document 1 (European Patent No. EP88877B1) “Multilingual Termination Extraction System” (Multilingual Term Extraction System) (hereinafter referred to as “Reference 3”), there is a method for extracting multilingual conceptual terms from a multilingual document. In this method, a certain language version of a term is used as input and the document is considered a word network, and other terms of the term are analyzed by analyzing the similarity between the word networks of a multilingual document. It is possible to derive a language version.

  Patent Document 6 (P. Resnik, NA's “The Web as a parallel corpus” (Web as a parallel corpus) announced at Computational Linguistic in March 2003) (hereinafter referred to as “Reference Document 4”) Presents a method for automatically establishing parallel relationships between multilingual documents, although not related to hierarchy or knowledge extraction. In this method, web pages on the web are treated as a set of documents, and the parallel relationship between different language web pages is identified using the similarity of the HTML structure of the web pages.

European Patent No. EP88877B1

Berners-Lee, J.M. Hender, O.D. Lassila "The Semantic Web", Scientific American, May 2001, pp. 199-001. 28-37 Nigel Shadbolt, Tim Berners-Lee, and Wendy Hall “The Semantic Web Revisited” (IEEE Semantic Web Revisited), IEEE Intelligent Systems 21 (3) pp. 199 96-101, May / June 2006 E. Hyvonen (edit) "Semantic Web Kick-Off in Finland-Vision, Technologies, Research, and Applications" (Start of the Semantic Web in Finland-Vision, Technology, Research, Applications, HIIT Publications, int. Technology (HIIT), Helsinki, Finland, pp. 304 Minutes of the 2008 ICICIC meeting, H.C. -C. Yang, D.C. -W. Chen, C.I. -HLee's “A multilingual hierarchy mapping method based on GHSOM” (multilingual hierarchical mapping method based on GHSOM) Minutes of 1999 EMNLP / VLC meeting, J. Daud, L.M. Padro, G .; Rigau's “Mapping Multilayer Hierarchies Using Relaxation Labeling” (Mapping of multilingual hierarchies using relaxation labeling method) In March 2003, P.P. announced at Computational Linguistic. Resnik, N.M. A. "The Web as a parallel corpus" (Web as a parallel corpus)

  Of the related solutions described above, Reference 1 requires manual construction of document parallel relationships. The disadvantages of this method are that it is less efficient, takes a lot of time and effort, and is less scalable. Therefore, it cannot be adopted when establishing a large-scale multilingual hierarchy. In the method of Reference 2, the extraction of the hierarchy and the mapping between a plurality of language versions are separated as two independent processes. As a result, since the context of the hierarchical source is lost during mapping, it is difficult to achieve high accuracy when using only an external multilingual dictionary. In the case of the method of Reference 3, in order to adopt this, it is essential to determine in advance a multilingual document for extracting concept terms as a multilingual version of the same document. At least one term needs to be given as input or drive. Therefore, this method cannot be applied in a situation where the presence or absence of the parallel relationship in the multilingual document is not specified, and cannot be used for the purpose of extracting new conceptual terms. Furthermore, since this method does not show the extraction of relationships between concepts, it cannot be used to construct a hierarchy. The method of Reference 4 can be used for the purpose of determining whether a parallel relationship exists between multilingual documents when extracting a hierarchy. However, only the correspondence between documents can be determined by this method, and it is impossible to determine the correspondence between internal elements of a document. This method cannot be applied directly to the extraction and mapping of multilingual hierarchies because conceptual objects in the hierarchy are very likely to correspond to only a portion of the document, not the entire document.

  To summarize the above, existing methods for multilingual hierarchy extraction have a number of drawbacks, which are impossible to fully automate multilingual hierarchy extraction and correspondence, and efficiency and extension It is summarized in two points that the performance is insufficient. In particular, these methods cannot respond quickly when a new knowledge domain or a new language needs to be dealt with, and require a large amount of preparation work such as document labeling and dictionary construction.

  The object of the present invention is to solve the above-mentioned problems of the multilingual hierarchy extraction method according to the related art. In the present invention, a method and system for automatically extracting a multilingual object hierarchy from a multilingual web site is provided. The method of the present invention extracts a monolingual hierarchy from individual monolingual subweb sites in a multilingual web site and automatically identifies internal parallel correspondences between these monolingual subweb sites The multilingual hierarchy of the multilingual web site is generated by deriving the correspondence between these single language hierarchies.

  According to a first aspect of the present invention, there is provided a method for extracting a multilingual object hierarchy from a multilingual web site, the step of inputting a web page to the multilingual web site, and the web of each subordinate web site・ Decomposing the web site into a set of subordinate web sites by language so that the pages are in the same language; extracting a single language object hierarchy from each subordinate web site; Recording a mapping relationship between the corresponding web pages, determining a parallel relationship between different language web pages in different sub-web sites, and simply extracting each sub-web site extracted. One language object hierarchy, the mapping relationship between the recorded object and the corresponding web page, and the determined tongue Accordance with the parallel relationship between web pages, multilingual object hierarchy extraction method characterized by comprising the steps of: generating a multi-language object hierarchy multilingual web site is provided.

  According to the second aspect of the present invention, there is provided a multilingual object hierarchy extraction system from a multilingual web site, an input means for inputting a web page to the multilingual web site, A single language sub-web site decomposition means for decomposing the web site into a set of sub-web sites by language so that the web page of the site is in the same language, and a single language from each sub-web site A monolingual object hierarchy extraction means for extracting an object hierarchy and recording a mapping relationship between each object on the hierarchy and the corresponding web page; Parallel relationship determination means for determining the parallel relationship between pages and monolingual object hierarchies of each extracted lower web site A multilingual object for generating a multilingual object hierarchy of a multilingual web site according to the mapping relationship between the recorded object and the corresponding web page and the determined parallel relationship between different language web pages There is provided a multilingual object hierarchy extraction system characterized by comprising hierarchy generation means.

  The multilingual hierarchy extraction method provided by the present invention is a fully automatic method that does not require manual labeling of documents, and the operating parameters are independent of both domain and language. According to the present invention, extraction efficiency and expandability are greatly improved as compared with existing methods. In addition, the method and system of the present invention utilizes the unique multilingual parallel relationships within the multilingual web site, thus ensuring high accuracy results.

  The above advantages and features of the present invention will become apparent upon reference to the following detailed description and drawings. However, it should be noted that the present invention is not limited to the examples shown in the drawings or specific embodiments.

  The invention will be more clearly understood from the following detailed description of embodiments of the invention and the accompanying drawings. In the accompanying drawings, similar parts are denoted by the same reference numerals.

1 is a block diagram illustrating a multilingual object hierarchy extraction system 100 according to the present invention. It is a flowchart for demonstrating the operation | movement process of the system 100 shown in FIG. It is a block diagram which shows the detail of the parallel relationship determination means and parallel relationship complementing means of the system 100 shown in FIG. It is the schematic for demonstrating the multilingual object hierarchy generation process by this invention.

  FIG. 1 is a block diagram illustrating a multilingual object hierarchy extraction system 100 according to the present invention. In FIG. 1, a system 100 is shown as a system including a multilingual object hierarchy extraction unit and a storage unit. The multilingual object hierarchy extraction process according to the present invention is implemented using a multilingual object hierarchy extraction unit as a processing unit. The multilingual object hierarchy extraction unit extracts an object hierarchy from a multilingual web site. In this multilingual web site, the object names included in the object hierarchy and the related documents (web pages) of the objects may have multiple versions of different languages. As shown in FIG. 1, the multilingual object hierarchy extraction unit includes an input unit 101, a page block set generation unit 102 (arbitrary), a single language subordinate web site decomposition unit 103, a single language object hierarchy extraction unit 104, a parallel A relationship determination unit 105, a parallel relationship complementing unit 106 (optional), and a multilingual object hierarchy generation unit 107 are provided. The storage unit is used for storing various processing results in combination with the processing unit. As shown in the figure, the storage unit includes a multilingual web site page storage device 108, a single language subordinate web site storage device 109, a single language object hierarchy storage device 110, and a multilingual subordinate web site parallel relationship storage device. 111, and a multilingual object hierarchy storage 112 may be included.

  FIG. 2 is a flowchart for explaining an operation process of the system 100 shown in FIG. Hereinafter, the principle and operation processing of the present invention will be described in detail with reference to FIG. 1 and FIG.

  As shown in FIG. 2, process 200 begins at step 201 where input means 101 inputs all web pages contained in a multilingual web site. The multilingual web site page storage device 108 stores all web pages of one or more multilingual web sites automatically collected by crawling from the Internet, and includes web page ID, web page content, web page Record various contents such as page links. In step 202, the page block set generation means 102 performs pre-processing for each input web page, and generates a page block set for each web page. A page block is a part of a web page and stores information about its visual size and position within the page. In addition, layout relationships between page blocks such as nesting and adjacency are also extracted. In order to further improve the accuracy of the results, it is desirable to use the similarity between the page block sets of different web pages as reference information in the multilingual object hierarchy extraction process.

  Next, in step 203, the monolingual sub-website decomposition means 103 can decompose the web pages in the input multilingual web site into a single-language subordinate web site set by language. . This is done by adding a language label for distinguishing each web page in the web site according to the language. Each monolingual subweb site can later be stored in monolingual subweb site storage 109. As shown in FIG. 1, the monolingual subordinate web site storage device 109 has various web page IDs, web page contents, web page links, etc. stored in the multilingual web site page storage device 108. In addition to web page content, each web page is labeled with a language ID based on the language of the page. In the following step 204, the monolingual object hierarchy extracting means 104 extracts the monolingual object hierarchy of each subordinate web site, and between each object in the hierarchy and the corresponding web page (or page block). Record the correspondence. The processing result of the single language object hierarchy extraction means 104 can be stored in the single object hierarchy storage device 110. In step 205, the parallel relationship determination means 105 determines the parallel relationship between different lower web sites. This relationship can be a parallel relationship between different language web pages / page blocks in different sub-web sites. The method for determining the parallel relationship between different language subordinate web sites includes a method using a Web directory structure, a method using a DOM structure of a different language web page, and a method using a structure topology of a page block set. There are many things. A method for determining the parallel relationship will be described in detail later.

  After the parallel relationship determination unit 105 determines the parallel relationship between different lower-level web sites, the parallel relationship complementing unit 106 determines the hyperlink relationship between the extracted web page or monolingual hierarchy in step 206 (optional). By analyzing, it is also possible to further complement the determined parallel relationship. A method for complementing the parallel relationship will be described in detail later. The parallel relationship between different language web pages or page blocks determined by the parallel relationship determining unit 105 and the parallel relationship complementing unit 106 can be stored in the multilingual sub-web site parallel relationship storage device 111.

  In step 207, the multilingual object hierarchy generation means 107 outputs the extracted single language object hierarchy (stored in the single language object hierarchy storage device 110) of each subordinate web site and the web page corresponding to the object (or Based on the recorded correspondence relationship between the page block) and the determined parallel relationship between different language web pages or page blocks (stored in the multilingual sub-web site parallel relationship storage device 111). Create a multilingual object hierarchy for language web sites. Each object on the multilingual object hierarchy may include a different language version. The generated multilingual object hierarchy is stored in the multilingual object hierarchy storage device 112. The process 200 ends here.

  Below, with reference to FIG. 3, the example of the determination process of a parallel relationship and a complement process is demonstrated. The structure and method shown in FIG. 3 are for illustrative purposes only and should not be considered as limiting the scope of the present invention.

  First, the internal structure of the parallel relationship determination unit 105 will be examined. FIG. 3 shows a configuration including a directory structure analysis unit 301, a DOM structure analysis unit 302, a page block set analysis unit 303, and a first matching unit 304 as an example of the parallel relationship determination unit 105. . The directory structure analysis unit 301, the DOM structure analysis unit 302, and the page block set analysis unit 303 respectively perform different language webs by performing web directory structure analysis, web page DOM structure analysis, and page block set structure analysis of a web site. Used to implement a process that determines the parallel relationship between pages (page blocks). The web directory structure analysis, the web page DOM structure analysis, and the page block set structure analysis can be used separately to determine the parallel relationship. These three are not necessarily used at the same time as shown in FIG. The combination of the three analysis methods in FIG. 3 is merely a consideration for improving the accuracy of the results, and should not be regarded as limiting the scope of the present invention. The first matching unit 304 is used to match the three parallel relationship results determined by the directory structure analysis unit 301, the DOM structure analysis unit 302, and the page block set analysis unit 303, respectively, and to resolve conflicts between the results. Is done. For example, the first matching unit 304 assigns a weight to each analysis unit, and determines a final result according to the assigned weight. As a method for determining the weight to be assigned to the analysis unit, any machine learning method based on a known training sample can be used in the art.

  The directory structure analysis unit 301 is a component for analyzing a web directory structure of a web site and determining a parallel relationship. For example, in directory structure analysis, the URL of a web page can be referred to infer the web site author's consideration for the parallel relationship between multilingual web pages. Looking at the URL pattern, a lot of information useful for determining the multilingual parallel relationship can be obtained. For example, on the Symantec website: http: // www. symantec. com / norton and http: // www. symantec. com / zh / cn / norton and http: // www. symantec. com / ja / jp / norton constitute one parallel web page group. These pages are English, Chinese and Japanese versions of the same content. In the Symantec website, the URL pattern for identifying the parallel relationship is “http://www.symantec.com/(language)/(region)/(content)”. These URL patterns are obtained by performing a similarity analysis on the URLs of all web pages in a single language subordinate web site to obtain a URL template for each single language subordinate web site. It can be detected by comparing URL templates of web sites.

  In addition to the web directory structure, internal structure analysis of web pages can also be used in determining parallel relationships between web pages or page blocks. For example, the DOM structure analysis unit 302 determines the parallel relationship between the web pages by analyzing the similarity of the DOM structures of the web pages, and the page block set analysis unit 303 selects the page blocks in each web page. The parallel relationship between page blocks is determined by analyzing the similarity of sets. First, the DOM structure analysis unit 302 detects web pages having a parallel relationship by analyzing the similarity of the DOM structures of the web pages. As an index indicating the similarity of the DOM structure, there is a similarity of an HTML node label sequence, a similarity of a node pattern, and the like. As described above, the page block set generation unit 102 further generates a page block set for each web page. The page block set analysis unit 303 then determines whether there is a parallel relationship between these web pages by analyzing the similarity of the page block sets of the two web pages. As an index indicating the similarity of the page block set, the similarity of the topological structure of the page block set (only the relation of the abstract space is considered) and the similarity of the spatial size and position information of the page block are used. . By using the similarity of page block sets, not only the parallel relationship between web pages, but also the parallel relationship between page blocks can be determined simultaneously.

  Referring again to FIG. 3, the first, second, and third parallel relationship results respectively determined by the directory structure analysis unit 301, the DOM structure analysis unit 302, and the page block set analysis unit 303 are the different parallel results. Sent to the first matching unit 304 to match the relationship results. As described above, the first matching unit 304 can assign different weights to these three analysis results. The matched parallel relation result is directly sent to the multilingual object hierarchy generation means 107 as a final result to generate a multilingual object hierarchy, or the intermediate relation is sent to the parallel relation complementing means 106 to determine the determined parallel relation. It can also be supplemented. So-called complementing means complements the determined parallel relationship by hyperlink structure analysis, single-language hierarchical analysis of lower-level web sites, or the like, in order to avoid possible missed parallel relationships.

  The parallel relationship complementing means 106 shown as an example in FIG. 3 includes a hyperlink structure complementing unit 305, a single language object hierarchy complementing unit 306, and a second matching unit 307. The configuration of the parallel relationship complementing means shown in FIG. 3 is also shown as an example only, and thus should not be regarded as limiting the scope of the present invention. The monolingual object hierarchy completion unit 306 and the second matching unit 307 can be used separately or in combination. The second matching unit 307 matches different complementary results according to the weight determined in advance for each complementing method, and determines the final complemented parallel relationship. It is obvious that the weight that can be applied during the parallel relationship complementing process and the weight that can be applied during the parallel relationship determining process are independent of each other.

The hyperlink structure complementing unit 305 obtains a semantic topology relationship between web pages in the same monolingual sub-web site by analyzing a hyperlink relationship between web pages. Thereafter, by comparing the semantic topology relationships of different monolingual sub-web sites, the presence or absence of parallelism between the two web pages is determined. As a method for expressing the semantic topology of the web page, for example, a navigation path can be selected. Once the navigation paths for each single language sub-web site are generated, the similarity of the navigation paths of different single language sub-web sites is compared to determine the existence of web pages with parallel relationships be able to. The rule in this case is, for example, “in the web page p in the web site 1, out of all the navigation paths related to the page p, the web page that directs p along these paths is p ₁ , ... and p _m, c ₁ a web page p is directly along these paths, a ... c _n, and, in a web page p'web site 2, all related to the page p' Of the navigation paths, web pages that direct p ′ along these paths are designated as p ′ ₁ ,... P ′ _k, and web pages that p ′ direct along these paths are designated as c ′ ₁ ,. If c ′ _n and m = k and (p _i , p _i ′) (i = 1,..., m) are all parallel web page pairs, then two sets {c _{i} (i} 1, ..., n) and _{{c j '} (j =} 1, ..., in r), the threshold t (t total q of the number of parallel web pages pair is set in advance between the two sets (p, p ′) can also be determined to be a parallel web page pair if it exceeds (related to the minimum value of n and r) ”.

Further, the monolingual object hierarchy complementing unit 306 uses the monolingual hierarchy of each monolingual subordinate web site extracted by the monolingual object hierarchy extracting means 104 to use a web page or page block between Determine whether there is a parallel relationship. For example, for a web page or pages block p written in language 1, it corresponds to the object o on the hierarchy, the web page or pages block corresponding to the o of the upper objects p _1, located in the ... p _m , O is assumed to be c ₁ ,... C _n and the web page or page block p ′ written in language 2 is hierarchical. The web page or page block corresponding to the object o ′ and corresponding to the upper object of o ′ is p ₁ ′,... P _m ′, and the web page or page block corresponding to the lower object of o ′ is c _{1. ',} ... c n' assuming a, p and p'is a web page or pages block, located in the m = k _{(P i, p i ')} (i = 1, ..., m) are all parallel relation pairs, and two sets _{{c i} (i = 1} , ..., n) and _{{c j'} (j} = 1,..., R) if the total number q of parallel pairs between the two sets exceeds a preset threshold t (t is related to the minimum of n and r) , (P, p ′) can also be determined to be a parallel relationship pair.

  After the parallel relationship between the monolingual subordinate web sites is determined, the multilingual object hierarchy generation unit 107 includes the objects on the monolingual hierarchy and the web / stored in the monolingual object hierarchy storage device 110. A parallel relationship between multilingual hierarchies can be acquired by directly referring to a correspondence relationship with a page or a page block. Thereby, the final multilingual object hierarchy is acquired. FIG. 4 is an example of the above processing. As shown in FIG. 4, the parallel relationship between the Chinese and English subordinate web sites, the single language hierarchy corresponding to each of the Chinese and English subordinate web sites, and the web page corresponding to the object (page By referring to the correspondence between the blocks A and B, a final object hierarchy having a multilingual version (in this example, Chinese + English) is generated.

  So far, the multilingual object hierarchy extraction method and system according to the present invention has been described with reference to the accompanying drawings. With the method of the present invention, there is no need to manually label the document and the operating parameters are independent of the specific domain and language. Therefore, according to the present invention, extraction efficiency and expandability are greatly improved as compared with existing methods. In addition, the method and system of the present invention utilizes a multi-language parallel correspondence within a multi-language web site, thus ensuring high accuracy results.

  Although specific embodiments of the invention have been described above with reference to the accompanying drawings, the invention is not limited to the specific configurations and processes shown in the drawings. In the above description, details of known methods and techniques are omitted for the sake of brevity. Also, in the above examples, some specific steps have been illustrated, but the method and process of the present invention are not limited to the specific steps used in the description and illustration, so those skilled in the art are familiar. For example, once the spirit of the present invention is understood, various modifications, changes and additions can be made, and the order of the steps can be changed.

  Each element of the invention may be implemented as hardware, software, firmware, or a combination thereof and utilized within the system, subsystem, component, or subcomponent. When implemented as software, each element of the present invention is a program or code section for performing necessary tasks. These programs or code sections can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via a data signal carried on a carrier wave. "Machine readable medium" includes any medium that can store or transmit information. Examples of the machine-readable medium include an electronic circuit, a semiconductor storage device, a ROM, a flash memory, an EROM, a floppy disk, a CD-ROM, an optical disk, a hard disk, an optical fiber medium, and an RF link. The code section can be downloaded via a computer network such as the Internet or an intranet.

  The present invention can be implemented in various other forms without departing from the spirit and essential characteristics thereof. For example, the algorithm described in the embodiments can be modified as long as the system architecture does not depart from the basic spirit of the present invention. Accordingly, the above embodiments are considered in all respects to be illustrative and not restrictive. Since the scope of the present invention is defined by the appended claims rather than the foregoing description, any variation or equivalent that falls within the scope of the claims is included in the scope of the invention.

101: Input means 102: Page block set generation means 103: Single language lower-level web site decomposition means 104: Single language object hierarchy extraction means 105: Parallel relation determination means 106: Parallel relation complement means 107: Multilingual object hierarchy generation Means 108: Multilingual web site page storage device 109: Single language subordinate web site storage device 110: Single language object hierarchical storage device 111: Multilingual subordinate web site parallel relationship storage device 112: Multilingual object hierarchical storage device Apparatus 301: Directory structure analysis unit 302: DOM structure analysis unit 303: Page block set analysis unit 304: First matching unit 305: Hyperlink structure complementing unit 306: Monolingual object hierarchy complementing unit 307: Second matching Knit

Claims (21)

A multilingual object hierarchy extraction method from a multilingual website,
Entering a web page on a multilingual website,
Decomposing the web site into a set of subordinate web sites related by language so that the web pages of each subordinate web site are in the same language;
Extracting a monolingual object hierarchy from each subordinate web site and recording a mapping relationship between each object on the hierarchy and its corresponding web page;
Determining parallel relationships between different language web pages in different subordinate web sites;
A multilingual web according to the monolingual object hierarchy of each extracted sub-web site, the mapping relationship between the recorded object and the corresponding web page, and the parallel relationship between the determined foreign language web pages A method of extracting a multilingual object hierarchy of a site, comprising: generating a multilingual object hierarchy of a site.   The method of claim 1, further comprising generating a page block set for each web page. Recording the correspondence between each object and its corresponding page block;
Determining the parallel relationship between page blocks of different language web pages,
The parallel relationship between the page blocks of different language web pages determined to be the correspondence relationship between each recorded object and its corresponding page block is used as a reference in the process of generating the multilingual object hierarchy. The multilingual object hierarchy extraction method according to claim 2.   The multilingual object hierarchy extraction method according to claim 1, wherein the parallel relationship between the different language web pages is determined by a web directory structure of a web site.   The multilingual object hierarchy extraction method according to claim 1, wherein the parallel relationship between the different language web pages is determined by comparing similarities of DOM structures of the different language web pages.   The multilingual object hierarchy extraction method according to claim 1, wherein the parallel relationship between the different language web pages is determined by comparing similarities of page block sets of different language web pages. Determining a parallel relationship between the different language web pages;
Determining a first parallel relationship result based on the web directory structure of the web site;
Determining a second parallel relationship result based on the similarity of the DOM structure of different language web pages;
Determining a third parallel relationship result based on the similarity of the page block set of different language web pages;
The first, second, and third parallel relationship results are matched based on predetermined weights for three analysis methods to determine an appropriate parallel relationship between different language web pages. Item 3. The multilingual object hierarchy extraction method according to Item 2.   The method according to any one of claims 4 to 7, further comprising a step of complementing the determined parallel relationship between different language web pages by analyzing the hyperlink structure of the web page. Multilingual object hierarchy extraction method.   5. The method of claim 4, further comprising the step of supplementing the determined parallel relationship between different language web pages by analyzing a monolingual block hierarchy of each extracted sub-web site. The multilingual object hierarchy extraction method according to any one of claims 7 to 9. By analyzing the hyperlink structure between web pages, complement the determined parallel relationship between different language web pages and obtain the first completed parallel relationship result,
Analyzing the monolingual block hierarchy of each extracted sub-web site to complement the determined parallel relationship between different language web pages and obtain a second completed parallel relationship result;
The final parallel relationship between different language web pages is determined by matching the first and second complemented parallel relationship results based on predetermined weights related to two completion methods. The multilingual object hierarchy extraction method according to claim 4. A multilingual object hierarchy extraction system from a multilingual website,
An input means for entering a web page on a multilingual web site;
A monolingual sub-web site decomposition means for decomposing the web site into a set of related sub-web sites by language so that the web pages of each sub-web site are in the same language;
A monolingual object hierarchy extraction means for extracting a monolingual object hierarchy from each subordinate web site and recording a mapping relationship between each object on the hierarchy and the corresponding web page;
A parallel relationship determination means for determining a parallel relationship between different language web pages in different subordinate web sites;
A multilingual web according to the monolingual object hierarchy of each extracted sub-web site, the mapping relationship between the recorded object and the corresponding web page, and the parallel relationship between the determined foreign language web pages A multilingual object hierarchy extraction system comprising: a multilingual object hierarchy generation means for generating a multilingual object hierarchy of a site.   12. The multilingual object hierarchy extraction system according to claim 11, further comprising page block set generation means for generating a page block set of each web page. The monolingual object hierarchy extraction means records the correspondence between each object and its corresponding page block,
The web page parallel relationship determining means determines a parallel relationship between page blocks of different language web pages,
Multilingual to use the parallel relationship between the page blocks of different language web pages determined to be the correspondence between each recorded object and its corresponding page block in the process of generating the multilingual object hierarchy The multilingual object hierarchy extraction system according to claim 12, wherein the system is provided to an object hierarchy generation means.   12. The web page parallel relation determining unit includes a directory structure analyzing unit that determines a parallel relation between different language web pages by analyzing a web directory structure of a web site. Multilingual object hierarchy extraction system.   The web-page parallel relationship determination unit includes a DOM structure analysis unit that determines a parallel relationship between different language web pages by comparing similarities of DOM structures of different language web pages. Item 12. The multilingual object hierarchy extraction system according to Item 11.   The web-page parallel relationship determination means includes a page block set analysis unit that determines the parallel relationship between different language web pages by comparing the similarity of the page block sets of different language web pages. The multilingual object hierarchy extraction system according to claim 11. The web / page parallel relationship determining means includes:
A directory structure analysis unit that determines a first parallel relationship result by analyzing a web directory structure of a web site;
A DOM structure analysis unit that determines a second parallel relationship result by comparing the DOM structure similarity of different language web pages;
A page block set analysis unit that determines a third parallel relationship result by comparing the similarity of page block sets of different language web pages;
Based on predetermined weights related to the directory structure analysis unit, the DOM structure analysis unit, and the page block set analysis unit, the first, second, and third parallel relationship results are matched to each other between different language web pages. The multilingual object hierarchy extraction system according to claim 11, further comprising: a first matching unit that determines an appropriate parallel relationship of the first and second matching units.   The multilingual object hierarchy extraction system according to any one of claims 14 to 17, further comprising parallel relation complementing means for complementing the judged parallel relation between different language web pages.   The parallel relationship complementing unit includes a hyperlink structure complementing unit that complements the determined parallel relationship between different language web pages by analyzing a hyperlink structure between web pages. The multilingual object hierarchy extraction system according to 18.   The parallel relationship complementing means includes a single language object hierarchy complementing unit that complements the determined parallel relationship between different language web pages by analyzing the single language block hierarchy of each extracted lower web site. The multilingual object hierarchy extraction system according to claim 18, comprising: The parallel relationship complementing means includes:
A hyperlink structure complementing unit that complements the determined parallel relationship between different language web pages by analyzing a hyperlink structure between web pages and generates a first complemented parallel relationship result;
A single language that complements the determined parallel relationship between different language web pages by analyzing the extracted single language block hierarchy of each subordinate web site and generates a second completed parallel relationship result An object hierarchy completion unit;
Based on a predetermined weight for the hyperlink structure complementing unit and the monolingual object hierarchy complementing unit, the first and second complemented parallel relationship results are matched to obtain a final result between different language web pages. The multilingual object hierarchy extraction system according to claim 18, further comprising a second matching unit that determines a parallel relationship.

Images