JP2001306592A - Catalog-preparing method for web page retrieval engine operating on internet, and retrieval method therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a catalog preparing method and a retrieval method for Web page retrieval engine operating on Internet, by which a retrieval engine can be efficiently operated and the suitable retrieved result suited to retrieval conditions can be presented. SOLUTION: In each of Web pages collected form the Internet, a language ID to be made correspondent to the relevant Web page is determined on the basis of language specification information described in the HTTP header or HTML document of that Web page (100-400). When the language ID cannot be determined through this, the language ID is determined on the basis of the top level domain name of URL storing that Web page (500-600). The catalog of respective Web pages prepared by a catalog preparation pogrom is manage in the data base of the retrieval engine by language ID determined for each Web page.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a catalog creation method and a search method in a Web page search engine.
The present invention relates to a technology for realizing a search engine that enables efficient operation of a search engine and presents qualified search results that meet search conditions.

[0002]

2. Description of the Related Art A Web page search engine provided as a function of a predetermined Web site on the Internet is an essential tool for efficiently searching for a target Web page from a huge number of Web pages existing on the Internet. Has become. FIG. 1 shows a conceptual configuration of a typical search engine. As shown in this figure, a search engine stores a list of a large number of web pages presented to a user as search results, and a search condition (hereinafter referred to as a query) received from a user's computer. It comprises a Web server for searching for and sending the searched list to a user computer, and software for operating a search engine.

[0003] The inventory stored in the database is
It is registered by a catalog creation program which is one of the software groups. The catalog creation program collects web pages from websites on the Internet using a so-called robot-type automatic web page collection system, and stores abstracts, summaries, and web pages based on the collected web pages. URL (Uniform Resourc
e Locator) is created and registered in the database.

The basic operation of a search engine is generally as follows. When a request is made from a user computer via the Internet, the search engine first sends a query setting page having input fields for search conditions such as keywords (hereinafter referred to as a query) to the computer. The user computer displays the sent query setting page on a Web browser to prompt the user to input a query. When the user performs a predetermined operation, the user computer sends the input query to a search engine. When receiving the query sent from the user computer, the search engine searches the database corresponding to the query from the database. Then, a home page describing the searched list is created and sent to the user computer.

[0005]

By the way, the number of catalogs stored in the above-mentioned database is enormous due to a rapid increase in the number of Web sites on the Internet, and a reduction in search speed due to this has become a problem. In addition, due to the enlargement of the database, the number of catalogs presented as search results to users has increased, and there has been a problem that it is difficult to find the target Web page unless the content of the query is changed and searched again and again. ing.

[0006] The latter has the following problems. In other words, due to the international nature of the Internet,
Since many b sites have multiple web pages in each language, the catalog will store many catalogs written in languages other than the target language. Therefore, for example, when an English character string keyword such as "trade" is specified as a query, the Web
In spite of the case where only the page is expected to be presented, the catalog of the web page described in another language such as English including the same keyword will be presented as a search result at the same time. It is becoming difficult to present search results to users with high accuracy.

[0007] The present invention has been made to solve the above-described problems, and enables efficient operation of a search engine and presents qualified search results that meet search conditions. An object of the present invention is to provide a catalog creation method and a search method in a Web page search engine operated on the Internet.

[0008]

Means for Solving the Problems The invention described in the first claim of the present invention for achieving this object satisfies the following specific items (1) to (5). (1) A catalog creation method in a web page search engine that holds a catalog describing the contents of web pages displayed on the Internet as a database and performs a keyword search on the database under conditions specified by a query. (2) A language ID to be associated with each Web page collected from the Internet to create the list is determined by one of the following means. If language specific information is described in the HTTP header of the Web page, it is associated with a language ID associated with the language specific information. If language specific information is described in the HTML document of the Web page, it is associated with a language ID associated with the language specific information. (3) If the language ID cannot be determined by the means of (2), the language ID is associated with the language ID associated with the top-level domain name of the URL where the Web page is stored. (4) Create an inventory corresponding to each Web page by an inventory creation program described by a predetermined algorithm. (5) The catalog created by each catalog creating program is managed in the database for each language ID determined in (2).

The following effects are obtained by adopting such a configuration. For example, by specifying a search target language by the above-described query and limiting the search target to the list associated with the language ID, the number of search target lists can be significantly reduced. Thus, the search speed is improved. Also, the web page HTTP
By determining the language ID based on the language identification information described in the header and the HTML document, the language ID corresponding to the language describing the Web page from which it was created with high accuracy can be determined. It is possible to classify, and if a language ID is specified in the query, as described above, the list of Web pages in languages other than the target language will be presented in the list presented as search results. It is possible to present a search result with high accuracy to the user without any problem.

According to a second aspect of the present invention, there is provided the catalog creating method according to the first aspect, wherein the means of (2) includes the step of: The language code defined in the "Content-Language" field is referred to as the language specifying information, and the web page is associated with a language ID associated with the language code.

According to a third aspect of the present invention, there is provided the catalog creating method according to any one of the first and second aspects, wherein the means in (2) is a web page. Refers to the character code system defined in the setting statement of "charset" of the "Content-Type" field in the HTTP header of the above as the language identification information, and associates the web page with the language ID associated with this character code system Means.

According to a fourth aspect of the present invention, there is provided the catalog creating method according to any one of the first to third aspects, wherein the means of (2) is a web page. H
The language code defined in the setting sentence of the “lang attribute” in the TML document is referred to as the language specifying information, and the web page is associated with the language ID associated with the language code.

According to a fifth aspect of the present invention, there is provided the catalog creation method according to any one of the first to fourth aspects, wherein the means in (2) is a web page. H
It is a means to refer to the character code system defined by the setting statement of "charset" in the META tag in the TML document as the language specifying information and to associate the relevant web page with the language ID associated with this character code system And

According to a sixth aspect of the present invention, there is provided the catalog creating method according to any one of the third and fifth aspects, wherein the character code system referred to is a plural language I.
If the character code system is associated with D,
Further, the character code A of the character constituting the HTML document of the Web page is compared with a predetermined character code B associated with the language ID, and when the character code A matches the character code B, the character code B Language ID associated with
Means to associate the Web page with the Web page.

According to a seventh aspect of the present invention, there is provided the catalog creating method according to any one of the third to fifth or sixth aspects, wherein the character code system referred to is a plurality of language IDs. In the case where the character code system is related to the Web page, the top-level domain name of the URL of the Web page is further referred to, and the language ID associated with the combination of the character code system and the top-level domain name is added to the Web ID. It is a means for associating pages.

Further, the invention described in the eighth aspect of the present invention satisfies the following specific items (81) to (84). (81) A method for executing a search in response to a request from a user in a Web page search engine to which the catalog creation method according to any one of claims 1 to 7 is applied. (82) The Web page search engine stores the language ID in the user computer accessed through the Internet.
Is sent out. (83) The Web page search engine receives the query set by the query setting page sent from the user computer, and the catalog classified into the language ID specified in the query among the catalogs of the database And performs a search process based on the query. (84) The Web page search engine creates a home page describing the list extracted by the search processing, and sends the created home page to the user computer that sent the query.

[0017]

DESCRIPTION OF THE PREFERRED EMBODIMENTS A search engine according to one embodiment of the present invention will be described below. The configuration of this search engine other than that described below is the same as that of the above-described conventional search engine, and will not be described here.

=== Catalog Creation Program === This search engine catalog creation program collects Web pages from Web sites on the Internet by using an automatic patrol system such as a robot, as in the above-described conventional case. . Next, when creating a catalog based on each Web page collected in this way, the catalog creating program determines a language ID to be associated with each Web page,
The catalog created based on the page is registered in the database for each language ID associated with the corresponding web page. Here, the above-mentioned language ID is determined by the following first to fifth means. Hereinafter, the means for determining the language ID will be described in detail with reference to the flowcharts of FIGS.

<First Means> First, the catalog creation program executes HTTP (Hypertext Transfer Protocol) of the Web page.
ol) Checks whether the “Content-Language” field is included in the header (whether or not a description “Content-Language:” is included) (100). The language ID is determined based on the language code that is present (110 to 120). Here, the language code is a two-letter code defined by ISO-639 (ISO; International Organization for Standardization). The catalog creation program manages a correspondence table indicating correspondence between language codes and language IDs shown in Table 1, and determines a language ID by comparing the language code set in the field with the correspondence table. In the example of FIG. 7, “ja” is set as the language code, and in this case, the language ID determined based on Table 2 is “2”.

[0020]

[Table 1]

<Second Means> If the language ID could not be specified by the above means, then the HTTP header contains a “Content-Type” field (“Content-Type: ) Is included) (200), and if it is included, the character code set in “charset” of this field (described after “charset =”) The language ID is determined based on the character string (210 to 230). Here, the character code system is referred to because, for example, the character code system “US-ASCII” is used for the description in English,
O-2022-JP "," SHIFT_JIS "," X-EUC-JP ", etc. are used for describing Japanese, so there is a fixed between the character code system and the language that describes the web page This is because there is a relationship. The catalog creation program manages a correspondence table indicating correspondence between each character code system and the language ID shown in Table 2, and compares the character code system set in “charset” with the language ID by comparing this correspondence table. Is determined (220 to 230). In the example of FIG. 7, "ch" of the HTTP header
Since “ISO-2022-JP” is set as the character code system in “arset”, the language ID determined based on Table 2 is “2” in this case.

[0022]

[Table 2]

Incidentally, the character code systems include, for example, the character code systems “ISO-8859-4”, “UTF7”, “UT
Some are commonly used in multiple languages, such as "F8", "UTF9", and "UNICODE" .If such a character code system is set to "charset", The language ID cannot be determined directly. Therefore, when such a character code system is set, the following processing (221 to 222) is further performed. First, the HTML (HyperText Markup La
nguage) Inspect the character codes describing the text characters that make up the document to see if there is a character code specific to each language. Here, the character code unique to each language is, for example, a character code of "Hiragana" in Japanese, and a "Hangul character" in Korean. If a unique character code exists, the language ID of the language having the character code is determined as the language ID of the web page. When the language ID cannot be determined only from the character code of one character, the language ID may be determined by a predetermined algorithm that investigates a set of a plurality of text characters constituting the HTML document.

On the other hand, if the language ID cannot be determined by the above character code search, the U
The language ID is specified based on the top-level domain name of the RL (223 to 224). The top level domain name is obtained from the request header when the web page was obtained. As is well known, the URL top-level domain is a generic top-level domain (generic TLD; gTLD) such as com or org, an international top-level domain such as int, or jp (Japan) specified by ISO3166. And kr
There are country-specific top-level domains such as (Korea), and the language that describes the web page can be specified from the top-level domain name. The catalog creation program manages the correspondence table between the top-level domain name and the language ID shown in Table 3, and checks the language ID corresponding to the top-level domain name of the web page from this correspondence table, and checks the language ID.
Language ID associated with the catalog created based on page b
To determine. In the case of a character code system in which it is apparent that the character code cannot be determined by the characteristics of the character code, the determination based on the top-level domain name may be performed immediately without determining the character code.

[0025]

[Table 3]

<Third Means> If the language ID cannot be specified by any of the above means, the catalog creation program next attempts to determine the language ID based on the description in the HTML document. In the catalog creation program, the META tag is described in the HTML document, and the HTTP-EQUIV attribute "charse
It is checked whether a character code is set in "t", and if it is, an attempt is made to determine a language ID based on the character code system (300 to 330). FIG. 8 is an example of an HTML document in which such a description exists. This processing is the same as the second means (200 to 2) described above except for the difference in whether the search target is an HTTP header or an HTML document.
The description is omitted because it is the same as 30).

<Fourth Means> If the language ID cannot be determined by any of the above means, the catalog creation program then proceeds to the “lang attribute” described in the HTML document.
Attempt to determine the language ID based on the language code defined in the setting sentence (see FIG. 8) (400 to 420). This processing is the same as that of the first means (100 to 120) except for the difference in whether the search target is an http header or an HTML document.

<Fifth Means> Finally, the above-described first to fourth aspects are described.
If the language cannot be specified by any of the above means, it is determined by the top level domain name of the Web page (500 to 600). This processing is the same as the processing (223 to 224) of the top-level domain name described in the second means, and the description is omitted here.

In the above description, it is described that the first to fifth means are processed in this order. However, the processing order of these means is not limited, and the processing efficiency is maximized. The processing order can be changed arbitrarily. In the above description, the case where the Web page is described in an HTML document is described.
(Standard Generalized Markup Language), XML (Exte
nsive Markup Language), HDML (Handheld Device Mar
kup Language), C-HTML (Compact-HyperText Markup L
Needless to say, the same processing can be performed even for a document other than HTML such as an anguage).

=== Database === In the database of the search engine in this embodiment, a large number of catalogs registered as described above are managed for each language ID. The contents of the catalog are the same as before,
An abstract or summary sentence created based on the collected Web pages, and the URL (Uni
form Resource Locator). The catalog is managed for each language ID as described above, but the management form is not particularly limited. As a method of managing each language ID, for example, a method of attaching a language ID to a list or a method of managing the list with a different database for each language ID can be considered.

=== Search Engine Usage Mode === Next, a search engine usage mode in the present embodiment will be described. When this search engine is accessed from a user computer through the Internet, the language I
A query setting page having a designation field of D is transmitted. The user computer reads the query setting page
Display it on a Web browser and prompt the user to set a query. When the user sets a language ID, a search key, and the like on this page and performs a predetermined sending operation, the user computer sends the query set on the page to the search engine in response.

The search engine receives a query sent from the user computer and searches the database for a list based on the query. Here, the search engine refers to the language ID set in the query in the search, and limits the search target to a list associated with the language ID. For example, when a search query is sent from a user computer with the search target language set to "Japanese", the search engine searches only the information associated with "Japanese" as the search target. By doing so, the search target is narrowed down, and high-speed search becomes possible. The search engine creates a home page describing the list extracted by the search process, and sends it to the user computer that sent the query.

[0033]

The present invention is applied as described above.
In the Web page search engine, the catalog is managed by language, and it is possible to search using only the catalog corresponding to the language ID included in the query sent from the user computer as a search condition, As a result, the search speed can be significantly improved as compared with the case where all the catalogs stored in the database are searched as a reference. Further, if a search program for only the list corresponding to the language ID is prepared for each language ID, it is possible to provide a search program in which a search algorithm optimized for the language ID is mounted. Thus, various ripple effects such as improvement in search speed and search accuracy and improvement in the quality of the catalog presented as search results can be expected.

Also, based on the language identification information described in the HTTP header of the Web page or the HTML document, a language ID
Is determined, it is possible to classify the language ID corresponding to the language describing the web page from which the web page was created with high accuracy, and the language ID is included in the query.
Is specified, the catalog presented as a search result does not show the catalog of Web pages in languages other than the target language as described above, and the user can search with high accuracy. The result can be presented.

[Brief description of the drawings]

FIG. 1 is a diagram showing a conceptual configuration of a typical search engine.

FIG. 2 is a flowchart illustrating processing of a catalog creation program according to an embodiment of the present invention.

FIG. 3 is a flowchart illustrating processing of an inventory creation program according to an embodiment of the present invention.

FIG. 4 is a flowchart illustrating a process of an inventory creation program according to an embodiment of the present invention.

FIG. 5 is a flowchart illustrating a process of an inventory creation program according to an embodiment of the present invention.

FIG. 6 is a flowchart illustrating a process of an inventory creation program according to an embodiment of the present invention.

FIG. 7 is a diagram illustrating an example of an HTTP header.

FIG. 8 is a diagram illustrating an example of an HTML document.

[Explanation of symbols]

100 to 120 Language ID determination process using “lang” in HTTP header 200 to 230 Language ID determination process using “charset” in HTTP header 300 to 330 Language ID determination process using “charset” in HTML document 400 to 420 Language ID by "lang" in HTML document
Decision processing

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Shingo Yamashita 1-34-14 Hatsudai, Shibuya-ku, Tokyo Inside Evolex Co., Ltd. (72) Inventor Hiroshi Fukuda 1-34-14 Hatsudai, Shibuya-ku, Tokyo Stock Company F-term in Evolex (reference) 5B075 NS01 PP22

Claims (8)

[Claims] An invention which satisfies the following specific items (1) to (5). (1) A catalog creation method in a web page search engine that holds a catalog describing the contents of web pages displayed on the Internet as a database and performs a keyword search on the database under conditions specified by a query. (2) A language ID to be associated with each Web page collected from the Internet to create the list is determined by one of the following means. If language specific information is described in the HTTP header of the Web page, it is associated with a language ID associated with the language specific information. If language specific information is described in the HTML document of the Web page, it is associated with a language ID associated with the language specific information. (3) If the language ID cannot be determined by the means of (2), the language ID is associated with the language ID associated with the top-level domain name of the URL where the Web page is stored. (4) Create an inventory corresponding to each Web page by an inventory creation program described by a predetermined algorithm. (5) The catalog created by each catalog creating program is managed in the database for each language ID determined in (2). 2. The catalog creation method according to claim 1, wherein said means of (2) is an HTT of the Web page.
The language code defined in the “Content-Language” field in the P header is referred to as the language identification information, and the language ID associated with the language code is referred to as the language ID.
It is a means for associating b pages. 3. The catalog creation method according to claim 1, wherein said means of (2) is a method of generating a list of “Content-Type” in an HTTP header of the Web page.
A character code system defined by a setting statement of "charset" in a field is referred to as the language specifying information, and the web page is associated with a language ID associated with the character code system. 4. The method for creating an inventory according to claim 1, wherein said means of (2) is executed by
Refer to the language code defined in the “lang attribute” setting sentence in the HTML document of page b as the language identification information, and add the language ID associated with this language code to the Web ID.
It is a means for associating pages. 5. The catalog creation method according to any one of claims 1 to 4, wherein the means in (2) is a list creation method.
Means for referring to the character code system defined by the “charset” setting sentence in the META tag in the HTML document of page b as the language specifying information, and associating the web page with the language ID associated with this character code system It is characterized by being. 6. The catalog creating method according to claim 3, wherein the referenced character code system is a character code system associated with a plurality of language IDs. The character code A of the characters constituting the HTML document of the Web page is compared with the predetermined character code B associated with the language ID, and the character code A is changed to the character code B.
When the web page is matched, the web page is associated with the language ID associated with the character code B. 7. The catalog creating method according to claim 3, wherein the referenced character code system is a character code system associated with a plurality of language IDs. Further, it is a means for referring to a top-level domain name of a URL of the web page and associating the web page with a language ID associated with a combination of the character encoding system and the top-level domain name. 8. An invention which satisfies the following specific items (81) to (84). (81) A method for executing a search in response to a request from a user in a Web page search engine to which the catalog creation method according to any one of claims 1 to 7 is applied. (82) The Web page search engine stores the language ID in the user computer accessed through the Internet.
Is sent out. (83) The Web page search engine receives the query set by the query setting page sent from the user computer, and the catalog classified into the language ID specified in the query among the catalogs of the database And performs a search process based on the query. (84) The Web page search engine creates a home page describing the list extracted by the search processing, and sends the created home page to the user computer that sent the query.