JP4824070B2 - Search processing apparatus, search processing method and program for selecting seed of crawler for specialized search using click log - Google Patents

Description

The present invention, the search processing apparatus for performing a seed selection of specialty search crawler using the click log, about the search process how Ho及 beauty program.

  Conventionally, in a general web page search, a crawler for collecting web page information targets all web pages. The crawler is controlled to collect web pages on the WWW (World Wide Web) at regular intervals.

  However, since it is inefficient for the crawler to collect all web pages under the same conditions, the crawler uses various techniques for efficiently collecting web pages.

As a technique related to such crawler information collection, for example , a technique as disclosed in Patent Document 1 has been proposed.
JP 2001-60198 A

  Here, in the search of web pages for a specific specialized field, the crawler delays the collection of web pages that are less frequently updated if all web pages are collected as in the case of general web page searches. Such a web page may not be reflected in search results.

  On the other hand, if a seed document for a specific specialized field is applied to the crawler, web pages to be collected are limited, so that web pages with a low update frequency can be included in the search results. However, in this case, there is a possibility that web pages in a specific specialized field that should be collected are not collected.

Accordingly, the present invention provides a web page for a particular area of expertise efficiently collected, and the search processing device capable of suppressing the collecting leakage web page, to provide a search process side Ho及 beauty program Objective.

  The present invention provides the following solutions.

(1) a storage unit that stores a query from a terminal device operated by a user and a click URL selected according to the operation of the terminal device among search results for the query, and the storage unit see, clicking included in the specialized field click logs extracted by specialty click log extracting means and the expertise click log extracting means for extracting a specialization click logs based on keywords characterizing the predetermined areas of expertise be searched and authority pages extracting means for extracting the authority page suitable for searching for the specialty based on click frequency, the back link and / or the forward link to the extracted the authority page by the authority page extraction means for URL and search means exploration that, the authority page Leaving the authority pages extracted by means a node, before and graph generation means for generating a directed graph to directed side of the back link and / or the forward link which is searched by Kisagu search means, said graph generating means If the score calculation means for calculating a score of the authority page is each node of the generated said directed graph, the score calculated by the score calculating means is equal to or greater than a predetermined value, the authority the score is calculated page And a hub page determining unit that determines a hub page for crawling a search target in the predetermined specialized field.

According to the configuration of (1), the search processing device associates the query from the terminal device operated by the user with the click URL selected according to the operation of the terminal device among the search results for the query. The storage means to be stored and the storage means are referred to, and the specialized field click log is extracted based on the keyword that characterizes the predetermined specialized field, and the specialized field is selected based on the click frequency for the click URL included in the extracted specialized field click log. Extract authority pages suitable for field search. Then, a back link to the extracted authority page and / or a forward link is searched, and a directed graph having the extracted authority page as a node and the searched back link and / or forward link as a directed edge is generated. Then, the score of the authority page that is each node of the generated directed graph is calculated, and when the calculated score is equal to or greater than a predetermined value, the authority page for which the score is calculated is used to crawl the search target in the predetermined specialized field. Determine as the hub page.

Thus, since the hub page is a web page having a score equal to or higher than a predetermined value, the hub page can be connected to a high-quality web page suitable for a search target in a predetermined specialized field. Therefore, the crawler, by performing a crawl in the search target of a given area of expertise by using a URL that identifies the hub page as a seed URL, it is possible to collect a large number of high-quality web pages in less time.

(2) The search processing device according to (1), wherein the score calculation means calculates any one of a PageRank score and a HITS Hub score as the score.

According to the configuration of (2), the search processing device calculates any one of the PageRank score and the HITS Hub score as the score. Thereby, a URL suitable as a seed URL can be extracted.

(3) Refer to a storage unit that associates and stores a query from a terminal device operated by a user and a click URL selected according to the operation of the terminal device among search results for the query, A specialized field click log extraction step for extracting a specialized field click log based on a keyword that characterizes a predetermined specialized field, and a click frequency for a click URL included in the specialized field click log extracted by the specialized field click log extraction step and authority page extraction step of extracting authority page suitable for searching for the specialty based, a back link and / or exploration search steps you search forward link of the authority pages extracted by the authority page extraction step, The authority page Extracted with the authority page node extracted in step, a graph generation step of generating a directed graph to directed side of the back link and / or the forward link that is pre-searched by Kisagu search step, generating step the graph If the score calculation step of calculating a score of the authority page is each node of the generated said directed graph, said score calculated by the score calculating step is a predetermined value or more, the authority the score is calculated page A search processing method for causing a computer to execute a hub page determination step for determining a hub page for crawling a search target in the predetermined specialized field.

  According to the configuration of (3), it is possible to provide, as a method, an invention that exhibits the same effects as the search processing device of (1).

(4) Referring to a storage unit that associates and stores a query from a terminal device operated by the user and a click URL selected according to the operation of the terminal device among search results for the query, A specialized field click log extraction step for extracting a specialized field click log based on a keyword that characterizes a predetermined specialized field, and a click frequency for a click URL included in the specialized field click log extracted by the specialized field click log extraction step and authority page extraction step of extracting authority page suitable for searching for the specialty based, a back link and / or exploration search steps you search forward link of the authority pages extracted by the authority page extraction step, The authority page Extracted with the authority page node extracted in step, a graph generation step of generating a directed graph to directed side of the back link and / or the forward link that is pre-searched by Kisagu search step, generating step the graph If the score calculation step of calculating a score of the authority page is each node of the generated said directed graph, said score calculated by the score calculating step is a predetermined value or more, the authority the score is calculated page A computer program for causing a computer to execute a hub page determining step for determining a hub page for crawling a search target in the predetermined specialized field.

  According to the configuration of (4), it is possible to provide an invention that exhibits the same effect as the search processing device of (1) as a computer program.

According to the present invention, the web pages for a particular area of expertise efficiently collected, and the search processing device capable of suppressing the collecting leakage web page, to provide a search process side Ho及 beauty program it can.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[overall structure]
FIG. 1 is a diagram showing an outline of a configuration including a search system 1 according to an embodiment of the present invention. Retrieval system 1, the terminal device 3及 beauty external server 4 is communicably connected via a network 2.

The terminal device 3 is represented by a terminal device such as a personal computer, a mobile phone, a PHS (Personal Handy-phone System: registered trademark), a mobile, a notebook PC, or a PDA (Personal Digital Assistant), and performs wired or wireless communication. Any information can be used as long as the information transmitted from the search system 1 can be referred to via the network 2 to be performed. Further, the search system 1 may be configured to include a web server, a database (hereinafter referred to as DB) server, and an application server as necessary. It may be configured. The search system 1 includes a search processing device 20.

[Hardware Configuration of Search Processing Device 20]
FIG. 2 is a diagram illustrating a hardware configuration of the search processing device 20 according to the present embodiment. As shown in FIG. 2, a central processing unit (CPU) 210 (a plurality of CPUs such as CPU 220 may be added in a multiprocessor configuration), a bus line 100, a communication I / F (I / F) Interface) 230, main memory 240, BIOS (Basic Input Output System) 250, I / O controller 260, hard disk 270, optical disk drive 280, and semiconductor memory 290. Incidentally, a hard disk 270, optical disk drive 280 and semiconductor memory 290 are collectively referred to as storage device 310.

The control unit 200 is a part that controls the search processing device 20 in an integrated manner, and by appropriately reading and executing various programs stored in the hard disk 270, the control unit 200 cooperates with the hardware described above, and relates to the present embodiment. Various functions are realized.

  The communication I / F 230 is a network adapter when the search processing device 20 transmits / receives information to / from other devices such as the terminal device 3 via the network 2.

  The BIOS 250 records a boot program executed by the CPU 210 when the search processing device 20 is started, a program depending on the hardware of the search processing device 20, and the like.

The I / O controller 260 can be connected hard disk 270, the optical disk drive 28 0及 beauty semiconductor memory 290 or the like of the storage device 310.

The hard disk 270 stores various programs for causing the hardware to function as the search processing device 20, a program for executing the functions of the present embodiment , a database to be described later, and the like. Note that the search processing device 20 can also use a hard disk (not shown) separately provided as an external storage device.

As the optical disk drive 280, for example, a DVD-ROM drive, a CD-ROM drive, a DVD-RAM drive, and a CD-RAM drive can be used. In this case, the optical disc 300 corresponding to each drive is used. A program or data can be read from the optical disk 300 by the optical disk drive 280 and provided to the main memory 240 or the hard disk 270 via the I / O controller 260.

  The computer referred to in the present invention refers to an information processing device including a storage device, a control unit, and the like, and the search processing device 20 includes an information processing device including a storage device 310, a control unit 200, and the like.

In the above example, the search processing device 20 has been mainly described. However, the functions described above can also be realized by installing a program in a computer and operating the computer as a server device. Therefore, the functions realized by the search processing device 20 described as an embodiment in the present invention can be executed by executing the above-described method by the computer, or by introducing the above-described program into the computer and executing it. It is feasible.

[Functional configuration of search system 1]
FIG. 3 is a functional block diagram showing a functional configuration of the search system 1 of the present embodiment. The search system 1 includes a search engine 11, a search processing device 20, a seed DB 31, and a crawler 32.

  The search engine 11 receives a query from the terminal device 3 and presents information including a URL corresponding to the received query to the terminal device 3 as a search result. A user who operates the terminal device 3 selects an arbitrary URL from the search results and browses a web page managed by the external server 4.

Search processor 20, click logs DB12 (storage means), an extraction unit 21 (specialization click log extracting means), and authority page extraction unit 22 (authority page extraction means), a search unit 23 (probe search means) A graph generation unit 24 (graph generation unit), a score calculation unit 25 (score calculation unit), a score determination unit 26, a hub page determination unit 27 (hub page determination unit), and an authority page DB 28.

  The click log DB 12 stores a URL (hereinafter referred to as a click URL) selected according to the operation of the terminal device 3 among the search results by the search engine 11 for the query from the terminal device 3 operated by the user. FIG. 4 is a click data table showing an example of the click URL stored in the click log DB 12. As shown in FIG. 4, the click log DB 12 includes an IP address of the terminal device 3 that has selected the click URL, a time at which the click URL is selected, a session, a search rank of the click URL, a query, and a click URL. Are stored in association with each other.

  The click log DB 12 stores a URL connection table indicating a link destination or link source from an arbitrary URL. FIG. 5 is a diagram illustrating an example of the URL connection table. As shown in FIG. 5, the click log DB 12 stores a URL, a URL link destination, and a URL link source as a URL connection table.

  The control unit 200 includes, as function units, an extraction unit 21, an authority page extraction unit 22, a search unit 23, a graph generation unit 24, a score calculation unit 25, a score determination unit 26, and a hub page determination unit 27. Have The click log DB 12 and the authority page DB 28 are stored in the storage device 310.

  The extraction unit 21 extracts a query that is a click log of a specialized field and a click URL from the click log DB 12 based on a keyword that characterizes a predetermined specialized field as a search target. Examples of search subjects in a predetermined specialized field include academic fields, images, products that are traded on the World Wide Web (WWW), voice, music, and the like.

  Specifically, from the click log DB 12 illustrated in FIG. 4, when “volcano” is a search target as a specialized field, the extraction unit 21 includes a query including a character string related to “volcano” as a keyword that characterizes the specialized field. All click URLs corresponding to are extracted from the click log DB 12.

When “image” is a search target as a specialized field, the extraction unit 21 includes “image”, “jpeg”, and “jpg” that are character strings related to “image” as keywords that characterize the specialized field. All click URLs corresponding to the queries are extracted from the click log DB 12.

  Note that the specialized field and the keyword that characterizes the specialized field may match, and if the specialized field and the keyword that characterizes the specialized field do not match, the specialized field and the specialized field are stored in the storage device 310 in advance. You may memorize | store the keyword to characterize.

  The authority page extraction unit 22 extracts the authority page based on the click URL extracted by the extraction unit 21. Specifically, the authority page extraction unit 22 calculates a click frequency that is the frequency at which the click URL extracted from the click log DB 12 by the extraction unit 21 is selected by the terminal device 3.

  And the authority page extraction part 22 determines whether the calculated click frequency is more than a fixed value. When the click frequency is equal to or higher than a certain value, the authority page extraction unit 22 stores the web page specified by the click URL corresponding to the click frequency as the authority page in the authority page DB 28. FIG. 6 is a table showing an example of the click URL stored in the authority page DB 28.

  Although the authority page extraction unit 22 extracts the authority page based on the click frequency, the authority page can be extracted using data related to the click URL. For example, the authority page may be extracted from the domain name for the click URL.

The search unit 23 searches for a back link to at least the authority page extracted by the authority page extraction unit 22. Specifically, the search unit 23 refers to the URL connection table and searches for the link destination (back link) and / or link source (forward link) URL for each authority page stored in the authority page DB 28. To do.

The graph generation unit 24 uses the authority page extracted by the authority page extraction unit 22 as a node, and uses the URL of the back link and / or the forward link searched by the search unit 23 as a directed side. Generates a directed graph composed of directional edges.

FIG. 7 is a schematic diagram showing a graph G created by the graph generation unit 24 based on the URL connection table shown in FIG. As shown in FIG. 7, the graph G, which is a directed graph, includes nodes that are URLs that specify each authority page, and directed edges that are URL backlinks and / or forward links.

  The score calculation unit 25 calculates the score of the authority page that is each node of the directed graph generated by the graph generation unit 24.

As a technique for calculating the score of the authority page, for example , a known technique such as a PageRank (registered trademark) score or a Hub score by HITS (Hyperlink Induced Topic Search) can be used.

  When using PageRank as the score of a web page, it is possible to calculate PageRank by performing processing mainly using Expression (1).

  PR (A) that is the score of PageRank of page A linked from pages T1, T2,..., TN is defined by equation (1).

PR (A) = (1−d) + d (PR (T1) / C (T1) + PR (T2) / C (T2) +... + PR (TN) / C (TN)) (1)
However, 0 ≦ d ≦ 1

  HITS is composed of an authority score auth (n) of page n and a hub score hub (n), and is defined as follows.

  Auth (n) = hub (n) = 1 of all pages (2)

  The processing of the following formulas (3) and (4) is repeated until the authority score and the hub score converge.

  auth (n) = Σhub (m) for all m (m → n) (3)

  hub (n) = Σauth (m) for all m (m → n) (4)

Scoring unit 26 determines whether the score calculated by the score calculation unit 25, i.e. score or Hub scores PageRank is a predetermined value or more.

  When the score determination unit 26 determines that the score is equal to or greater than a predetermined value, the hub page determination unit 27 uses the authority page for which the score is calculated to crawl the search target in a predetermined specialized field. Determine as. Then, the hub page determination unit 27 stores the URL as a seed URL in the seed DB 31 in order to specify the determined hub page.

  As illustrated in FIG. 8, the seed DB 31 stores a seed URL for the crawler 32 to crawl a search target in a predetermined specialized field.

  The crawler 32 collects document information such as a web page on the WWW 33 via the network 2. The crawler is generally called a search robot, and is a program that collects web pages and the like through the network 2. There are various means for the crawler to search for a website and the types of target web pages, and the types and fields of collected web pages vary depending on the settings of the crawler administrator. Here, the crawler 32 of the present embodiment searches for a website based on at least the seed URL stored in the seed DB 31, and collects web pages.

As described above, according to the search processing device 20 of the present embodiment, referring to the click log DB 12, the query and click URL of the specialized field are extracted based on the keyword characterizing the predetermined specialized field, and the extracted click URL is extracted. Based on the click frequency, an authority page suitable for a specialized field search is extracted. Then, a back link and / or forward link to the extracted authority page is searched, and a directed graph is generated with the extracted authority page as a node and the searched back link and / or forward link as a directed edge. Then, the score of the authority page that is each node of the generated directed graph is calculated, and when the calculated score is equal to or greater than a predetermined value, the authority page for which the score is calculated is used to crawl the search target in the predetermined specialized field. Determine as the hub page.

  The hub page is a web page having a score equal to or higher than a predetermined value, and can be connected to a high-quality web page suitable for a search target in a predetermined specialized field. Therefore, the crawler can collect a large number of high-quality web pages in a short time by performing crawling on a search target in a predetermined specialized field using a URL specifying a hub page as a seed URL.

  In addition, by preferentially crawling to a seed URL that can be connected to a high-quality web page, it is possible to frequently collect higher-quality web pages and reflect the collected web pages in search results for queries. .

Further, according to the search processing device 20 of the present embodiment, any one of a PageRank score and a HITS Hub score is calculated as the score. Thereby, a URL suitable as a seed URL can be extracted.

Further, according to the search processing apparatus 20 of the present embodiment, it is possible to suitably perform a crawl of disciplines, image, audio, disciplines and a sound Raku及 beauty product information search.

[Process flow]
Next, the processing flow of the search processing device 20 of this embodiment will be described with reference to the flowchart of FIG.

  In step S <b> 1, the extraction unit 21 extracts all click URLs corresponding to the query including the character string related to the specialized field from the click log DB 12 based on the keyword characterizing the specialized field to be searched.

  In step S <b> 2, the authority page extraction unit 22 aggregates the click URLs extracted from the click log DB 12 by the extraction unit 21, and calculates a click frequency that is the frequency with which the click URL is selected by the terminal device 3.

  In step S3, the authority page extraction unit 22 determines whether or not the click frequency calculated in step S2 is a certain value or more. If the click frequency is greater than or equal to a certain value (Yes), the process proceeds to step S4. On the other hand, if the click frequency is less than a certain value (No), the series of processes is terminated.

  In step S4, the authority page extraction unit 22 stores the web page specified by the click URL corresponding to the click frequency determined in step S3 in the authority page DB 28 as the authority page.

In step S5, the search unit 23 searches the URL of the back link and / or forward link for each authority page stored in the authority page DB 28 with reference to the URL connection table.

In step S <b > 6, the graph generation unit 24 determines whether or not the search for the URL of the authority page back link and / or forward link web page in step S <b > 5 is a predetermined number of times or more. If the URL search is more than the predetermined number of times (Yes), the process proceeds to step S7. On the other hand, if the URL search is less than the predetermined number of times (No), the series of processing is terminated.

In step S7, the graph generation unit 24 uses the URL that specifies the authority page as a node, and uses the back link and / or the forward link for each authority page as the directed side, and uses these nodes and the directed side. Generate a directed graph consisting of

  In step S <b> 8, the score calculation unit 25 calculates the score of the authority page that is each node of the graph generated by the graph generation unit 24.

  In step S9, the score determination unit 26 determines whether the score calculated by the score calculation unit 25 is greater than or equal to a predetermined value. If the score is greater than or equal to the predetermined value (Yes), the process moves to step S10. On the other hand, when the score is less than the predetermined value (No), the series of processes is terminated.

In step S10, the hub page determination section 27, the authority page based on the score calculated in step S8, and determines as a Habupe di for searching crawled disciplines, identifying the determined hub pages The URL is stored in the seed DB 31 as a seed URL.

  As mentioned above, although embodiment of this invention was described, this invention is not restricted to embodiment mentioned above. The effects described in the embodiments of the present invention are only the most preferable effects resulting from the present invention, and the effects of the present invention are limited to those described in the embodiments of the present invention. is not.

It is a figure showing an outline of composition containing search system 1 concerning one embodiment of the present invention. It is a figure which shows the hardware constitutions of the search processing apparatus 20 of this embodiment. It is a functional block diagram which shows the function structure of the search system 1 of this embodiment. It is a table which shows an example of click URL memorize | stored in click log DB12. It is a figure which shows an example of a URL connection table. It is a table which shows an example of the authority page memorize | stored in authority page DB28. FIG. 4 is a schematic diagram showing a graph G created by a graph generation unit 24. It is a table which shows an example of seed URL. It is a flowchart shown about the flow of a process of the search processing apparatus 20 of this embodiment.

DESCRIPTION OF SYMBOLS 1 Search system 2 Network 3 Terminal device 4 External server 11 Search engine 12 Click log DB (storage means)
20 Search processing device 21 Extraction unit (Special field click log extraction means)
22 Authority Page Extraction Unit (Authority Page Extraction Means)
23 search unit (probe search means)
24 Graph generation unit (graph generation means)
25 Score calculation unit (score calculation means)
26 Score determination unit 27 Hub page determination unit (hub page determination means)
28 Authority Page DB
31 Seed DB
32 crawler

Claims (4)

A storage unit that associates and stores a query from a terminal device operated by a user and a click URL selected according to an operation of the terminal device among search results for the query;
A specialized field click log extracting means for extracting a specialized field click log based on a keyword characterizing a predetermined specialized field with reference to the storage means;
Authority page extraction means for extracting an authority page suitable for searching in the specialized field based on a click frequency for a click URL included in the specialized field click log extracted by the specialized field click log extracting means;
And search means exploration you search back link and / or the forward link to the extracted the authority page by the authority page extraction means,
A graph generating means and extracted the authority pages node, generating a directed graph to directed side of the back link and / or the forward link which is searched by the pre Kisagu search unit by the authority page extraction means,
Score calculating means for calculating a score of the authority page that is each node of the directed graph generated by the graph generating means;
If the score calculated by the score calculating means is equal to or greater than a predetermined value, the hub page determining means for determining the authority pages in which the score is calculated as a hub page for searching crawled the predetermined areas of expertise A search processing apparatus comprising: The search processing device according to claim 1, wherein the score calculation unit calculates one of a PageRank score and a HITS Hub score as the score. Referring to a storage unit that associates and stores a query from a terminal device operated by a user and a click URL selected according to the operation of the terminal device among search results for the query, and stores a predetermined specialization A specialized field click log extraction step for extracting a specialized field click log based on a keyword characterizing the field;
An authority page extracting step for extracting an authority page suitable for searching the specialized field based on a click frequency for a click URL included in the specialized field click log extracted by the specialized field click log extracting step;
A search step exploration you search back link and / or the forward link of the authority pages extracted by the authority page extraction step,
And graph generation step of the authority page extracting the authority page and node extracted in step, to generate a directed graph with a directed edge of the back link and / or the forward link that is pre-searched by Kisagu search step,
A score calculating step of calculating a score of the authority page that is each node of the directed graph generated by the graph generating step;
If the said score calculated by the score calculating step is a predetermined value or more, the hub page determining step of determining a authority page to which the score is calculated as a hub page for searching crawled the predetermined areas of expertise A search processing method for causing a computer to execute . Referring to a storage unit that associates and stores a query from a terminal device operated by a user and a click URL selected according to the operation of the terminal device among search results for the query, and stores a predetermined specialization A specialized field click log extraction step for extracting a specialized field click log based on a keyword characterizing the field;
An authority page extracting step for extracting an authority page suitable for searching the specialized field based on a click frequency for a click URL included in the specialized field click log extracted by the specialized field click log extracting step;
A search step exploration you search back link and / or the forward link of the authority pages extracted by the authority page extraction step,
And graph generation step of the authority page extracting the authority page and node extracted in step, to generate a directed graph with a directed edge of the back link and / or the forward link that is pre-searched by Kisagu search step,
A score calculating step of calculating a score of the authority page that is each node of the directed graph generated by the graph generating step;
If the said score calculated by the score calculating step is a predetermined value or more, the hub page determining step of determining a authority page to which the score is calculated as a hub page for searching crawled the predetermined areas of expertise A computer program that causes a computer to execute.

Images