JP5226725B2 - Search engine and method for confirming status of processing sequentially performed by a plurality of servers - Google Patents

Description

The present invention relates to a system, a method, and a search engine for confirming the status of processes sequentially performed by a plurality of servers. In particular, the present invention relates to a search engine and a method for identifying a server causing a problem without interrupting processes sequentially performed by a plurality of servers.

  2. Description of the Related Art Conventionally, Internet search sites provide a service that provides search results to a user as a list in response to a search request such as keyword input by the user. The operation side of such a search site usually collects data such as documents and images on a Web page on the Internet using a search engine having a program called a crawler or a robot.

  Here, regarding a search engine, a technique for dealing with Web pages updated one after another all over the world is provided. For example, according to the technique described in Patent Document 1, information related to a specific site to be accessed, information indicating the date and time when information collection of the specific site is started, and the current time coincides with the information collection start time. Then, when it is determined that information collection is started, the Web crawler efficiently collects Web pages on the Internet.

JP 2004-318746 A

  By the way, the search engine performs various processes such as adding metadata to such data collected by the Web crawler and checking whether or not it is listed in a predetermined black list. A data analysis unit is provided, and an index is added to the data processed by the data analysis unit and stored in the database. Then, a search result list corresponding to the search request by the user is created with reference to the database and transmitted to the user terminal.

  Here, in the search result list transmitted to the user's terminal, the hit Web pages are displayed in a predetermined order (ranking) based on the priority (score) for the search keyword. Such an order with respect to the search keyword greatly affects the access frequency to the Web page, and is therefore very important for an administrator who manages the Web page. Therefore, the operation side of the search site pays great attention to the order in the search result, and when there is an abnormal change in the order, it is necessary to quickly identify and deal with the cause.

The change in the order in the search result list is caused by a number of causes, and it is known that there is also a cause for the processing of the data analysis unit performed before index creation. Here, the data analysis unit is configured by a plurality of servers, and the plurality of servers sequentially execute processing on data collected by the Web crawler. Therefore, when there is an abnormal change in the order in the search result list, there is a problem that it is difficult to confirm at which stage of the data analysis unit the abnormal change has occurred.
Further, in recent years, services based on search engines are required to be continuously provided without stopping. Therefore, when confirming an abnormality, it is desirable to identify the server in question without interrupting the processing of the data analysis unit, that is, the processing sequentially performed by a plurality of servers.

The present invention has been made in view of such a problem, and confirms the status of processing sequentially performed by a plurality of servers, and identifies a server causing the problem without interrupting the sequential processing. The present invention relates to a search engine and method capable of performing the above.

  The present invention provides the following solutions.

(1) A web crawler that circulates web pages and collects crawl data, a data analysis unit that creates data used for a search index from the crawl data collected by the web crawler, and a data analysis unit A search engine comprising: an indexer that creates a search index based on the generated data; and a search server that responds to a search request from a user using the search index created by the indexer, wherein the data The analysis unit includes a plurality of servers that sequentially process the crawl data to create data used for the search index, and an aggregation server that aggregates the processing results of the plurality of servers. Each time a process is executed, the transaction log associated with that process The in response to the accepted and belt transaction log storage means to store in association with the URL of the Web page corresponding to the crawl data, a status confirmation request specifying the URL, associated with the URL transaction log extracted before Quito transaction log storage hands stage or, et al., and a log extracting means for sending to the aggregation server, the aggregation server includes a log aggregation means for aggregating a plurality of said transaction log accepted search Engine .

According to the search engine of (1), when a plurality of servers execute processing on data identified by the URL , a transaction log as the processing content is stored in the transaction log storage means. When a status confirmation request specifying a URL is received, transaction logs associated with the URL are extracted and aggregated. As a result, the contents of the processing executed by each of the plurality of servers for the data identified by the URL can be grasped via the extracted transaction log, and the status of the processing sequentially performed by the plurality of servers can be determined. It can be confirmed via the aggregated transaction log.
Further, since a transaction log is extracted when a status confirmation request is received, the (main) processing sequentially performed by a plurality of servers is not affected.
Therefore, it becomes possible to grasp the contents of the processing of the plurality of servers without interrupting the processing sequentially performed by the plurality of servers, and it is possible to identify the server causing the problem.

(2) The status confirmation request includes designation of server identification information for identifying the plurality of servers in addition to designation of the URL , and the log extracting unit is associated with the URL included in the status confirmation request. The search engine according to (1), wherein a transaction log is extracted from a transaction log storage unit provided in a server identified by the server identification information.

According to the search engine of (2), a transaction log is extracted from a specific server identified by server identification information included in the status confirmation request among a plurality of servers. As a result, when an abnormality occurs in the processing that is sequentially performed by a plurality of servers, it is possible to confirm the processing contents of only the server that is assumed to have performed the processing that causes it, and perform new processing as the entire system The cause can be quickly identified while limiting the processing load by limiting the number of servers.

(3) A web crawler that circulates web pages and collects crawl data, a data analysis unit that creates data used for a search index from the crawl data collected by the web crawler, and a data analysis unit A search engine comprising: an indexer that creates a search index based on the generated data; and a search server that responds to a search request from a user using the search index created by the indexer, wherein the data The analysis unit includes a plurality of servers that sequentially process the crawl data to create data used for the search index, and an aggregation server that aggregates the processing results of the plurality of servers. Each time a process is executed, the transaction log associated with that process A transaction log storage means for storing the URL in association with the URL of the Web page corresponding to the crawl data, and a transaction log associated with a specific URL among the transaction logs stored in each of the transaction log storage means. Log extracting means for extracting the log and sending it to the aggregation server, wherein the aggregation server associates the extracted transaction logs with server identification information for identifying the URL and the plurality of servers. A search engine having log aggregation means for aggregating.

According to the search engine of (3), it is possible to confirm the status of processing sequentially performed by a plurality of servers by confirming a transaction log associated with a specific URL extracted at a predetermined cycle.

(4) A web crawler circulates web pages and collects crawl data, a data analysis unit creates data used for a search index from the crawl data collected by the web crawler, and an indexer creates the data analysis unit A search method for creating a search index based on the data created by the index server and responding to a search request from a user using the search index created by the indexer by a search server, wherein the data analysis unit Each of the plurality of servers sequentially processes the crawl data to create data used for the search index, and the aggregation server aggregates the processing results of the plurality of servers. Each time the log storage means executes a process, The transaction log associated with the process is stored in association with the URL of the Web page corresponding to the crawl data, and the log extracting unit receives the status confirmation request designating the URL. A search method comprising: extracting a transaction log associated with the transaction log storage unit and sending the transaction log to the aggregation server; and a step of log aggregation unit in the aggregation server aggregating the plurality of received transaction logs .

According to how the (4), provides the system with similar effects of (1). In particular, it is possible to confirm whether abnormal fluctuations in order in any stage in the search results in a list of configured data analysis unit from multiple servers occurs.

  According to the present invention, it is possible to confirm the state of processing sequentially performed by a plurality of servers, and to identify a server causing a problem without interrupting the sequential processing.

It is a figure which shows the whole structure of the search engine of this invention. It is a figure which shows the structure of the data analysis part of this invention. It is a figure which shows the transaction log which is the content of the process by the data analysis part of this invention. It is a flowchart which shows the main process performed by the data analysis part of this invention. It is a flowchart which shows the log aggregation process performed by the data analysis part of this invention. It is a flowchart which shows the log aggregation process performed by the data analysis part of deformation | transformation embodiment.

(Embodiment)
Hereinafter, a search engine 1 including a data analysis unit 11 that is a preferred example of the system will be described with respect to a system that sequentially performs processing by a plurality of servers of the present invention. Note that the present invention is applicable to any system that sequentially performs processing by a plurality of servers, and is not limited to the data analysis unit 11 of the search engine 1.

Search engine configuration
FIG. 1 is a diagram showing an overall configuration of a search engine 1 according to the present invention.
The search engine 1 includes a web crawler 10, a data analysis unit 11, an indexer 12, and a search server 13.

  The Web crawler 10 is a program that periodically circulates Web pages on the Internet, and automatically collects data of various Web pages by following links in HTML (HyperText Markup Language). Web page data collected by the Web crawler 10 (hereinafter referred to as “crawl data”) is associated with a URL (Uniform Resource Locator) of the Web page and is not illustrated (hereinafter, “database” is referred to as “DB”). Stored). In the present embodiment, a URL is used as the “predetermined key”, and crawl data is used as the “data identified by the predetermined key”.

The data analysis unit 11 includes a plurality of analysis servers 20A... 20N, and the plurality of analysis servers 20A... 20N sequentially execute processing on the crawl data identified by the URL. Perform a series of preprocessing to generate the index.
As preprocessing, the data analysis unit 11 sequentially performs processing such as HTML tag removal, metadata addition, and score determination on the crawl data, for example. “Removal of HTML tag” refers to, for example, extracting the title of the Web page specified in the title tag, and “Addition of metadata” includes, for example, the length of the path, the Web page. The keyword, HTML size, and the like are extracted, and “score determination” calculates, for example, various scores such as an adult score and an NG score, and whether or not the calculated score exceeds a threshold value. It means judging. If it is determined by the score determination that the threshold is exceeded, a process corresponding to the corresponding score is performed. For example, if the NG score exceeds the threshold, the process is not displayed as a search result. Is done.
The crawl DB is updated each time processing is performed by the data analysis unit 11 (analysis server 20).

  The indexer 12 analyzes the data in the crawl DB updated by the data analysis unit 11 and creates a search index 121. For example, the indexer 12 can create a search index 121 that is ranked based on the linked information, and also creates a search index 121 that is ranked based on the relationship between the keyword and title of the Web page. be able to.

  The search server 13 creates a search result list with reference to the search index 121 for the search keyword received from the user terminal 2, and returns the search result list to the user terminal 2.

  As described above, the search engine 1 is configured to respond to a search request from the user terminal 2 using the search index 121 for search processing created by the Web crawler 10, the data analysis unit 11, and the indexer 12. ing.

[Configuration of data analysis section]
Next, a specific configuration of the data analysis unit 11 which is a preferred embodiment of the present invention will be described. FIG. 2 is a block diagram illustrating a configuration of the data analysis unit 11.

  The data analysis unit 11 aggregates a plurality of analysis servers 20A, 20B,... 20N that sequentially process the crawl data identified by the URL, and processing statuses in the analysis servers 20A, 20B,. And the server 30. In the present embodiment, the crawl data is processed in the order of analysis server 20A to analysis server 20B... Analysis server 20N.

The analysis server 20A includes a main processing unit 21A, a transaction log storage unit 22A, and a log extraction unit 23A.
The configuration of the analysis servers 20B... 20N is the same as the configuration of the analysis server 20A. That is, the analysis servers 20B ... 20N are configured to include main processing means 21B ... 21N, transaction log storage means 22B ... 22N, and log extraction means 23B ... 23N. Hereinafter, when not distinguishing each, they are simply referred to as “analysis server 20”, “main processing means 21”, “transaction log storage means 22”, and “log extraction means 23”.

  The main processing means 21 executes various processes on the crawl data and updates the crawl DB. For example, the main processing means 21 performs processing such as HTML tag removal, metadata addition, and score determination, and inputs values associated with the processing to the crawl DB. The content of the processing (transaction log) performed by the main processing means 21 is transmitted to the analysis server 20 in the next order.

The transaction log storage unit 22 stores the transaction log received from the analysis server 20 in the previous order. Thus, every time the main processing unit 21 of the analysis server 20 executes processing on the crawl data, the contents of the processing are stored in the transaction log storage unit 22 in association with the URL.
The transaction log stored in the transaction log storage unit 22 is read by the main processing unit 21, and thereafter the processing is executed by the main processing unit 21. The reading of the transaction log stored in the transaction log storage unit 22 and the subsequent processing by the main processing unit 21 are the main processing of the data analysis unit 11 (a series of preprocessing for generating a search index). ).

  In the present embodiment, the transaction log storage unit 22 stores the contents of the processing performed by the analysis server 20 in the previous order. It is good also as memorize | storing the content of the process which performed. That is, in this embodiment, the contents of the processing performed by the main processing means 21A of the analysis server 20A are stored in the transaction log storage means 22B of the analysis server 20B in the following order, but the analysis server 20A The contents of the processing performed by the main processing means 21A may be stored in the transaction log storage means 22A of the analysis server 20A.

  When the log extraction unit 23 receives a status confirmation request for designating a URL from the aggregation server 30, the log extraction unit 23 extracts a transaction log associated with the crawl data identified by this URL from the transaction log storage unit 22, 30.

  The aggregation server 30 includes a log request unit 31 and a log aggregation unit 32.

  The log request unit 31 transmits a state confirmation request to the analysis server 20 and receives a transaction log corresponding to the state confirmation request from the analysis server 20. Here, in addition to the URL, the status confirmation request may include server identification information (for example, an IP address) indicating the types of the analysis servers 20A... 20N. By including the server identification information in the status confirmation request, the transaction log can be acquired only from any analysis server 20 among the plurality of analysis servers 20A.

  The log aggregating unit 32 aggregates the acquired transaction logs so that the administrator can check the processing status for the crawl data identified by the URL.

Here, FIG. 3 shows a transaction log for the crawl data identified by the URL “http://www.abcdefg.co.jp/index.html”, and FIG. 3 (1) shows the transaction log of the analysis server 20A. (Stored in the transaction log storage unit 22B of the analysis server 20B), FIG. 3B shows the transaction log of the analysis server 20B (stored in the transaction log storage unit 22C of the analysis server 20C). .
The transaction log indicates the update contents of the crawl DB, and defines the value to be input for a predetermined item. For example, referring to FIG. 3A, the process 200A inputs a value “http://www.abcdefg.co.jp/index.html” to the “URL” item of the crawl DB. Means.

  The log aggregation unit 32 extracts the contents of the processing performed by each analysis server 20 from the acquired transaction log. For example, the process 200 performed by the analysis server 20A is extracted from the transaction log of the analysis server 20A and the transaction log of the analysis server 20B, and the process 201 performed by the analysis server 20B is extracted.

  From the processing of each analysis server 20 extracted by the log aggregating means 32, the administrator can check the processing status for the crawl data identified by the URL. For example, when an abnormality occurs in which the Web page “http://www.abcdefg.co.jp/index.html” is not displayed as a search result in the search engine 1, the administrator processes the analysis server 20B. By 201A, it can be understood that the cause is that the calculation result of the NG score indicating that the search result is not displayed is determined to be greater than or equal to a predetermined value.

  As described above, in the data analysis unit 11 according to the present embodiment, the log extraction unit 23 extracts the transaction log from the transaction log storage unit 22 independently of the main processing unit 21. The state of the processing executed by each of the plurality of analysis servers 20 can be confirmed without affecting the processing (a series of preprocessing for generating the search index). That is, the administrator can grasp the contents of the processing without interrupting the processing sequentially performed by the analysis server 20 and can identify the analysis server 20 causing the problem.

  Note that the hardware of the analysis server 20 and the aggregation server 30 can be configured by a general computer. For example, a general computer includes a central processing unit (CPU) as a control unit, and a memory (RAM, ROM), a hard disk (HDD), and an optical disk (CD, DVD, etc.) as a storage unit, and a network communication device. Various wired and wireless LAN devices as display devices, for example, various displays such as liquid crystal displays and plasma displays, and as input devices, for example, keyboards and pointing devices (mouse, tracking balls, etc.) Are connected by a bus line. In such a general computer, the CPU controls the analysis server 20 and the aggregation server 30 in an integrated manner, and reads and executes various programs as appropriate, thereby cooperating with the hardware described above, and according to the present invention. Implement various functions.

[Data analysis processing]
Next, processing of the data analysis unit 11 will be described. 4 is a flowchart of main processing performed by each analysis server 20, and FIG. 5 is a flowchart of log aggregation processing for confirming the state of processing performed by each analysis server 20.

The main process will be described with reference to FIG.
S1: The control unit of the analysis server 20 receives a transaction log indicating the contents of processing executed by the main processing unit 21 of the analysis server 20 in the previous order.

  S2: Subsequently, the control unit of the analysis server 20 stores the received transaction log in the storage unit (transaction log storage unit 22).

  S3: Subsequently, the control unit (main processing unit 21) of the analysis server 20 reads the transaction log stored in the storage unit (transaction log storage unit 22), and is identified by the URL associated with the transaction log. A series of preprocessing for generating a search index is performed on the crawl data.

  S4: Subsequently, the control unit of the analysis server 20 transmits the content of the processing (transaction log) by the main processing means 21 to the analysis server 20 in the next order, and ends the main processing performed by each analysis server 20. .

  Next, the log aggregation process will be described with reference to FIG. The log aggregation process is a process performed in parallel with the main process of FIG.

  S <b> 11: The control unit (log request unit 31) of the aggregation server 30 transmits a status confirmation request to the analysis server 20. Here, it is assumed that the status confirmation request includes at least a URL for identifying crawl data whose processing status is to be confirmed. In addition, when it is possible to guess which of the plurality of analysis servers 20 has an abnormality (for example, immediately after changing the version of a specific analysis server), an IP address or the like for identifying the analysis server 20 Server identification information is also preferably included in the status confirmation request.

S12: Subsequently, when the control unit (log extraction unit 23) of the analysis server 20 receives the state confirmation request, the transaction associated with the URL included in the state confirmation request is stored from the storage unit (transaction log storage unit 22). Extract logs.
At this time, when the server identification information is included in the status confirmation request, only the log extraction unit 23 of the analysis server 20 identified by the server identification information extracts the transaction log. When the server identification information is included in the status confirmation request, the status confirmation request is transmitted only to the analysis server 20 identified by the server identification information in the first place, and the status confirmation request is not transmitted to the other analysis servers 20. It is good as well.

  S13: Subsequently, the control unit (log extraction unit 23) of the analysis server 20 transmits the extracted transaction log to the aggregation server 30.

  S14: Subsequently, the control unit (log aggregation means 32) of the aggregation server 30 aggregates the received transaction logs and extracts the contents of the processing performed by each analysis server 20. Thereafter, the log aggregation process ends.

  As described above, the data analysis unit 11 according to the present embodiment can specify the contents of the processing executed by each of the plurality of analysis servers 20 on the crawl data identified by the URL via the transaction log. As a result, the administrator can confirm the state of processing sequentially performed by the plurality of analysis servers 20. In particular, since the log aggregation processing is performed in parallel with the main processing, it is possible to check the status of the processing sequentially performed by the plurality of analysis servers 20 without stopping the flow of the main processing.

  At this time, the log extracting unit 23 extracts the transaction log only when the status confirmation request is received from the log requesting unit 31 and extracts the transaction log independently of the main processing unit 21. It is possible to check the status of the processes executed by each of the plurality of analysis servers 20 without affecting the 11 main processes.

  In addition, by including server identification information for identifying each of the analysis servers 20 in the status confirmation request, only the processing contents of the analysis server 20 that is presumed to have performed the cause of the processing when an abnormality occurs are confirmed. It is possible to quickly identify the cause while limiting the processing load by limiting the number of analysis servers 20 that perform new processing as the entire system.

(Deformation)
As mentioned above, although embodiment of this invention was described, this invention is not restricted to embodiment mentioned above. The effects described in the embodiments of the present invention are only the most preferable effects resulting from the present invention, and the effects of the present invention are limited to those described in the embodiments of the present invention. is not.

  In the above embodiment, the log extracting unit 23 extracts a transaction log when a status confirmation request is received. However, the present invention is not limited to this, and as shown in FIG. It is good also as extracting at the extraction timing. A modified form of log aggregation processing A will be described with reference to FIG.

  S21: The control unit (log extraction unit 23) of the analysis server 20 determines whether it is the extraction timing. The extraction timing can be arbitrarily set, and may be a regular interval such as 5 minutes. If it is the extraction timing in S21, the process proceeds to S22. If it is not the extraction timing, the process of S21 is repeated until the extraction timing is reached.

  S22: Subsequently, the control unit (log extraction unit 23) of the analysis server 20 extracts a transaction log from the storage unit (transaction log storage unit 22).

  S23: Subsequently, the control unit (log extracting unit 23) of the analysis server 20 transmits the extracted transaction log to the aggregation server 30.

  S24: Subsequently, the control unit (log aggregating unit 32) of the aggregation server 30 converts the received transaction log into the URL associated with the transaction log and the server identification information that identifies the type of the received analysis server 20. Associate and aggregate.

  As a result, by confirming the transaction log aggregated in the aggregation server 30 using the URLs specified in the processing sequentially performed by the plurality of analysis servers 20 as a key, the processing status for the specified URLs can be confirmed. Can be monitored.

  In FIG. 6, the timing at which the log extraction unit 23 extracts the transaction log is set to an arbitrary timing, but the timing at which the log extraction unit 23 transmits the transaction log to the aggregation server 30 may be set to an arbitrary timing. . Specifically, each time a transaction log is stored in the transaction log storage unit 22, the log extraction unit 23 extracts the transaction log, and thereafter, the transaction is sent to the aggregation server 30 at a predetermined cycle (for example, every 5 minutes). A log may be transmitted.

DESCRIPTION OF SYMBOLS 1 Search engine 10 Web crawler 11 Data analysis part 12 Indexer 13 Search server 20 Analysis server 21 Main processing means 22 Transaction log storage means 23 Log extraction means 30 Aggregation server 31 Log request means 32 Log aggregation means

Claims (4)

A web crawler that circulates web pages and collects crawl data, a data analysis unit that creates data used for a search index from the crawl data collected by the web crawler, and data created by the data analysis unit A search engine comprising: an indexer that creates a search index, and a search server that responds to a search request from a user using the search index created by the indexer;
The data analysis unit includes a plurality of servers that sequentially process the crawl data to create data used for the search index, and an aggregation server that aggregates the processing results of the plurality of servers,
Each of the plurality of servers is
Transaction log storage means for storing the transaction log associated with the process in association with the URL of the Web page corresponding to the crawl data each time the process is executed;
A log extraction unit that extracts a transaction log associated with the URL from the transaction log storage unit and sends the transaction log to the aggregation server in response to receiving a status confirmation request that specifies the URL;
With
The aggregation server is a search engine comprising log aggregation means for aggregating a plurality of received transaction logs. The status confirmation request includes designation of server identification information for identifying the plurality of servers in addition to designation of the URL ,
The search according to claim 1, wherein the log extraction unit extracts a transaction log associated with a key included in the state confirmation request from a transaction log storage unit provided in a server identified by the server identification information . Engine . A web crawler that circulates web pages and collects crawl data, a data analysis unit that creates data used for a search index from the crawl data collected by the web crawler, and data created by the data analysis unit A search engine comprising: an indexer that creates a search index, and a search server that responds to a search request from a user using the search index created by the indexer;
The data analysis unit includes a plurality of servers that sequentially process the crawl data to create data used for the search index, and an aggregation server that aggregates the processing results of the plurality of servers,
Each of the plurality of servers is
Transaction log storage means for storing the transaction log associated with the process in association with the URL of the Web page corresponding to the crawl data each time the process is executed;
Log extraction means for extracting a transaction log associated with a specific URL from transaction logs stored in each of the transaction log storage means at a predetermined cycle, and sending the log to the aggregation server;
With
A search engine comprising log aggregation means, wherein the aggregation server aggregates the extracted transaction logs in association with the specific URL and server identification information for identifying the plurality of servers. A web crawler circulates web pages and collects crawl data, a data analysis unit creates data used for a search index from the crawl data collected by the web crawler, and an indexer creates the data by the data analysis unit. A search method for creating a search index based on the data and responding to a search request from a user using the search index created by the indexer by a search server,
The data analysis unit sequentially processes the crawl data in each of a plurality of servers to create data used for the search index, and aggregates the processing results of the plurality of servers in an aggregation server,
In each of the plurality of servers,
Each time the transaction log storage means executes a process, the transaction log associated with the process is stored in association with the URL of the web page corresponding to the crawl data;
In response to receiving a status confirmation request designating the URL, the log extraction means extracts a transaction log associated with the URL from the transaction log storage means, and sends the transaction log to the aggregation server;
And a log aggregating unit in the aggregation server aggregating the received plurality of transaction logs .

Images