JP5084857B2 - Analysis processing adjustment apparatus and method - Google Patents

Description

  The present invention relates to an analysis processing adjustment apparatus and method.

  Conventionally, a user obtains a Web page on which information he wants is posted by inputting a keyword that hits the desired information from a search Web page where a search keyword or the like can be input. When a keyword is entered in a search Web page and the start of processing is started, the keyword is transmitted from the user terminal to the index server. Then, the index server searches the index DB (DB: database) stored in the server, and in order from the Web page having the highest score for the keyword, the heading of the Web page and the snippet that is the text including the keyword are searched. Output.

  For example, as described in Patent Document 1, such an index server acquires a Web page by a crawling robot called a crawler, and extracts an URL or a keyword of the Web page to construct an index DB. .

  The score is mechanically determined based on predetermined program logic. In general, an index server is a mechanism that outputs a higher score as a score is higher. The level of a score is an important factor in a corporate strategy, for example. Therefore, a technique for increasing the score by using a SEO (Search Engine Optimization search engine optimization) countermeasure or the like is used.

International Publication No. 2002/044946

  The construction of the index DB is normally performed uniformly by a predetermined program and does not allow exception processing. However, what is given an unnatural score has a need to be managed and updated manually, for example.

  SUMMARY OF THE INVENTION An object of the present invention is to provide an analysis processing adjustment apparatus and method that allow an administrator to manually manage and update index construction without changing a normal analysis processing program. .

  The present inventor has found that the needs of the administrator are satisfied by providing a mechanism for modifying some data, and has completed the present invention. Specifically, the present invention provides the following.

  (1) Archive data receiving means for receiving archive data that is the contents of a Web page obtained by a crawler or a feeder, and an input target for extracting a URL of the Web page based on the archive data and storing an input target list The input storage stored by the input target list storage means, the list storage means, and the analysis data to be reflected in the index DB used in the search of the Web page, obtained as a result of executing the analysis processing on the archive data Analysis means added to the target list, archive data storage means for storing the archive data received by the archive data reception means, and archive data stored in the archive data storage means and related to a predetermined URL of the Web page A data correction And archive data correction means, the analyzing process adjusting device and a archive data transfer means for transferring the archive data receiving means corrected archived data by said archive data correction means.

  According to such a configuration of the present invention, the analysis processing adjustment apparatus associates the analysis data obtained by executing the analysis processing on the archive data with the URL of the Web page extracted based on the archive data. Then, an input target list used for reflecting in the index DB is created. Then, the analysis processing adjustment device delivers the modified archive data obtained by modifying the archive data to the archive data receiving unit. Therefore, the analysis processing adjustment device corrects the archive data and delivers it to the archive data receiving means, so that the input target list can be forcibly changed halfway by correcting the archive data. As a result, the index used during the search created using the input target list reflects the user's intention. In addition, since the program logic of the analysis process is not changed, confusion such as a change in the input target list other than the Web page corresponding to the modified archive data can be avoided.

  (2) The method according to (1), further comprising analysis data correction means for correcting the analysis data related to the input target list added by the analysis means based on the archive data stored in the archive data storage means. Analysis processing adjustment device.

  According to such a configuration of the present invention, the analysis processing adjustment device corrects the analysis data related to the input target list based on the archive data, so that the input target list can be changed more directly by correcting the analysis data. As a result, the analysis processing adjustment apparatus can adjust the input target list more directly.

  (3) The analysis data correction means corrects the analysis data related to the input target list added by the analysis means by an input from a management terminal connected to the analysis processing adjustment apparatus via a communication network. The analytical processing adjustment device according to (2).

  According to such a configuration of the present invention, the user can correct the analysis data related to the input target list of the analysis processing adjustment device from the management terminal connected via the communication network. The user can change the input target list more directly.

  (4) The archive data correction means corrects the archive data stored in the archive data storage means by an input from a management terminal connected to the analysis processing adjustment apparatus via a communication network. (1) The analytical processing adjustment apparatus according to any one of (3).

  According to such a configuration of the present invention, the user can correct the archive data by referring to the input target list of the analysis processing adjustment device from the management terminal connected via the communication network. The correction of the archive data reflected in the input target list by the user is accepted, and the user can forcibly change the input target list halfway.

  (5) The analysis unit sequentially executes the analysis process on the archive data through a plurality of different processes, and adds the obtained plurality of analysis data to the input target list. The analytical processing adjustment device according to any one of (4) to (4).

  According to such a configuration of the present invention, in the analysis processing adjustment device, the analysis unit sequentially executes the analysis processing on the archive data through a plurality of different processes, and the obtained plurality of analysis data is input to the input target list. Therefore, a plurality of analysis results can be used by executing analysis processing from a plurality of viewpoints.

  (6) A reception step of receiving archive data that is content of a Web page obtained by a crawler or a feeder, and an input target list storage that extracts the URL of the Web page based on the archive data and stores an input target list And an addition for adding analysis data, which is obtained as a result of executing analysis processing on the archive data, to be reflected in the index DB used for Web page search to the stored input target list Archive data related to a predetermined URL of the Web page, stored in the archive data storage step by an input from a management terminal connected via a communication network, and an archive data storage step for storing the archive data Fix And the input target list storage step, the addition step, and the archive data storage step, using the archive data after the correction corrected by the correction step as the archive data until there is no input from the management terminal. And an iterative step for repeating the correction step.

  According to the present invention, it is possible to provide an analysis processing adjustment apparatus and method that allow an administrator to manually manage and update index construction without changing a normal analysis processing program. .

It is a figure which shows the whole structure of the search system which concerns on this embodiment, and the function structure of an analysis process adjustment apparatus. It is a flowchart of the analysis processing adjustment apparatus which concerns on this embodiment. It is a figure explaining the outline | summary of the data which concern on this embodiment.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. This is merely an example, and the technical scope of the present invention is not limited to this.

(Embodiment)
[Overall Configuration of Search System 100 and Functional Configuration of Analysis Processing Adjustment Apparatus 1]
FIG. 1 is a diagram showing the overall configuration of the search system 100 and the functional configuration of the analysis processing adjustment apparatus 1 according to the present embodiment.

  The search system 100 includes an analysis processing adjustment device 1, a plurality of Web servers 3, an index server 4, a management terminal 5, and a communication network 9.

  The analysis processing adjustment apparatus 1 is a server that creates data (analysis data) that builds an index of an index DB (DB: database) by reflecting on the index server 4. The analysis processing adjustment device 1 includes a control unit 10 and a storage unit 20.

  The control unit 10 includes an archive data reception unit 11, a metadata creation unit 12, a URL list storage control unit 13, an archive data analysis unit 14, an archive data modification unit 15, an archive data delivery unit 16, and analysis data. And a correction unit 17.

  The archive data receiving unit 11 has a function as an archive data receiving unit, and is a control unit that receives archive data that is content of a Web page transmitted by executing a program called a crawler or a feeder. A crawler is a program that collects Web pages stored in any Web server 3 in the world to create an index DB. The feeder refers to a program for inputting a Web page that does not need to be collected by a crawler, such as the Web server 3 operated by itself or the Web server 3 operated by a partner. The crawler, the feeder, and the web page input by the feeder may be included in the storage unit 20 of the analysis processing adjustment apparatus 1, but are not illustrated connected to the analysis processing adjustment apparatus 1 via the communication network 9. It is desirable that other servers such as a collection server have.

  The metadata creation unit 12 divides the archive data into a plurality of morphemes by, for example, morpheme analysis, and is an index based on the appearance frequency of words for each morpheme, for example, TF (Term Frequency) IDF ( Word weighting is performed using Inverse Document Frequency). By doing so, the metadata creating unit 12 is a control unit that creates metadata (attribute data) indicating the characteristics of the archive data from the archive data. Note that the metadata creation unit 12 may create metadata by extracting the contents of the meta tag from the archive data.

  The URL list storage control unit 13 generates a URL list (input target list) in which a URL (Uniform Resource Locator) indicating a Web page that is the basis of archive data is associated with the metadata created by the metadata creation unit 12. A control unit that creates and stores the URL list in the URL list storage unit 21. A URL indicating a Web page can be extracted based on archive data.

  The archive data analysis unit 14 has a function as an archive data analysis unit, and is a control unit that executes an analysis process on the archive data and adds the analysis data obtained as a result to the URL list. Specifically, the analysis process includes tagging to increase the search rank for a predetermined search keyword, or setting a flag for preventing registration in the index DB by spam determination. It may have a plurality of analysis processes depending on the content (process) of the process.

  The archive data correction unit 15 has a function as an archive data correction unit, and corrects the archive data stored in the archive data storage unit 22 by the operation from the management terminal 5 to update the archive data after the correction. It is a control unit.

  The archive data delivery unit 16 has a function as an archive data delivery unit, and is a control unit that delivers the modified archive data modified by the archive data modification unit 15 to the archive data reception unit 11.

  The analysis data correction unit 17 is a control unit that corrects the analysis data analyzed by the archive data analysis unit 14 and added to the URL list by an operation from the management terminal 5, and updates the analysis data to the corrected analysis data.

  The storage unit 20 includes a URL list storage unit 21 and an archive data storage unit 22.

  The URL list storage unit 21 has a function as an input target list storage unit, and is a storage area for storing a URL list in which URLs are associated with metadata. The URL list is associated with analysis data in addition to the URL and metadata. The URL list is transmitted to the index server 4 and used as data for constructing the index DB.

  The archive data storage unit 22 has a function as archive data storage means, and is a storage area for storing the archive data itself. When the archive data stored in the archive data storage unit 22 is corrected by the processing of the archive data correction unit 15, the content thereof is different from the content of the original Web page.

  The hardware of the analysis processing adjustment apparatus 1 of this embodiment may be configured by a general computer. For example, a general computer includes a central processing unit (CPU) as the control unit 10 and a memory (RAM, ROM), a hard disk (HDD), and an optical disk (CD, DVD, etc.) as a storage unit 20 in a network. As a communication device, various wired and wireless LAN devices, as a display device, for example, various displays such as a liquid crystal display and a plasma display, and as an input device, for example, a keyboard and a pointing device (mouse, tracking ball, etc.) are appropriately provided. These are connected by a bus line. In such a general computer, the CPU controls the analysis processing adjustment apparatus 1 in an integrated manner, reads and executes various programs as appropriate, and cooperates with the hardware described above, thereby providing various functions according to the present invention. Is realized.

  The Web server 3 is a server that includes a storage unit that stores a plurality of Web pages and a control unit that controls the entire Web server 3. The hardware of the Web server 3 may be configured by a general computer. A group of pages that are a plurality of web pages stored in the web server 3 and are under a specific domain or host are collectively referred to as a web site. The Web page of the Web site stored in the Web server 3 is data that is the source of archive data.

  The index server 4 is a server including a storage unit that stores an index DB and a control unit that controls the entire index server 4. The condition for the analysis processing adjustment apparatus 1 to transmit the URL list to the index server 4 may be, for example, after the analysis processing adjustment apparatus 1 receives archive data and executes a series of processes, or weekly These periodic timings may be set in the analysis processing adjustment apparatus 1. Further, the condition for the analysis processing adjustment apparatus 1 to transmit the URL list to the index server 4 may be a case where a transmission request is received from the index server 4. The hardware of the index server 4 may be configured by a general computer.

  The management terminal 5 is, for example, a personal computer (PC) used by the administrator. The management terminal 5 has a communication function that can be connected to the analysis processing adjustment device 1 via the communication network 9, and an operation unit and a display that are used by the user to correct data stored in the storage unit 20 of the analysis processing adjustment device 1. Any device may be used as long as it has a terminal.

  The communication network 9 is a communication line such as the Internet for performing communication among the analysis processing adjustment device 1, the Web server 3, the index server 4, and the management terminal 5. The communication network 9 may be wired or part or all of it may be wireless.

[Processing of analysis processing adjustment apparatus 1]
Next, processing in the analysis processing adjustment apparatus 1 will be described. FIG. 2 is a flowchart of the analysis processing adjustment apparatus 1 according to the present embodiment. FIG. 3 is a diagram illustrating an outline of data according to the present embodiment.

  S1: Control unit 10 (archive data receiving unit 11) receives archive data obtained by a crawler and a feeder. A server that stores programs of crawlers and feeders and that is connected via the communication network 9 acquires and acquires archive data at a predetermined timing, for example, daily or whenever a web page is updated. The archive data thus transmitted is transmitted to the analysis processing adjustment apparatus 1.

  S2: The control unit 10 (metadata creation unit 12) performs archive data analysis processing. In the archive data analysis process, metadata is created from archive data by, for example, morphological analysis.

  S3: The control unit 10 (URL list storage control unit 13) creates a URL list in which the URL of the Web page from which archive data is created is associated with the metadata created from the archive data that is the content of the Web page. And stored in the URL list storage unit 21.

  As shown in FIG. 3, the archive data 30 and the URL 31 indicating the Web page that is the original data of the archive data 30 are associated with each other. Then, the metadata 33 is created from the archive data 30 by the analysis by the metadata creating unit 12 (FIG. 3 (1)). The created metadata 33 and URL 31 are associated with each other by the control unit 10 to create a URL list 34.

  Returning to FIG. 2, S4: The control unit 10 (archive data analysis unit 14) performs an archive data analysis process. In the archive data analysis process, various analysis processes are sequentially performed on the archive data, and as a result, a plurality of analysis data is obtained.

  S5: The control unit 10 (archive data analysis unit 14) adds the analysis data obtained in S4 to the URL list.

  As shown in FIG. 3, by analyzing the archive data 30, a plurality of analysis data 35 is created from the archive data 30 (FIG. 3 (2)). The plurality of created analysis data 35 is added to the URL list 34 in which the metadata 33 and the URL 31 are associated with each other.

  As described above, in the analysis processing adjustment apparatus 1, the archive data analysis unit 14 sequentially analyzes the archive data via a plurality of different processes, and adds the obtained plurality of analysis data to the URL list. Therefore, a plurality of analysis results can be used by executing analysis processing from a plurality of viewpoints.

  Returning to FIG. 2, S6: The control unit 10 stores the archive data received in S1 in the archive data storage unit 22.

  S7: The control unit 10 determines whether an archive data correction request has been received from the management terminal 5. When the archive data correction request is received (S7: YES), the control unit 10 moves the process to S8. On the other hand, when the archive data correction request is not received (S7: NO), the control unit 10 moves the process to S10.

  S8: The control unit 10 (archive data correction unit 15) performs archive data correction processing. Specifically, the control unit 10 refers to the archive data storage unit 22 based on the archive data modification request received from the management terminal 5 and modifies the archive data. The archive data from the management terminal 5 can be modified by modifying the archive data directly from an archive data modification screen (not shown) of the analysis processing adjustment apparatus 1 or by the control unit 10 in response to a request from the management terminal 5. The designated archive data is transmitted to the management terminal 5, and the modified archive data received from the management terminal 5 is received by the control unit 10, whereby the archive data stored in the archive data storage unit 22 is converted into the modified archive data. You may update to data.

  S9: The control unit 10 (archive data delivery unit 16) delivers the modified archive data to the archive data reception unit 11. Thereafter, the control unit 10 moves the process to S1.

  As shown in FIG. 3, by correcting the archive data 30, the control unit 10 creates post-correction archive data 38 from the archive data 30 (FIG. 3 (3)). The created modified archive data 38 is overwritten on the archive data 30 (FIG. 3 (4)).

  Returning to FIG. 2, S <b> 10: The control unit 10 determines whether an analysis data correction request is received from the management terminal 5. When the analysis data correction request is received (S10: YES), the control unit 10 moves the process to S11. On the other hand, when the analysis data correction request has not been received (S10: NO), the control unit 10 ends this process.

  S11: The control unit 10 (analysis data correction unit 17) performs analysis data correction processing. Specifically, the control unit 10 refers to the URL list storage unit 21 from the management terminal 5 and corrects the analysis data of the URL list. The analysis data from the management terminal 5 may be modified by directly modifying the analysis data from an analysis data modification screen (not shown) included in the analysis processing adjustment apparatus 1, or by the control unit 10 according to a request from the management terminal 5. The record of the specified URL list is transmitted to the management terminal 5, the URL list data including the corrected analysis data received from the management terminal 5 is received, and the corresponding record is updated with respect to the URL list before the correction. May be. Then, the control part 10 complete | finishes this process.

  As shown in FIG. 3, by correcting the analysis data 35, the control unit 10 creates corrected analysis data 36 from the analysis data 35 (FIG. 3 (5)). The created modified analysis data 36 is overwritten on the analysis data 35.

  As described above, the analysis processing adjustment apparatus 1 directly corrects the archive data while referring to the analysis result to the archive data receiving unit 11 for the information that is not successfully added only by the normal archive data analysis processing. By passing the corrected archive data, appropriate information can be added as analysis data without changing the analysis processing program itself. Therefore, the search system 100 can generate an index using the analysis data and use the index for the search.

  Further, the analysis processing adjustment apparatus 1 can directly reflect the data without performing a process such as executing the logic again by directly correcting the analysis data, so that the load on the system can be reduced. Further, the analysis processing adjustment apparatus 1 can directly add appropriate information as analysis data for generating an index.

(Deformation)
In the present embodiment, the URL list is stored in the URL list storage unit, but the present invention is not limited to this. For example, the URL list may be stored in another storage area connected to the analysis processing adjustment device via a communication network, or data may be retained on a program executed as the analysis processing adjustment device. Good.

  As mentioned above, although embodiment of this invention was described, this invention is not restricted to embodiment mentioned above. The effects described in the embodiments of the present invention are only the most preferable effects resulting from the present invention, and the effects of the present invention are limited to those described in the embodiments of the present invention. is not.

DESCRIPTION OF SYMBOLS 1 Analysis processing adjustment apparatus 3 Web server 4 Index server 5 Management terminal 10 Control part 11 Archive data receiving part 12 Metadata production part 13 URL list storage control part 14 Archive data analysis part 15 Archive data correction part 16 Archive data delivery part 17 Analysis Data correction unit 20 Storage unit 21 URL list storage unit 22 Archive data storage unit 30 Archive data 34 URL list 35 Analysis data 36 Analysis data after correction 38 Archive data 100 after correction 100 Search system

Claims (6)

Archive data receiving means for receiving archive data that is the contents of a web page obtained by a crawler or a feeder;
Input target list storage means for extracting the URL of the Web page based on the archive data and storing the input target list;
Analysis data to be reflected in the index DB used in the search of the Web page obtained as a result of executing the analysis processing on the archive data is stored in the input target list stored in the input target list storage unit. Additional analytical means;
Archive data storage means for storing the archive data received by the archive data receiving means;
Archive data correction means stored in the archive data storage means for correcting archive data relating to a predetermined URL of the Web page;
Archive data delivery means for delivering archive data modified by the archive data modification means to the archive data reception means;
An analysis processing adjustment apparatus comprising: Analysis data correction means for correcting the analysis data related to the input target list added by the analysis means based on the archive data stored in the archive data storage means;
The analysis processing adjustment apparatus according to claim 1. The analysis data correction means corrects the analysis data related to the input target list added by the analysis means by input from a management terminal connected to the analysis processing adjustment device via a communication network.
The analysis processing adjustment apparatus according to claim 2. The archive data correction means corrects the archive data stored in the archive data storage means by an input from a management terminal connected to the analysis processing adjustment apparatus via a communication network.
The analysis processing adjustment apparatus according to any one of claims 1 to 3. The analysis means executes the analysis processing in order through the archive data through a plurality of different processes, and adds the obtained plurality of analysis data to the input target list.
The analysis processing adjustment apparatus according to any one of claims 1 to 4. A receiving step of receiving archive data that is a content of a web page obtained by a crawler or a feeder;
An input target list storage step for extracting the URL of the web page based on the archive data and storing the input target list;
An addition step of adding analysis data, which is obtained as a result of executing the analysis processing on the archive data, to be reflected in the index DB used for Web page search to the stored input target list;
An archive data storage step for storing the archive data;
A correction step of correcting archive data related to a predetermined URL of the Web page stored by the archive data storage step by an input from a management terminal connected via a communication network;
Until the input from the management terminal ceases, the archive data after the correction in the correction step is used as the archive data, the input target list storage step, the addition step, the archive data storage step, and the correction step. A repeating step that repeats
Analytical processing adjustment method including

Images