JP2003303196A - Knowledge information collecting system and knowledge information collecting method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To efficiently collect as knowledge information various types of document files having different file formats from the Internet. <P>SOLUTION: A Web collecting module 111 sequentially obtains the document files from the Internet/intranet 30 based on set information held by a setting file 203. The Web collecting module 111 performs analysis for detecting link information and extracting text data for each obtained document file, and the text data extracted from each document file is described on a registering file and outputted. Since the text data extracted from each document file is described on the registering file and outputted, a registering module 12 can efficiently register the document information in a knowledge DB 131 in a common format without considering difference between file formats of obtained document files. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a knowledge information collecting system and a knowledge information collecting method used in a knowledge management system, and particularly to a knowledge information collecting system and knowledge for collecting document information to be registered in a knowledge database from a network. Regarding information collection method.

[0002]

2. Description of the Related Art In recent years, the introduction of groupware for sharing information among a plurality of users has been promoted mainly in companies. E-mail systems and workflow systems are known as typical groupware,
Recently, a knowledge management system to support sharing of knowledge information is also being developed.

This knowledge management system is for accumulating and managing knowledge information such as personal know-how in a knowledge database. By combining it with a search function such as natural language search, the accumulated knowledge information can be efficiently stored. It is possible to make full use of it.

By the way, in such a knowledge management system, how to efficiently collect and store knowledge information is an important point. In particular, since Web information on the Internet is a treasure trove of knowledge, a mechanism for efficiently collecting necessary information from the Internet is needed.

[0005]

Recently, however, not only text-based document files such as HTML documents but also document files of various file formats created by various application programs have begun to be used for homepages. In this case, if the document file acquired from the Internet is directly registered in the knowledge database, it becomes difficult to search the document file in the knowledge database, and the knowledge cannot be effectively used.

The present invention has been made in consideration of such circumstances, and a knowledge information collecting system and a knowledge information collecting system capable of efficiently collecting various types of document files having different file formats from the Internet as knowledge information. The purpose is to provide a method for collecting knowledge information.

[0007]

In order to solve the above-mentioned problems, the present invention shows a predetermined information collecting condition in a knowledge information collecting system for collecting document information to be registered in a knowledge database from a network. A document collection unit that collects document files from the network based on setting information, and performs an analysis process for extracting text data and detecting link information for each document file collected from the network. Document collection means for collecting, from the network, a document file of a link destination designated by the detected link information when the link information is detected, and document information of each of a plurality of types of document files in the knowledge database. A registration file with a predetermined description format as a common interface for registration. Output means for describing and outputting the text data extracted from each of the collected document files, and text data of each document file described therein from the registration file and extracting the text data as the knowledge. A registration means for registering in a database is provided.

In this knowledge information collecting system, the document collecting means executes the collection of the document file from the network based on the setting information designating the collecting condition of the information from the network. In the document collection means,
A process of detecting link information from each document file and, when the link information is detected, collecting the document files of the link destination specified by the detected link information is recursively executed. In this case, since the document file must be analyzed to detect the link information, the document collecting unit not only detects the link information but also extracts the text data during the analysis process. As described above, the document collection unit performs the text data extraction and the link information detection in the analysis process of each document file, whereby the text data can be efficiently extracted. Then, the text data extracted from each document file is described in the registration file and output. Here, the registration file is an output file having a predetermined description format as a common interface for registering the document information of each of a plurality of types of document files in the knowledge database. By writing the text data extracted from each document file on the registration file and outputting the text data, the registration means does not consider the difference in the file format of the document files acquired from the network, and the document information of the document files Can be efficiently registered in the knowledge database in a common format. Therefore,
Various types of document files having different file formats can be efficiently collected as knowledge information from the Internet.

Further, the document collection means can perform an analysis process for extracting text data and detecting link information for each of all document files. Configured so that only the document files of a specified file format including hypertext files that may be included are targeted, and text files of other file formats that do not include link information are extracted by the registration means. You can also do it.
In this case, the registration means is provided with means for extracting text data from the document file specified by the path name described in the registration file and registering it in the knowledge database, and the output means is the document collection means. It is preferable to provide a means for describing in the registration file the path name of the document file for which the analysis processing has not been performed in step 2.

Further, in the registration file, mode information for instructing the registration means to register or delete the text data in the knowledge database for each document file is defined. Means for managing a list of collected document files, status information indicating success or failure of collection of each document file by the document collecting means, and a list of collected files based on the list of collected files Means for detecting a document file deleted from the network in
A means for describing mode information indicating deletion of text data of the detected document file on the registration file so that the text data of the document file deleted from the network is deleted from the knowledge database; By providing the above, it becomes possible to automatically delete a Web page on the Internet registered in the knowledge database from the knowledge database when the Web page is deleted.

Further, according to the present invention, in a knowledge information collecting system for collecting document information to be registered in a knowledge database from a network, a document file from the network is acquired based on setting information indicating a predetermined information collecting condition. A document collecting means for collecting attribute information from a document file collected from the network, and determining whether the document file is a document file of a predetermined file format including a hypertext file. Then, an analysis process for extracting text data and detecting link information for a document file of the predetermined file format, and a document specified by the detected link information when the link information is detected Document collecting means for recursively executing processing for collecting files from the network For a document file of the predetermined file format, a predetermined description format is used as a common interface for registering the attribute information and text data of the document file, and the document information of each of a plurality of types of document files in the knowledge database. An output unit that describes the attribute information and path name of a document file of a file format other than the predetermined file format described in the registered file, and outputs the document file in the registered file. A registration means for extracting the attribute information and text data described for each document file and registering the attribute information and text data in the knowledge database, where the path name of the document file is described in the registration file. , Extract the text data from the document file specified by the path name. To characterized by comprising a registration means including means for registering in said knowledge database.

In this structure, the document collecting means extracts the attribute information of each document file, and further determines whether or not the document file is a document file of a predetermined file format including a hypertext file. A document collection unit performs an analysis process for extracting text data and detecting link information for a document file of a predetermined file format including a hypertext file. For document files of other file formats, the path name is described in the registration file, and the text data is extracted by the registration means. As a result, the text data extraction processing from each of the plurality of document files can be performed in a well-balanced manner by the document collection means and the registration means.

[0013]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows the overall configuration of a knowledge management system using a knowledge information collecting system according to an embodiment of the present invention. This knowledge management system is for providing services such as collection, analysis, and search of knowledge information, and includes a Web collection system 11, a registration module 12, and a knowledge engine 13.
Etc. These Web collection system 1
1, the registration module 12, and the knowledge engine 13 are realized as a program executed on the server computer. The web collection system 11 and the registration module 12 configure a knowledge information collection system for collecting knowledge information used in the knowledge management system. This knowledge information collecting system executes a process of collecting a document such as a Web page from the Internet / Intranet 30 and registering the content of the document in a knowledge database (knowledge DB) 131 of the knowledge management system. Web collection system 11 is We
b collection module 111, management interface 11
2 and the registration directory 113.

The Web collection module 111 is a program for collecting various document files on the Internet / Intranet 30 and outputting them in a format that can be registered in the knowledge DB 131. This Web collection module 11
1 has a multi-thread structure, and HTTP (Hype
Document file acquisition processing from the Internet / Intranet 30 by rText Transfer Protocol) can be performed in parallel. HTTP is used by each Web server 31, which is an information site (Web site) on the Internet / Intranet 30, to send HTML (Hyper Text Markup).
Language) is a communication protocol for obtaining a document described in Language. Web collection module 111
URL specified in the setting information that specifies the collection condition (Unif
(orm Resource Locator) document file is acquired from the Internet / Intranet 30, and if the acquired document file contains the URL of the link destination, the URL
By further performing the recursive processing of further acquiring the document file of, the related document file group is sequentially acquired from the Internet / Intranet 30. In this case, it is necessary to analyze the acquired document file in order to detect the link information (URL) to another document file. In the analysis process, not only the detection of the link information but also the knowledge Extraction of text data to be registered in the DB 131 is also executed.

The document file to be collected is created not only by a hypertext file (HTML file) in which a URL which is link information to another file can be described, but also by a text file (plane text) and various application programs. Document files of various file formats (for example, Portable Document Format files, document files created by word processing software, files created by spreadsheet software, presentation data files created by presentation data creation software, etc.) Be the target.

The Web collection module 111 acquires attribute information to be registered in the knowledge DB 131 and extracts the above-mentioned text data for each document file collected from the Internet / Intranet 30. The attribute information acquired from each document file is, for example, the URL of the document file, file creation date / time (update date / time), and the like. Then, the Web collection module 111 creates a registration file in which the attribute information of each of the document files and the text data are described, and outputs it to the registration directory 113. Here, the registration file is
It means a file having a predetermined description format as a common interface for registering document information of each of a plurality of types of document files having different file formats in the knowledge DB 131. This registration file is used to register the document information of each of a plurality of types of document files in the knowledge DB 131 in a common format. In this embodiment,
XML (eXtensible Markup Langua) as a registration file
ge) is used.

The management interface 112 is a program for setting the contents of each Web collection process (hereinafter referred to as a Web collection process task) to be executed by the Web collection module 111, and the Web collection conditions for each Web collection process task. Of the Web collection module 111, and the functions of managing and presenting the collection status of each Web collection processing task. This management interface 112 allows the web server 22 to execute CGI (Common) so that an administrator user can perform necessary operations on the web browser 21 of the user terminal.
It is realized as an external program (CGI program) that can be started through the Gateway Interface.

The Web collection system 11 has a Web
As a file for managing and controlling the operation of the collection module 111, a lock file 201,
A control file 202, a setting file 203, a result file 204, and a log file 205 are provided.

The lock file 201 is a file for exclusive control for preventing double activation of the Web collection module 111. The control file 202 is a file used by the management interface 112 to stop the Web collection module 111. For example, in the case of interrupting a Web collection processing task that is being executed in response to an instruction from an administrator user. Used for. The administrator user can specify the interruption during the collection execution through the management interface 112.

The setting file 203 is a file for describing setting information for designating the Web collecting condition for each Web collecting processing task, and here, a plurality of setting information corresponding to each of the plurality of Web collecting processing tasks is set. Can hold. The web collection condition of each web collection processing task is set by the administrator user. The Web collection conditions include at least the origin address information (origin URL) indicating the location of the document file that should be the origin of information collection from the Internet / Intranet 30, and the upper limit of the number of document files or the number of link stages to be collected. It includes the collection range information, etc. The web collection operation of the web collection module 111 is controlled based on the web collection conditions. That is, the Web collection module 111 sequentially collects the document file group related to the document file specified by the starting point URL within the range defined by the collection range information.

The Web collection condition also includes a knowledge database name that specifies the knowledge database of the registration destination. That is, in the knowledge DB 131, a plurality of knowledge databases having different types of accumulated documents are defined, and the plurality of knowledge databases are managed by respective knowledge database names. Configuration file 20
By designating the knowledge database name of the registration destination for each Web collection task by the setting information of 3, the collected document information can be registered in an arbitrary knowledge database in the knowledge DB 131.

Further, as the collection range information, it is possible to set the type of file format to be collected, the collection condition when recollecting, and the like. Here, recollection means
For example, it refers to the second and subsequent web collection processes when the same web collection processing task is periodically executed by the web collection module 111, and the web collection process when the interrupted web collection processing task is restarted. In this embodiment, three recollection modes are prepared.
b The recollection mode to be used can be specified for each collection processing task. The contents of each of the three recollection modes will be described in detail from FIG.

The result file 204 includes a list of previously collected document files as a result of the Web collection process for each W.
This is a file for managing each eb collection task, and here, a list of URLs acquired as a result of Web collection, acquisition date and time, the number of acquired document files, etc.
It is output for each eb collection processing task. The result file 204 is used for the purpose of presenting the collection status of each Web collection processing task to the user and for the purpose of improving the efficiency of recollection. This result file 204 is used to detect a document file deleted from an information site on the Internet / Intranet 30 among the document files that have been collected in the past, or a restart point when the Web collection process is interrupted midway. Can be detected, and further, the document file updated on the Internet / Intranet 30 from the time of the previous collection can be detected.

The log file 205 is a file for managing the success or failure of acquisition of a document file by the Web collection module 111 and the type of error.

The registration directory 113 is a knowledge DB 13
This is a storage area for outputting the document content to be registered in 1. The above-mentioned registration file in which the attribute information and the text data extracted from each document file by the Web collection module 111 are described is output here. Web
The collection module 111 has two first and second processing modes for extracting text data.

In the first processing mode, analysis processing is performed on document files of all file formats to extract text data, and link information (URL) is detected (URL is detected only in HTML files. Target).
In the second processing mode, only the HTML file and the text file (PlainText) are analyzed to extract the text data and further the link information (URL) is detected (URL is detected only in the HTML file. ), For example, the analysis process is not performed on a document file of another file format such as Portable Document Format (hereinafter referred to as a content file).

When the second processing mode is used, HTML
With regard to PlainText, the text data and attribute information are described in the registration file and output to the registration directory 113. On the other hand, Portable Document
For files of other formats such as Format, the content file is output as it is to the registration directory 113, and the attribute information of the file and the path name of the content file are described on the registration file. In addition, in the second processing mode, it is important to perform the extraction of the text data in the analysis process for the file that needs the analysis process for detecting the link information. You may make it extract text data only for a target.

In both of the first and second processing modes, the Web collection module 111 executes the Web being executed.
When the processing of the collection task is completed, the registration module 12
To request the registration module 12 to register the document. In the registration request file, the file name of the registration file, the knowledge database name of the registration destination, etc. are described.

The registration module 12 registers the attribute information and text data of each document file collected by the Web collection module 111 in the knowledge DB 131.
That is, the registration module 12 acquires the corresponding registration file in response to the registration request from the Web collection module 111, extracts the attribute information and text data described for each document file on the registration file, and acquires the knowledge. It is registered in the registration destination knowledge database of the DB 131. The registration module 12 also has a text data extraction function, and if the path name of the content file is described in the registration file, the registration module 12 will use the text from the content file specified by the path name. The data is extracted and registered in the relevant registration destination knowledge database of the knowledge DB 131.

Further, the registration module 12 also executes the registration process of the documents collected by the collection modules other than the Web collection module 111. Other collection modules include, for example, a file collection module that collects document information from a file server of an electronic filing system, an RDB collection module that collects document information managed by the RDB database server, and an electronic bulletin board format community. There are a community collection module that collects the document information posted therein, a user collection module that the user uses to convert the document files of various application programs into the output format of the registration file (XML), and the like. The registration module 12 is commonly used by each of these collection modules. Registration module 1 for all collection modules
The above-mentioned XML-formatted registration file is used as an interface for 2.

The knowledge engine 13 performs a knowledge analysis process for utilizing the information accumulated in the knowledge DB 131. In this knowledge analysis process, a process of analyzing a feature of each of a large amount of document information accumulated in the knowledge DB 131 and extracting an important word, a clustering process of classifying and organizing the document information into a plurality of categories according to the feature, and the like are performed. Be seen.
The knowledge engine 13 is provided with a knowledge engine 132 for performing a natural language search, and each user accesses the knowledge engine 132 from the web browser 41 of the user terminal through the web server 42.
The knowledge accumulated in the knowledge DB 131 can be searched from the Web browser 41. Since the attribute information of each document stored in the knowledge DB 131 also includes the URL of the document, the original document can be acquired from the corresponding information site and displayed on the Web browser 41. The search process can be performed by selecting a certain knowledge database name and targeting only the selected knowledge database, or by targeting all knowledge databases in the knowledge DB 131.

Next, the functional configuration of the Web collection module 111 will be described with reference to FIG. The web collection module 111 includes a collection control unit 301,
It has an attribute extraction unit 302, a text extraction unit 303, and a format conversion unit 304. Collection control unit 301
For each of all the setting information held in the setting file 203, the Web specified by the setting information
Run collection tasks one at a time. A group of related document files is sequentially collected from the Internet / Intranet 30 starting from the starting point URL designated by the setting information for each Web collecting task. The URL to the linked document included in the acquired document file is the URL list 30
5, the collection processing is recursively executed while acquiring URLs from the URL list 305. URL
The result file 204 described above can be used as the list 305. The collection range is limited by the collection conditions held in the setting file 203.

Each collected document file is first sent to the attribute extraction unit 302, where the attribute information of each document file is acquired. Next, the text extraction unit 303 analyzes the document and extracts the text data to be registered in the knowledge DB 131 and the link destination URL to be acquired next. For example, for an HTML file, extraction of text data is performed by extracting a portion other than the HTML tag. The extracted text is divided into a title and a body (body). The link destination URL is acquired by acquiring the HREF value of the A tag and the AREF tag, the SRC value of the FRAME tag, the IFRAME tag, the LAYER tag, and the REFRESH value of the META tag. When operating in the above-described second processing mode, the processing of the text extracting unit 303 is H
This is done only for TML files and text files, not for document files of other file formats such as Portable Document Format.

The attribute information and text data acquired from each document file are sent to the format conversion unit 304,
Then, it is formatted into an output format according to the description format of the XML registration file and is output on the registration directory 113. In one registration file, for example, attribute information and text data for about 1000 document files are described. The attribute information and path name of the document file for which the text extraction has not been performed are described in the registration file.

The above processing is similarly performed for each document file collected by recollection.

Next, an example of the output format of the registration file will be described with reference to FIGS. 3 and 4.

FIG. 3 shows an example of an output format when the text extraction unit 303 extracts text. Tag at the beginning of the file <? Xml version = "1.0" encod
ing = "Shift_JIS"?>, <KnowledgeSystem> indicates the beginning of the file, and the end tag </ KnowledgeSystem>
Indicates the end of the file.

Attribute information and text data of one document file are described in each record enclosed by <RECORD> and </ RECORD>. The meaning of the tag in each record is as follows.

MODE: Mode information for designating the operation mode of the registration module 12. This mode information instructs registration or deletion of document information (text data and attributes) in the knowledge DB 131 for each document file.
Either 2 = registration (overwrite) or 0 = delete. In case of deletion, only TYPE and UNIQUE of the following tags are output.

TYPE: Indicates the type of collection. In this example, it is always "Web collection". UNIQUE: A unique key for identifying the document file registered in the knowledge DB 131. Usually, the URL of the document is used as a unique key.

ORGDATE: Indicates the creation date (or update date) of the document file. TITLE: Indicates the title of the document file. The text data extracted from the title part of the HTML file becomes the title. No title is output for files other than HTML files. This title is used as the title of each document file displayed on the search screen.

AUTHOR: Describe the host name (URL host address) of the information site that owns the document file. DATE: Describe the date part of ORGDATE above. URL: URL of the document file. It has the same value as UNIQUE. BODY: The text data extracted from the document file is described.

FIG. 4 shows an example of an output format relating to a document file in which the text extraction unit 303 has not performed text extraction.

No text data is described in BODY.
In PATH1 in the area enclosed by BDYFILE></BDYFILE>, the path name to the content file output on the registration directory 113 is described. DEL = 1 instructs the registration module 12 to delete the original file in the registration directory 113 after extracting the text data from the content file. If the Web collection module 111 operates in the second processing mode, the HT
Text data is described in BODY for the ML file and plain text, and the path name of the content file is described in BDYFILE for document files (content files) of other file formats.

Next, referring to the flowchart of FIG.
Web collection module 1 for collected document files
A procedure of a series of processing executed in 11 will be described.

First, from the collected document files, the knowledge DB
Attribute information to be registered in 131 (URL, AUTHOR, ORGDATE,
DATE) is acquired (step S101). The attribute information can be acquired by using the value returned from the information site by HTTP, the value added in the collected document file, or the like. After that, in the second processing mode, the file type is determined based on the extension of the collected document file, and the file is an HTML file, a plain text file, or a file of another file format other than that. Is determined (steps S102 and S103). The collected document file is H
If the file is a TML file or a plain text file (YES in step S103), the above-mentioned text extraction processing (text extraction and link URL detection in the case of an HTML file) is executed (step S10).
4), and processing to describe the attribute information and text data in the above-mentioned format on the registration file (text data is BODY
Is inserted) (step S105). On the other hand, HT
If the file has a file format other than the ML file or the plain text file (step S103)
No), the file is directly output to the registration directory 113 (step S106), and then the attribute information and the path name of the file are described in the registration file (the path name is described in BDYFILE) ( Step S107).

If the document file of the target URL cannot be acquired from the Internet / Intranet 30, the content of the document file is already in the knowledge DB1.
Mode information (MOD
E) = 0 (delete) is described in the registration file.

Further, in the first processing mode, the processes of steps S102 and S103 are not performed, and the processes of steps S104 and S105 are executed for all the acquired files.

Next, referring to the flowchart of FIG.
The procedure of the registration process executed by the registration module 12 will be described.

Upon receiving the registration request from the Web collection module 111, the registration module 12 acquires the registration file (XML file) specified by the registration request file from the Web collection module 111 from the registration directory 113, and from the registration file. While fetching records one by one, the following processing is performed for each record. First, it is checked whether the mode information in the record to be processed is MODE = 0 or MODE = 2 (step S111,
S112).

When MODE = 2, the registration module 12
Is the data item (TYPE, UNIQUE, ORGDATE, TITLE, AUTH) described in each tag according to each tag in the record.
The contents of OR, DATE, URL, and BODY) are registered in the registration destination knowledge database in the knowledge DB 131 specified by the registration request file (step S113). If the text data does not exist in the BODY tag, the text data will not be registered. Then, it is judged whether or not the path name is described in the tag of BDYFILE (step S114). If the path name is described (YES in step S114), the corresponding content file is acquired from the storage area specified by the path name (step S115), and the process of extracting text data from the content file is executed. Performed (step S1
16). Then, the extracted contents are registered in the registration destination knowledge database as text data of the corresponding document file (step S117).

When MODE = 0, the attribute information and text data of the registered document file designated by UNIQUE in the record are searched from the knowledge DB 131, and the registered content is deleted from the registration destination knowledge database (step S118).

FIG. 7 shows the relationship between the state of the document file (Web content) on the Internet / Intranet 30 and the registration / deletion process to be performed on the knowledge DB 131. In the present embodiment, the following processing is performed based on the policy of reflecting the state of the latest Web content on the Internet / Intranet 30 in the knowledge DB 131 as much as possible.

(1) The updated Web content is overwritten on the knowledge DB 131 (MODE = 2). Updated W
When registering the attribute information and text data of the eb content, the UNIQUE is the same as the UNIQUE of the pre-update Web content already registered in the knowledge DB 131. Therefore, MODE = 2 updates (overwrites) the attribute information and text data of the pre-update Web content already registered in the knowledge DB 131 with the updated Web content attribute information and text data.

(2) The added Web contents are additionally registered in the knowledge DB 131 (MODE = 2). When registering the attribute information and text data of the added Web content, the UNIQUE is not registered in the knowledge DB 131. Therefore, by setting MODE = 2, the attribute information and text data of the added Web contents are stored in the knowledge DB 13.
You can additionally register to 1.

(3) The deleted Web contents are also deleted from the knowledge DB 131 (MODE = 0).

Next, the functions provided by the management interface 112 will be specifically described. As described above, the management interface 112 sets the Web collection task settings /
It is a user interface for managing and executing,
1) A function of creating a plurality of settings of a Web collection task and saving them in the setting file 204. 2) A function of performing management work such as displaying a list of collection settings and deleting. 3) Starting and ending collection (interruption and interruption). It provides the administrator user with the function of resuming 4) and the function of 4) checking the collection status and results in real time.

The screen displayed on the Web browser 21 of the administrator user by the management interface 112 is composed of three screens of "list screen", "setting screen" and "status screen" as shown in FIG. When the management interface 112 is activated, the list screen is first displayed.
The three screens transit according to the operation performed by the administrator user on each screen. The list screen displays a list of currently created settings and the latest collection result of each setting. On this list screen, for example, if you select a created setting and transition to the setting screen or status screen related to the selected setting, you can add / delete the setting contents on the setting screen. Also, on the status screen, it is possible to specify the start and end (suspend and restart) of the setting. Also, when a new setting creation is selected on the list screen, a transition is made to the setting screen, and processing for new setting creation can be performed on the setting screen. The list screen, the setting screen, and the status screen will be specifically described below.

FIG. 9 shows an example of the list screen.
On this list screen, as shown in the figure, "List" anchor 5
01, “Setting anchor” 502, and “Status” anchor 503 are displayed. These “list” anchor 501, “setting anchor” 502, and “status” anchor 503 are links to the list screen, setting screen, and status screen, respectively. When the “list” anchor 501 is clicked on the list screen, the display content of the list screen is updated to the latest state. Further, on the list screen, the "new setting" field 50
4, "Create" button 505, and status list display section 50
6 is provided. Enter the setting name of the Web collection task to be newly created in the "New setting" field 504,
When the “Create” button 505 is clicked, the setting screen is displayed and setting information regarding the setting name can be newly created.

The status list display unit 506 displays a list of collection statuses for each of the Web collection tasks that have already been created / registered, based on the contents held in the result file 204 described above. The following items are displayed for each Web collection task.

"Setting name": Name of registered setting "Knowledge database name": Database name for registering collected data "Status": Current status of setting (uncollected / collecting / finished / interrupted / abnormal) Either end) "Start": Date and time of last start "End": Date and time of last end "File": Number of files collected by the previous task execution
(This time) and the maximum value of the total number of files collected so far (whole) As described above, in this embodiment, a plurality of Web collection tasks can be set, and moreover, a plurality of Web collection tasks can be set. We
b Collection status of each collection task can be confirmed on the list screen.

Next, an example of the setting screen will be described with reference to FIG. On the setting screen, as shown in the figure, in addition to the “list” anchor 501, the “setting anchor” 502, the “status” anchor 503, a “setting name” field 601,
“Knowledge database name” field 602, “proxy server” setting form 603, “collection file format” setting form 604, “acquisition request” setting form 605,
An “authentication information” setting form 606, a “collection site” setting form 607, and the like are provided, and the following setting editing operations can be performed using these.

(1) Setting name: "Setting name" field 6
In 01, the setting name designated by creating a new setting or selecting an existing setting on the list screen is displayed. (2) DB selection: By inputting the knowledge database name in the “knowledge database name” field 602, the registration destination knowledge database for registering the collected data is selected.

(3) Proxy: The "proxy server" setting form 603 is used to set the proxy server used for collection. (4) File format to collect: “Collecting file format”
A file format to be collected is selected on the setting form 604. HTML and TEXT files are selected by default. In addition, Portable Document Format,
Various other file formats can be selected for collection.

(5) Acquisition request: Specify the timeout time and the number of retries for URL acquisition on the "acquisition request" setting form 605. (6) Authentication Information: On the “authentication information” setting form 606, the authentication information required when accessing a URL that requires user authentication is set.

(7) Collection site: The following items are set on the "collection site" setting form 607.

Setting of starting URL-Upper limit setting of the number of link stages: Designate the maximum number of link stages from the starting URL to be collected. -Upper limit setting of the number of files: Specify the maximum number of files to be collected. -Re-collection setting: Origin U for the same collection task to be set
Specify the behavior when recollecting from RL. There are three recollection modes for recollection (“collect all files newly”, “collect only uncollected files”, “collect updated files (with delete function)”)
Any one recollection mode can be specified for each collection task. The meaning of each recollection mode is as follows.

1) Recollection mode 1: "collect all files newly" mode means "collect all URLs newly" mode. That is, all the document files are newly collected from the origin URL regardless of whether they have been collected before.

2) Recollection mode 2: "collect only uncollected files""collect only uncollected files" mode means that the previously collected URLs are not collected including their link destinations and the starting point UR
This is a mode of collecting only the document files that have not been collected by the previous collection among all the document file groups from L.

3) Recollection mode 3: "collect only updated files (with delete function)""collect only updated files (include delete function)" mode only document files updated from the previous collection point It is a mode to collect. The document to be checked for the presence or absence of update is the URL collected in the previous collection more than the specified number of days before the current collection. The value entered in the "Update check frequency" field is the specified number of days. The default number of designated days is "1 day". In the previous collection, if the URL collected more than the specified number of days before the present is deleted from the corresponding information site on the Internet / Intranet 30 and the document file cannot be acquired from the URL, the document file The registration information corresponding to is deleted from the knowledge DB 131.

Collection range: When "collect only in domain" is specified, collection is performed only in the domain of the origin URL.

URL pattern to be collected: The pattern of the character string of the URL to be collected is specified. Among the URLs to be collected, only URLs that include the specified character string pattern are collected.

URL pattern not to be collected: Designate a character string pattern of the URL that should be excluded from the collection target. Among the URLs to be collected, the URL including the specified character string pattern is not collected.

FIG. 11 shows an example of the status screen. For example, selecting the “status” anchor 503 on the list screen or the setting screen causes a transition to the status screen of the currently displayed task. The "setting name", "knowledge database name", "state", etc. of the task are displayed on this status screen. Next to the “setting name”, a button 701 for instructing start / stop (restart / interrupt) of collection is displayed. In "Status", the status of the task (collecting / finished / not collected / suspended / abnormally terminated) is displayed, and further, the starting URL of the task, the number of link stages collected (current / maximum / upper limit), and the number of files The status information 702 such as (current / total / upper limit) is also displayed.

When the status of the task is uncollected, ended, or suspended, the button 701 displays "collection started", as shown in FIG. "Start collection" button 70
When 1 is selected, the execution of the task is started by the web collection module 111. If the current state is ended or suspended, recollection will be started. If the current status of the task is being collected,
As shown in FIG. 12, the button 701 displays “collection interruption”. If you select the “Collect collection” button 701,
Collection tasks that are running are suspended. The list of URLs collected before the interruption is held in the results file 204.

Next, with reference to FIGS. 13 to 15, the operation of suspending and resuming (re-collecting) the collection process by operating the buttons on the status screen will be specifically described.

The suspend / resume function is a function used when the Web collection processing in progress is temporarily stopped and restarted at a later date. W by Web collection module 111
The eb collection process is performed by the internet / intranet 30
Every time the document file is acquired from the URL, the URL, a status code indicating success or failure, and the like are written in the result file 204 as progress status (collection result) information. If the acquired document file has a URL of the link destination, the URL is added to the result file 204, and the process is repeated until there is no uncollected URL in the result file 204. If the "collection suspension" button 701 is pressed during this collection processing, the Web collection task in execution is suspended at that point. And later, the interrupted We
b Start Collect button 70 with the collect task selected
When 1 is pressed, the suspended Web collection task is restarted and recollection is started. For example, when the above-mentioned “collect only uncollected files” mode is selected as the recollection mode, the collection processing is restarted from the uncollected URL held in the result file 204.

FIG. 14 shows an example of a URL list held as progress status information in the result file 204. As shown in the figure, in the result file 204, in addition to the “URL” of each URL to be acquired, “collection date”, “collection date and time”, “status code”, “starting URL”, and “number of link stages” are included. Will be recorded. "Number of links" is
Indicates the number of links in the acquisition target URL when viewed from the starting URL. "Collection date" for uncollected URLs
The information of “collection date and time” and “status code” are blank. This makes it possible to detect the restart point when the Web collection process is interrupted midway. The “status code” indicates the success or failure of the collection process, and this is the HTTP status code returned from the partner web server from which the web page is acquired. This H
The types of TTP status codes are as follows.

200: Indicates normal termination, that is, file acquisition has succeeded. 304: Not updated (response to conditional acquisition request). In the "collect only updated files" mode, the URL for which this status code is returned is not acquired. 403: Indicates an error due to access authority. 404: Indicates an error due to the file not being found. It is a response to the acquisition request of the deleted document file. 408: Timeout Error The flowchart of FIG. 15 shows a series of processing procedures from the start of Web collection to the interruption thereof.

The Web collection module 111 first acquires the document file designated by the starting point URL (step S121), and as a result (collecting date, collecting date and time, status code, starting point URL, link stage number, URL (starting point U
The same as RL) is registered in the result file 204 to update the progress status information (step S122). If the acquisition is successful, the knowledge DB 131 is acquired from the collected document files.
The attribute information to be registered in is acquired (step S12
2) Next, text extraction processing (text extraction and detection of link destination URL in the case of HTML file) is executed (step S124). When the link destination URL is detected, the URL, the starting point URL, and the link stage number described above are additionally registered in the URL list of the result file 204 (step S125). After that, a process of describing the attribute information and the text data on the registration file is performed (step S126). Here, it is determined whether or not there is an interruption instruction (step S127). If there is no interruption instruction (NO in step S127), it is determined whether or not the uncollected URL is registered in the result file 204.
It is determined whether the collection process is completed (step S
128).

If uncollected URLs are registered in the result file 204 (NO in step S128), the processes from step S121 described above are repeated. That is, the document file specified by the URL added to the result file 204 is acquired (step S121),
The result (collection date, collection date and time, status code) is registered in the result file 204 to update the progress status information (step S122). If the acquisition is successful, the attribute information to be registered in the knowledge DB 131 is acquired from the collected document file (step S122), and then the text extraction process (text extraction and detection of link destination URL in the case of HTML file) is executed. (Step S12
4). If the linked URL is detected, its UR
L, the starting URL and the number of links are the result file 20
4 is additionally registered in the URL list (step S12).
5). After that, a process of describing the attribute information and the text data on the registration file is performed (step S126).

Here, it is determined whether or not there is an interruption instruction (step S127). If the “collection suspension” button 701 is pressed, the management interface 112 issues a suspension instruction to the web collection module 111 through the control file 202. In this case (Y in step S127)
ES), the Web collection task being executed is interrupted by the Web collection module 111.

Next, the operation of the recollection process will be described with reference to FIGS. 16 to 19. The recollection processing means the collection processing when the previously executed collection task is executed again as described above. The Web collection process when the suspended Web collection process task is restarted is a recollection process, and the Web collection process when the terminated Web collection process task is executed next is also a recollection process.
These recollection processes may be interrupted or terminated by the W
eb collection task is selected with the eb collection task selected
It is executed when the button 701 is pressed. Also on the Web
The Web collection module 111 can also be automatically started periodically by registering the collection module 111 with the task management function of the operating system. In this case, the interrupted or terminated Web collection task is automatically executed. The recollection process will be started.

As shown in the flowchart of FIG. 16, when the recollection process is started, first, the recollection mode relating to the Web collection task is determined based on the setting information of the setting file 203 (step S131). If the mode is “collect all files newly”,
b The contents of the previous result file 204 regarding the collection task are discarded, and the collection processing from the starting point URL is started (step S132). In this case, the same processing as the normal Web collection processing described above is executed. If the mode is “collect only uncollected files”, the already collected document files are not collected, but only the uncollected document files are collected.
In this case, the W stored in the result file 204
The restart point is detected from the contents of the previous collection result related to the eb collection task, and the UR that has not been collected in the previous collection process is detected.
Collection processing from L is started (step S133).
The collection result of the recollection process is added to the previous collection result regarding the Web collection task held in the result file 204. If the mode is "collect only updated files (with deletion function)" mode, only the document files updated from the previous collection point are collected (step S134). The collection result of the re-collection process is
It is added to the previous collection result regarding the Web collection task held in the result file 204.

The flowchart of FIG. 17 shows the procedure of recollection processing in the "collect only uncollected files" mode. First, an uncollected URL (the beginning of a URL whose collection date and collection date and time are not registered) is detected from the content of the previous collection result related to the Web collection task held in the result file 204, and it is used as the restart point. It is determined (step S201). Then, the document file specified by the URL of the restart point is acquired (step S202), and the result (collection date, collection date and time, status code) is registered in the result file 204, and the progress status information is updated ( Step S203).
If the acquisition is successful, the attribute information to be registered in the knowledge DB 131 is acquired from the collected document file (step S204), and then the text extraction processing (text extraction and detection of link destination URL in the case of HTML file) is executed. (Step S205). When the link destination URL is detected, the URL, the starting point URL, and the link stage number are additionally registered in the URL list of the result file 204 (step S206). After that, a process of describing the attribute information and the text data on the registration file is performed (step S207). Then, until the interruption instruction is issued or the collection processing is completed, the added URL is acquired from the result file 204 while the step S is performed.
The processing from 202 is repeatedly executed (step S2).
08, S209).

Next, the procedure of the recollection process in the "collect only updated files (with deletion function)" mode will be described.

In this mode, as shown in FIG. 18, assuming that the current date and time to start recollection is D1 and the time point that is a specified number of days before the current time is D2, the Web collection task acquired by D2. Among the URLs related to, only the URLs that have been updated so far are collected.

As shown in the flow chart of FIG. 19, first, the URLs previously collected are acquired one by one from the contents of the previous collection results regarding the Web collection task held in the result file 204 (step S3
01), the following processing is performed for each URL.
First, it is determined whether the document file corresponding to the URL is a document file collected by D2 (step S302). In the case of the document files collected up to D2, it is checked whether the document file specified by the URL has been updated from the previous collection date and time (step S303). If it has been updated, the document file of the URL is acquired (step S304). The processing in steps S303 and 304 is
For example, it is realized by using a mechanism in which a conditional acquisition request including a previous collection date and time is issued to the Web server specified by the URL, and the document file is returned to the Web server only when it is updated. You can

If the document file of the URL is updated and the acquisition is successful (step S30
5), the result (collection date, collection date and time, status code, origin URL, URL) of the result file 204
To update the progress information (step S30).
6). Then, from the collected document files, the knowledge DB 13
The attribute information to be registered in 1 is acquired (step S30
7) Next, text extraction processing (text extraction and detection of link destination URL in the case of HTML file) is executed (step S308). When the link destination URL is detected, the URL, the starting point URL, and the link stage number are additionally registered in the URL list of the result file 204 (step S309). After that, a process of describing the attribute information and the text data on the registration file is performed (step S310). Then, until the suspension instruction is issued or the collection process is completed, the URs previously collected are collected.
The process of step S301 is repeatedly executed for L (steps S311 and S312).

If a status code 404 is returned from the Web server in response to the acquisition request, that is, the UR concerned
If the L document file has been deleted from the Web server (NO in step S305), the result file 20
It is determined whether or not the document file was previously normally collected depending on whether or not the status code of the previous collection result of the URL held in No. 4 is 200 (step S313). If the document file has been normally collected previously, a deletion process for deleting the document information from the knowledge database 131 is executed (steps S314 and S310). Here, first, the result (collection date, collection date and time, status code 200, starting point UR
(L, URL) in the result file 204, then MODE
= 0 is selected, MODE = 0 and UNIQUE of the URL are described in the registration file. As a result, after the recollection is completed, the knowledge database 13 is registered by the registration module 12.
The corresponding document information is deleted from 1.

As described above, according to this embodiment, We
In the analysis processing for the b collection module 111 to detect the link information from the document file, the knowledge DB 131
By also extracting the text data to be registered in, it is possible to efficiently perform the extraction process of the text data. Further, in the second processing mode, the above-described analysis processing is performed only on the document file of the predetermined file format including the hypertext file that may include the link information, and the document files of other file formats are processed by the analysis processing. By using the mechanism in which the path name is described in the registration file, it is possible to automatically switch the processing mode according to the type of the acquired document file. Therefore, even when a plurality of collection tasks are set, the tasks can be efficiently processed in sequence.

Since all the functions of the knowledge information collecting system of this embodiment are realized by a computer program, the computer program is stored in a computer-readable storage medium, and the computer program is stored through the storage medium. It is possible to easily obtain the same effect as that of the present embodiment only by installing the program in a normal computer that can be connected to a computer network and executing the program.

Further, the present invention is not limited to the above-mentioned embodiment, and can be variously modified at the stage of implementation without departing from the gist thereof. Furthermore, the embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiment, the problem described in the section of the problem to be solved by the invention can be solved, and the effect described in the section of the effect of the invention can be solved. When the above is obtained, the configuration in which this constituent element is deleted can be extracted as the invention.

[0094]

As described above, according to the present invention,
Various types of document files having different file formats can be efficiently collected as knowledge information from the Internet.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of a knowledge information collecting system according to an embodiment of the present invention.

FIG. 2 is an exemplary block diagram showing the functional configuration of a Web collection module provided in the knowledge information collection system of the same embodiment.

FIG. 3 is a diagram showing an example of an output format of a registration file used in the knowledge information collecting system of the embodiment.

FIG. 4 is an exemplary view showing another example of an output format of a registration file used in the knowledge information collecting system of the embodiment.

FIG. 5 is an exemplary flowchart showing a processing procedure of a Web collection module provided in the knowledge information collection system of the embodiment.

FIG. 6 is an exemplary flowchart showing a processing procedure of a registration module provided in the knowledge information collecting system of the embodiment.

FIG. 7: We in the knowledge information collecting system of the same embodiment
The figure which shows the relationship between b content and its registration processing.

FIG. 8 is a diagram for explaining three screens provided by the knowledge information collecting system of the embodiment and their transitions.

FIG. 9 is an exemplary view showing an example of a list screen used in the knowledge information collecting system of the embodiment.

FIG. 10 is an exemplary view showing an example of a setting screen used in the knowledge information collecting system of the embodiment.

FIG. 11 is an exemplary view showing an example of a situation screen used in the knowledge information collecting system of the embodiment.

FIG. 12 is an exemplary view showing an example of a situation screen used in the knowledge information collecting system of the embodiment.

FIG. 13 is an exemplary view for explaining a suspending / resuming operation of the Web collecting process in the knowledge information collecting system of the embodiment.

FIG. 14 is a diagram showing an example of contents of a result file provided in the knowledge information collecting system of the embodiment.

FIG. 15 is an exemplary flowchart showing a processing procedure from the start to the interruption of the Web collection processing in the knowledge information collection system of the embodiment.

FIG. 16 is an exemplary flowchart showing a procedure of recollection processing in the knowledge information collecting system of the embodiment.

FIG. 17 is an exemplary flowchart showing the procedure of recollection processing in the “collect only uncollected files” mode in the knowledge information collecting system of the embodiment.

FIG. 18 is an exemplary view for explaining recollection processing in the “collect only updated files” mode in the knowledge information collection system of the embodiment.

FIG. 19 is an exemplary flowchart showing the procedure of recollection processing in the “collect only updated files” mode in the knowledge information collecting system of the embodiment.

[Explanation of symbols]

11 ... Web collection system 12 ... Registration module 13 ... Knowledge engine 30 ... Internet / Intranet 111 ... Web collection module 112 ... Management interface 113 ... Registration directory 131 ... Knowledge database 132 ... Search engine 201 ... Lock file 202 ... control file 203 ... Setting file 204 ... Result file 205 ... log file 301 ... Collection control unit 302 ... Attribute extraction unit 303 ... Text extraction unit 304 ... Format conversion unit

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) // G06F 13/00 520 G06F 13/00 520D (72) Inventor Koji Shioda 2 Suehirocho, Ome-shi, Tokyo No. 9 F-term in Toshiba Ome Factory, Ltd. (reference) 5B075 ND03 NK44 QP01 5B082 GC04

Claims (10)

[Claims] 1. A knowledge information collecting system for collecting document information to be registered in a knowledge database from a network, in which a document file is collected from the network based on setting information indicating a predetermined information collecting condition. Collecting means, which performs an analysis process for extracting text data and detecting link information on each document file collected from the network, and detects the link information when the link information is detected. Document collection means for collecting the document file of the link destination specified by the above from the network, and registration having a predetermined description format as a common interface for registering the document information of each of a plurality of types of document files in the knowledge database. On the file, extracted from each of the collected document files Output means for describing and outputting the text data, and registration means for extracting the text data of each document file described therein from the registration file and registering the text data in the knowledge database. Knowledge information collection system. 2. The registration means includes means for extracting text data from a document file specified by a path name described in the registration file and registering the text data in the knowledge database, and the document collecting means collects the collected text data. A first processing mode for performing an analysis process for extracting text data and detecting link information for each document file, and performing the analysis process for a document file of a predetermined file format including a hypertext file, And a second processing mode in which the analysis processing is not performed on a document file of another file format, and the output unit is configured to perform the analysis processing when the document collection unit operates in the second processing mode. 2. The knowledge information collecting system according to claim 1, further comprising means for describing a path name of a document file which has not been executed on the registration file. Stem. 3. The document collecting means includes means for acquiring attribute information from each document file collected from the network, and the output means outputs the text data and the attribute information for each document file. 2. The knowledge information collecting system according to claim 1, wherein the knowledge information collecting system is described and output on a registration file. 4. The registration file defines mode information for instructing the registration unit to register or delete text data in the knowledge database for each document file, and the output unit includes: A means for managing a list of document files collected in the past, status information indicating success or failure of collection of each document file by the document collection means, and a list of collected files based on the list of collected files Means for detecting a document file deleted from the network in the file, and the detected document file such that the text data of the document file deleted from the network is deleted from the knowledge database Means for describing mode information indicating deletion of text data on the registration file Knowledge information collection system according to claim 1, characterized in that it comprises a. 5. A unit for displaying a setting screen for creating the setting information on the user's terminal for each collection process of the document files to be executed by the document collection unit, and for each collection process on the setting screen. A means for holding the created plurality of setting information and a progress status of the collection processing by the document collecting means corresponding to each of the held plurality of setting information to the user terminal in response to a request from the user. The knowledge information collecting system according to claim 1, further comprising: a means for displaying a list. 6. The setting information includes starting point address information indicating a location of a document file to be a starting point for collecting information from the network, and a document to be collected from a plurality of types of collectable document files. And the information for designating the file format of the file, the document collection means is designated as the collection target in the document file group related to the document file designated by the origin address information as a starting point. The knowledge information collecting system according to claim 1, wherein a document file having a different file format is collected. 7. The knowledge database defines a plurality of databases having different types of stored documents, and the setting information includes a starting point indicating a location of a document file to be a starting point for collecting information from the network. Address information and a registration destination database name indicating a database in which the collected document files should be registered are included. The origin address information and the registration destination database are included in each collection process of the document files to be executed by the document collecting means. Further comprising means for displaying a setting screen for creating setting information including the name on the user's terminal, and means for holding a plurality of setting information created for each collection process on the setting screen. The knowledge information collecting system according to claim 1, which is characterized in that. 8. A knowledge information collecting system for collecting document information to be registered in a knowledge database from a network, in which a document file is collected from the network based on setting information indicating a predetermined information collection condition. Collecting means for acquiring attribute information from the document files collected from the network, and determining whether the document file is a document file of a predetermined file format including a hypertext file, Analysis processing for extracting text data and detecting link information for document files of the following file formats:
When the link information is detected, a document collection unit that recursively executes a process of collecting the document file specified by the detected link information from the network, and the document file of the predetermined file format, The attribute information and text data of the document file are described on a registration file having a predetermined description format as a common interface for registering the document information of each of a plurality of types of document files in the knowledge database, and the predetermined information is stored. For a document file of a file format other than the file format, output means for describing and outputting the attribute information and path name on the registration file, and attribute information described for each document file on the registration file, The text data is extracted and the attribute information and text data are extracted from the knowledge database. A registration means for registering a document file with a path name of a document file in the registration file, a means for extracting text data from the document file specified by the path name and registering the text data in the knowledge database. A knowledge information collecting system comprising: a registration means including the knowledge information collecting means. 9. A knowledge information collecting method for collecting document information to be registered in a knowledge database from a network, in which a document file is collected from the network based on setting information indicating a predetermined information collection condition. In the collecting step, analysis processing for extracting text data and detecting link information is performed on each document file collected from the network, and when the link information is detected, the detected link information is detected. A document collection step of collecting from the network the document files of the link destination specified by the above, text data extracted from each of the collected document files, and document information of each of a plurality of types of document files is registered in the knowledge database. A predetermined description format as a common interface for An output step of describing and outputting in one registration file, and a registration step of extracting text data of each document file described therein from the registration file and registering the text data in the knowledge database. Characteristic knowledge information collection method. 10. A knowledge information collecting method for collecting document information to be registered in a knowledge database from a network, wherein a document file is collected from the network based on setting information indicating a predetermined information collection condition. In the collecting step, the attribute information is acquired from the document files collected from the network, and it is determined whether the document file is a document file of a predetermined file format including a hypertext file. Analysis processing for performing extraction of text data and detection of link information for a document file of the file format described above, and when the link information is detected, the document file specified by the detected link information is used as the network. A document collection step of recursively executing the process of collecting from A document file having a predetermined file format is registered with a predetermined description format as a common interface for registering attribute information and text data of the document file, and document information of each of a plurality of types of document files in the knowledge database. An output step of writing on a file and writing and outputting the attribute information and path name of the document file of a file format other than the predetermined file format on the registered file; and each document on the registered file. A registration step of extracting the attribute information and text data described for each file and registering the attribute information and text data in the knowledge database, and when the path name of the document file is described in the registration file, Extract text data from the document file specified by the path name. Knowledge information collection method characterized by comprising a registration step includes the step of registering the knowledge database.