JP2003303198A - Knowledge information collecting method and knowledge information collecting system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To select a collection mode having no collection by preparing an environment setting mode for an error ('404' error) having no content, even when a conventional content revives under an environment where contents are frequently re-collected for occurrence of the burdensome '404' error. <P>SOLUTION: A Web collecting module 111 checks whether 'not found = '1'' or 'not found = '1'' is described in an environment setting file 206 when the '404' error is described in a 'status code' column of a result file 204. When 'not found = '1'' is described (YES in step S293), the module again tries to collect URL where the '404' error occurs previously. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a knowledge information collecting system and a knowledge information collecting method used in a knowledge management system, and particularly to a knowledge information collecting method and knowledge for collecting document information to be registered in a knowledge database from a network. Information collection system.

[0002]

2. Description of the Related Art In recent years, the introduction of groupware for sharing information among a plurality of users has been promoted mainly in companies. E-mail systems and workflow systems are known as typical groupware,
Recently, a knowledge management system to support sharing of knowledge information is also being developed.

This knowledge management system is for accumulating and managing knowledge information such as personal know-how in a knowledge database. By combining it with a search function such as natural language search, the accumulated knowledge information can be efficiently stored. It is possible to make full use of it.

By the way, in such a knowledge management system, how to efficiently collect and store knowledge information is an important point. In particular, since Web information on the Internet is a treasure trove of knowledge, a mechanism for efficiently collecting necessary information from the Internet is needed.

[0005]

Recently, however, not only text-based document files such as HTML documents but also document files of various file formats created by various application programs have begun to be used for homepages. In this case, if the document file acquired from the Internet is directly registered in the knowledge database, it becomes difficult to search the document file in the knowledge database, and the knowledge cannot be effectively used.

The present invention has been made in consideration of such circumstances, and prepares an environment setting mode for an error (“404” error) in which content does not exist at the time of collection, and a file format is available on the Internet. It is an object of the present invention to provide a knowledge information collecting method and a knowledge information collecting system capable of efficiently collecting various types of document files of different types as knowledge information.

[0007]

In order to solve the above-mentioned problems, the present invention refers to the information set in the environment setting file and the information recorded in the result file, and a document to be registered in the knowledge database. In the knowledge information collecting method for collecting information from the network, in the environment setting file, information specifying whether or not to collect again when an error that content does not exist is set, and in the result file, For each of the starting URL and the link to be traced when the document information is collected, the collection status including the occurrence of an error that the content does not exist is described, and when the document information is collected, the starting URL and the link to be traced are respectively described. , Refer to the result file, and when the occurrence of an error that the content does not exist is described in the file, Serial to a preferences file, to the file, when the information for specifying the re-collection process has been set, and executes the acquisition process.

Further, according to the present invention, in a knowledge information collecting system for collecting document information to be registered in a knowledge database from a network, information for designating execution / non-execution of recollection when an error occurs in which a content does not exist is provided. For each of the set environment setting file, the starting point URL and the link to follow when collecting the document information,
For each of the result file in which the status information of the collection including the occurrence of an error that the content does not exist is described, and the starting point URL and the link to follow when collecting the document information, refer to the result file, and the content When the occurrence of an error that does not exist is described, refer to the environment setting file, and in the file,
When the information designating the recollection process is set, a collecting means for executing the collection process of the corresponding URL is provided.

In this knowledge information collecting method knowledge and knowledge information collecting system, in the environment setting file, information (not found) for specifying recollection when an error (“404” error) in which the content does not exist is generated. =
By setting 1), the U
Attempts are made to collect the RL document information again, and even contents that have been lost once can be collected if they exist at the time of the recollection. By providing an environment setting mode for such a "404" error, notfou
By setting nd = 1, it is possible to reduce the frequency of occurrence of annoying "404" errors, and in an environment where the above-mentioned recollection is frequently performed. , Not found = 0, it is possible to select a collection mode in which the contents are not collected even if the contents are restored as before, and thus it is possible to flexibly deal with the occurrence of the “404” error.

[0010]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows the overall configuration of a knowledge management system using a knowledge information collecting system according to an embodiment of the present invention. This knowledge management system is for providing services such as collection, analysis, and search of knowledge information, and includes a Web collection system 11, a registration module 12, and a knowledge engine 13.
Etc. These Web collection system 1
1, the registration module 12, and the knowledge engine 13 are realized as a program executed on the server computer. The web collection system 11 and the registration module 12 constitute a knowledge information system for collecting knowledge information used in the knowledge management system. This knowledge information collecting system collects documents such as Web pages from the Internet / Intranet 30 and registers the contents of the documents in the knowledge database (knowledge DB) 131 of the knowledge management system. Web collection system 11
Collection module 111, management interface 112,
And a registration directory 113.

The Web collection module 111 is a program for collecting various document files on the Internet / Intranet 30 and outputting them in a format that can be registered in the knowledge DB 131. This Web collection module 111 has a multi-thread structure, and HTTP
Document file acquisition processing from the Internet / Intranet 30 by (HyperText Transfer Protocol) can be performed in parallel. HTTP is an information site on the Internet / Intranet 30 for each Web
From server 31 to HTML (Hyper Text Markup Languag
It is a communication protocol for acquiring the document described in e). The web collection module 111 uses the URL (Uniform Reload) specified by the setting information that specifies the web collection conditions.
Source Locator) document file is acquired from the Internet / Intranet 30, and if the acquired document file contains the URL of the link destination, the document file of that URL is further acquired. , The related document files are sequentially acquired from the Internet / Intranet 30. In this case, it is necessary to analyze the acquired document file in order to detect the link information (URL) to another document file. In the analysis process, not only the detection of the link information but also the knowledge Extraction of text data to be registered in the DB 131 is also executed.

The document file to be collected is created not only by a hypertext file (HTML file) capable of describing a URL that is link information to another file, but also by a text file (plane text) and various application programs. Document files in various file formats (for example, Portable Docu
ment Format files, document files created with word processor software, spreadsheet files, presentation data files created with presentation data creation software, etc.) Become.

The Web collection module 111 acquires the attribute information to be registered in the knowledge DB 131 and extracts the above-mentioned text data for each document file collected from the Internet / Intranet 30. The attribute information acquired from each document file is, for example, the URL of the document file, file creation date / time (update date / time), and the like. Then, the Web collection module 111 creates a registration file in which the attribute information of each of the document files and the text data are described, and outputs it to the registration directory 113. Here, the registration file is
It means a file having a predetermined description format as a common interface for registering document information of each of a plurality of types of document files having different file formats in the knowledge DB 131. This registration file is used to register the document information of each of a plurality of types of document files in the knowledge DB 131 in a common format. In this embodiment,
XML (eXtensible Markup Langua) as a registration file
ge) is used.

The management interface 112 is a program for setting the contents of each Web collection process (hereinafter referred to as a Web collection process task) to be executed by the Web collection module 111, and the Web collection conditions for each Web collection process task. Of the Web collection module 111, and the functions of managing and presenting the collection status of each Web collection processing task. This management interface 112 allows the web server 22 to execute CGI (Common) so that an administrator user can perform necessary operations on the web browser 21 of the user terminal.
It is realized as an external program (CGI program) that can be started through the Gateway Interface.

The Web collection system 11 has a Web
As a file for managing and controlling the operation of the collection module 111, a lock file 201,
A control file 202, a log file 203, a result file 204, a setting file 205 and an environment setting file 206 are provided.

The lock file 201 is a file for exclusive control for preventing double activation of the Web collection module 111. The control file 202 is a file used by the management interface 112 to stop the Web collection module 111. For example, in the case of interrupting a Web collection processing task that is being executed in response to an instruction from an administrator user. Used for. The administrator user can specify the interruption during the collection execution through the management interface 112.

The log file 203 is a file for managing the success or failure of acquisition of a document file by the Web collection module 111 and the type of error.

The result file 204 is a file for managing a list of document files collected as a result of Web collection processing for each Web collection task, and here, a list of URLs acquired as a result of Web collection. ,
The number of acquired document files and the like are output for each Web collection processing task. This result file 204 is a web
It is used for the purpose of presenting the collection status of each collection processing task to the user and for the purpose of improving the efficiency of recollection. By using this result file 204, the Internet / Intranet 3 can be used among the document files collected in the past.
It is also possible to detect a document file deleted from the information site on 0 or a restart point when the Web collection process is interrupted midway. In this embodiment,
The status code column of this result file 204 (see FIG.
When a "404" error occurs, refer to (4)
04 "error) is described.

The setting file 205 is a file for describing setting information for designating the Web collection condition for each Web collection processing task, and here, a plurality of setting information corresponding to each of the plurality of Web collection processing tasks. Can be held. The web collection condition of each web collection processing task is set by the administrator user. In the Web collection condition, at least the origin address information (origin URL) indicating the location of the document file to be the origin of the information collection from the Internet / Intranet 30 and the collection range are specified only within the domain, or the URL for collection is specified. Includes multiple types of collection range information, such as specifying the pattern of URLs that are not collected, specifying the maximum number of document files or links that should be collected, and specifying the file format to be collected. Has been. This Web
We of the Web collection module 111 based on the collection condition
b The collection operation is controlled. That is, the Web collection module 111 sequentially collects the document file group related to the document file specified by the starting point URL within the range defined by the collection range information. The plurality of types of collection range information described in the setting file 205 and the setting means will be described later with reference to the GUI screen.

The Web collection condition also includes a knowledge database name that specifies the knowledge database of the registration destination. That is, in the knowledge DB 131, a plurality of knowledge databases having different types of accumulated documents are defined, and the plurality of knowledge databases are managed by respective knowledge database names. By designating the knowledge database name of the registration destination for each Web collection task by the setting information of the setting file 205 described above, the collected document information can be registered in an arbitrary knowledge database in the knowledge DB 131.

Further, as the collection range information, collection conditions for recollection can be set. Here, recollection means, for example, the same W as the Web collection module 111.
The eb collection processing is the second and subsequent Web collection processing when the eb collection processing task is periodically executed, and the Web collection processing when the once interrupted Web collection processing task is restarted.

In the environment setting file 206, environment setting information for Web collection (information for specifying detailed operation of the Web collection module 111) is described. In this embodiment,
As the environment setting information that specifies execution / non-execution of recollection for the “404” error, “not found =
“0” ”or“ not found = “1” ”is described (see FIG. 16). "Not found =
When "0" is described, it is not collected even if the once lost content is restored. That is, as before, "40
4 ”No collection is performed when an error occurs. In addition, "n
When “ot found =“ 1 ”” is described, even contents that have been lost once are collected if they have been restored. That is, the URL for which the “404” error occurred last time is tried to collect again. This "not foun
Here, the recollection processing when d = “1” ”is“ 40
4 ”This is called error recollection processing. The contents described in the environment setting file 206 can be rewritten by a text editor.

The registration directory 113 is a knowledge DB 11
3 is a storage area for outputting the content of the document to be registered. The registration file 114 in which the attribute information and the text data extracted from each document file by the Web collection module 111 are described is output here.
The web collection module 111 has two processing modes, first and second, regarding extraction of text data.

In the first processing mode, analysis processing is performed on document files of all file formats, text data is extracted, and link information (URL) is detected (URL is detected only in HTML files. Target).
In the second processing mode, only the HTML file and the text file (PlainText) are analyzed to extract the text data and further the link information (URL) is detected (URL is detected only in the HTML file. ), For example, Portable Document
The analysis process is not performed on a document file (hereinafter, referred to as a content file) having another file format such as a format file.

When the second processing mode is used, HTML
For PlainText and PlainText, the text data and attribute information are described in the registration file 114 and output to the registration directory 113. On the other hand, Portab
For files in other formats, such as files in the le Document Format format, the content file 115 is output to the registration directory 113 as it is, and the registration file 114 describes the attribute information of the file and the path name of the content file 115. It
Note that in the second processing mode, it is important that the text data extraction is also executed in the analysis process for the file that requires the analysis process for detecting the link information. The text data may be extracted only for the HTML file.

In both of the first and second processing modes, the Web collection module 111 executes the Web being executed.
When the processing of the collection task is completed, the registration module 12
A registration request file is issued to request registration of the document from the registration module 12. In the registration request file,
The file name of the registration file and the knowledge database name of the registration destination are described.

The registration module 12 registers the attribute information and text data of each document file collected by the Web collection module 111 in the knowledge DB 131.
That is, the registration module 12 acquires the corresponding registration file 114 in response to the registration request from the Web collection module 111, and extracts the attribute information and text data described for each document file on the registration file 114. , The registration destination knowledge database of the knowledge DB 131 is registered. The registration module 12 also has a text data extraction function, and the registration file 11
When the path name of the content file 115 is described in 4, the registration module 12 extracts text data from the content file 115 specified by the path name and registers it in the corresponding registration destination knowledge database of the knowledge DB 131. .

Further, the registration module 12 also executes a registration process of the documents collected by the collection modules other than the Web collection module 111. Other collection modules include, for example, a file collection module that collects document information from a file server of an electronic filing system, an RDB collection module that collects document information managed by the RDB database server, and an electronic bulletin board format community. There are a community collection module that collects the document information posted therein, a user collection module that the user uses to convert the document files of various application programs into the output format of the registration file (XML), and the like. The registration module 12 is commonly used by each of these collection modules. Registration module 1 for all collection modules
The above-mentioned XML-formatted registration file is used as an interface for 2.

The knowledge engine 13 performs a knowledge analysis process for utilizing the information accumulated in the knowledge DB 131. In this knowledge analysis process, a process of analyzing the features of each of a large amount of document information accumulated in the knowledge DB 131 to extract important words, a clustering process of classifying and organizing features into a plurality of categories, and the like are performed. This knowledge engine 13 includes a knowledge engine 132 for performing a natural language search.
Is provided for each user, and each user
By accessing the knowledge engine 132 from the browser 41 through the web server 42, the web browser 4
The knowledge accumulated in the knowledge DB 131 can be searched from above. Since the attribute information of each document stored in the knowledge DB 131 also includes the URL of the document, the original document is acquired from the corresponding information site and we
b It can be displayed on the browser 41. The search process can be performed by selecting a certain knowledge database name and targeting only the selected knowledge database, or by targeting all knowledge databases in the knowledge DB 131.

Next, the functional configuration of the Web collection module 111 will be described with reference to FIG. The web collection module 111 includes a collection control unit 301,
It has an attribute extraction unit 302, a text extraction unit 303, and a format conversion unit 304. Collection control unit 301
Is for each of all the setting information held in the setting file 205, the Web specified by the setting information.
Run collection tasks one at a time. A group of related document files is sequentially collected from the Internet / Intranet 30 starting from the starting point URL designated by the setting information for each Web collecting task. The URL to the linked document included in the acquired document file is the URL list 30
5, the collection processing is recursively executed while acquiring URLs from the URL list 305. URL
The result file 204 described above can be used as the list 305. The collection range is limited by the collection conditions held in the setting file 205.

Each collected document file is first sent to the attribute extraction unit 302, where the attribute information of each document file is acquired. Next, the text extraction unit 303 analyzes the document and extracts the text data to be registered in the knowledge DB 131 and the link destination URL to be acquired next. For example, for an HTML file, extraction of text data is performed by extracting a portion other than the HTML tag. The extracted text is divided into a title and a body (body). To obtain the link destination URL, the A tag, the HREF value of the AREF tag, FRAME
This is performed by acquiring the SRC value of the tag, the IFRAME tag, the LAYER tag, and the REFRESH value of the META tag. When operating in the second processing mode described above, the processing of the text extraction unit 303 is performed only on the HTML file and the text file, and Port
It is not performed for a document file of another file format such as a file of the Document Format format.

The attribute information and text data acquired from each document file are sent to the format conversion unit 304,
Then, it is formatted into an output format according to the description format of the XML registration file and is output on the registration directory 113. In one registration file 114, for example, 100
Attribute information and text data about 0 document files are described. The attribute information and path name of the document file for which text extraction has not been performed are described in the registration file 114.

Next, an example of the output format of the registration file 114 will be described with reference to FIGS. 3 and 4.

FIG. 3 shows an example of an output format when the text extraction unit 303 extracts text. Tag at the beginning of the file <? Xml version = "1.0" encod
ing = "Shift_JIS"?>, <KnowledgeSystem> indicates the beginning of the file, and the end tag </ KnowledgeSystem>
Indicates the end of the file.

Attribute information and text data of one document file is described in each record enclosed by <RECORD> and </ RECORD>. The meaning of the tag in each record is as follows.

MODE: Mode information for designating the operation mode of the registration module 12. This mode information instructs registration or deletion of document information (text data and attributes) in the knowledge DB 131 for each document file.
Either 2 = registration (overwrite) or 0 = delete. In case of deletion, only TYPE and UNIQUE of the following tags are output.

TYPE: Indicates the type of collection. In this example, it is always "Web collection". UNIQUE: A unique key for identifying the document file registered in the knowledge DB 131. Usually, the URL of the document is used as a unique key.

ORGDATE: Indicates the creation date (or update date) of the document file. TITLE: Indicates the title of the document file. The text data extracted from the title part of the HTML file becomes the title. No title is output for files other than HTML files. This title is output as the title of the document file on the search screen.

AUTHOR: Describe the host name (host address of URL) of the information site that owns the document file. DATE: Describe the date part of ORGDATE above. URL: URL of the document file. It has the same value as UNIQUE. BODY: The text data extracted from the document file is described.

FIG. 4 shows an example of an output format relating to a document file in which the text extraction unit 303 has not performed text extraction.

No text data is described in BODY.
In PATH1 in the area enclosed by BDYFILE></BDYFILE>, the path name to the content file 115 output on the registration directory 113 is described. DEL = 1 instructs the registration module 12 to delete the original file in the registration directory 113 after extracting the text data from the content file 115. We
When the b collection module 111 operates in the second processing mode, text data is described in BODY for HTML files and plain text, and BD for document files (content files) in other file formats.
The path name of the content file will be described in YFILE.

Next, referring to the flowchart of FIG.
Web collection module 1 for collected document files
A procedure of a series of processing executed in 11 will be described.

First, from the collected document files, the knowledge DB
Attribute information to be registered in 131 (URL, AUTHOR, ORGDATE,
DATE) is acquired (step S101). The attribute information can be acquired by using the value returned from the information site by HTTP, the value added in the collected document file, or the like. After that, in the second processing mode, the file type is determined based on the extension of the collected document file, and the file is an HTML file or a plain text file, or a file of other file format. Is determined (steps S102 and S103). The collected document file is H
If the file is a TML file or a plain text file (YES in step S103), the above-mentioned text extraction processing (text extraction and link URL detection in the case of an HTML file) is executed (step S10).
4) Then, the attribute information and the text data are described in the above-mentioned format on the registration file 114 (the text data is inserted into BODY) (step S105). On the other hand, if the file has a file format other than the HTML file or the plain text file (NO in step S103), the file is the registration directory 1
13 is output as it is (step S106), and then the attribute information and the path name of the file are registered file 1
The process described in 14 (the path name is described in BDYFILE) is performed (step S107).

If the document file of the target URL cannot be acquired from the Internet / Intranet 30, the content of the document file is already in the knowledge DB1.
Mode information (MOD
E) = 0 (delete) is described on the registration file 114.

Further, in the first processing mode, the processes of steps S102 and S103 are not performed, and the processes of steps S104 and S105 are executed for all the acquired files.

Next, referring to the flowchart of FIG.
The procedure of the registration process executed by the registration module 12 will be described.

The registration module 12, which has received the registration request from the Web collection module 111, retrieves the records one by one from the registration file (XML file) specified by the registration request file from the Web collection module 111, and Process. First, is the mode information in the processing target record MODE = 0 or MODE = 2
Is checked (steps S111 and S11).
2).

When MODE = 2, the registration module 12
Is the data item (TYPE, UNIQUE, ORGDATE, TITLE, AUTH) described in each tag according to each tag in the record.
The contents of OR, DATE, URL, and BODY) are registered in the registration destination knowledge database in the knowledge DB 131 specified by the registration request file (step S113). If the text data does not exist in the BODY tag, the text data will not be registered. Then, it is judged whether or not the path name is described in the tag of BDYFILE (step S114). If the path name is described (YES in step S114), the corresponding content file is acquired from the storage area specified by the path name (step S115), and the process of extracting text data from the content file is executed. Performed (step S1
16). Then, the extracted contents are registered in the registration destination knowledge database as text data of the corresponding document file (step S117).

When MODE = 0, the attribute information and text data of the registered document file designated by UNIQUE in the record are searched from the knowledge DB 131, and the registered content is deleted from the registration destination knowledge database (step S118).

FIG. 7 shows the relationship between the state of the document file (Web content) on the Internet / Intranet 30 and the registration / deletion process to be performed on the knowledge DB 131. In the present embodiment, the following processing is performed based on the policy of reflecting the state of the latest Web content on the Internet / Intranet 30 in the knowledge DB 131 as much as possible.

(1) The updated Web content is overwritten on the knowledge DB 131 (MODE = 2). Updated W
When registering the attribute information and text data of the eb content, the UNIQUE is the same as the UNIQUE of the pre-update Web content already registered in the knowledge DB 131. Therefore, MODE = 2 updates (overwrites) the attribute information and text data of the pre-update Web content already registered in the knowledge DB 131 with the updated Web content attribute information and text data.

(2) The added Web contents are additionally registered in the knowledge DB 131 (MODE = 2). When registering the attribute information and text data of the added Web content, the UNIQUE is not registered in the knowledge DB 131. Therefore, by setting MODE = 2, the attribute information and text data of the added Web contents are stored in the knowledge DB 13.
1 can be additionally registered.

(3) The deleted Web contents are also deleted from the knowledge DB 131 (MODE = 0).

Next, the functions provided by the management interface 112 will be specifically described. As described above, the management interface 112 sets the Web collection task settings /
It is a user interface for managing and executing,
1) A function of creating a plurality of settings of a web collection task and saving them in the setting file 204. 2) A list of collection settings,
The administrator user is provided with a function of performing management work such as deletion, 3) a function of starting and ending (suspending and resuming) collection, and 4) a function of confirming the status and achievement of collection in real time.

The screen displayed on the Web browser 21 of the administrator user by the management interface 112 is composed of three screens of "list screen", "setting screen" and "status screen" as shown in FIG. When the management interface 112 is activated, the list screen is first displayed.
The three screens transit according to the operation performed by the administrator user on each screen. The list screen displays a list of currently created settings and the latest collection result of each setting. On this list screen, for example, when a created setting is selected and the selected setting transitions to the setting screen or status screen related to the setting, addition / deletion of the setting content can be performed on the setting screen. It is possible to perform, and on the status screen, it is possible to specify activation and termination (suspension and resumption) of the setting. If a new setting is selected on the list screen, the setting screen is displayed, and a process for creating a new setting can be performed on the setting screen. The list screen, the setting screen, and the status screen will be specifically described below.

FIG. 9 shows an example of the list screen.
On this list screen, as shown in the figure, "List" anchor 5
01, “Setting anchor” 502, and “Status” anchor 503 are displayed. These “list” anchor 501, “setting anchor” 502, and “status” anchor 503 are links to the list screen, setting screen, and status screen, respectively. When the “list” anchor 501 is clicked on the list screen, the display content of the list screen is updated to the latest state. Further, on the list screen, the "new setting" field 50
4, "Create" button 505, and status list display section 50
6 is provided. Enter the setting name of the Web collection task to be newly created in the "New setting" field 504,
When the "Create" button 505 is clicked, it is possible to transition to the setting screen and newly create setting information regarding the setting name.

The status list display unit 506 displays a list of collection statuses for each of the Web collection tasks that have already been created and registered, based on the contents held in the result file 204 described above. The following items are displayed for each Web collection task.

"Setting name": Name of registered setting "Knowledge database name": Name of database to register collected data "Status": Current status of setting (uncollected / collecting / finished / interrupted / abnormal) Either end) "Start": Date and time of last start "End": Date and time of last end "File": Number of files collected by the previous task execution
(This time) and the maximum value of the total number of files collected so far (whole) As described above, in this embodiment, a plurality of Web collection tasks can be set and Multiple We
b The collection status of each collection task can be confirmed on the list screen.

Next, an example of the setting screen will be described with reference to FIG. On the setting screen, as shown in the figure, in addition to the “list” anchor 501, the “setting anchor” 502, the “status” anchor 503, a “setting name” field 601,
“Knowledge database name” field 602, “proxy server” setting form 603, “collection file format” setting form 604, “acquisition request” setting form 605,
An “authentication information” setting form 606 and a “collection site” setting form 607 that constitutes a collection site setting screen are provided, and the following setting editing operations can be performed using these.

(1) Setting name: "Setting name" field 6
In 01, the setting name designated by creating a new setting or selecting an existing setting on the list screen is displayed. (2) DB selection: By inputting the knowledge database name in the “knowledge database name” field 602, the registration destination knowledge database for registering the collected data is selected.

(3) Proxy: The proxy server used for collection is set using the "proxy server" setting form 603. (4) File format to collect: “Collecting file format”
A file format to be collected is selected on the setting form 604. HTML and TEXT files are selected by default. Other than this, for example, Portable
Document format file, document file created by word processor software,
Attachments of various general-purpose file formats such as files created by spreadsheet software and presentation data files created by presentation data creation software can be selected as collection targets.

(5) Acquisition request: The timeout time and the number of retries for URL acquisition are specified on the "acquisition request" setting form 605. (6) Authentication Information: On the “authentication information” setting form 606, the authentication information required when accessing a URL that requires user authentication is set.

(7) Collection site: The following items are set on the "collection site" setting form 607.

Setting of starting URL: The URL which is the starting point of collection is specified. In this embodiment, a maximum of 100 excluding half-width characters and special characters is set on the setting field of the origin URL.
You can enter up to 0 characters.

Setting the upper limit of the number of link stages: Designate the maximum number of link stages from the collection origin URL. -Upper limit setting of the number of files: Specify the maximum number of files to be collected. -Re-collection setting: Origin U for the same collection task to be set
Specify the behavior when recollecting from RL. There are three modes for recollection ("collect all files newly", "collect only uncollected files", "collect only updated files (with deletion function)", and select any one mode. Can be specified.

Collection range: When "collect only in domain" is specified, collection is performed only in the domain of the origin URL. The default is no specification. If you select "Collect only in domain" from the check box, the source URL set in the field for setting the source URL will be documented from the relevant information site on the Internet / Intranet 30 only within that domain. To collect. The collection process at this time will be described later.

URL pattern to be collected: Designate a character string pattern of URLs to be collected. Among the URLs to be collected, only URLs that include the specified character string pattern are collected.

URL pattern not to be collected: Designate the character string pattern of the URL that should be excluded from the collection target. Among the URLs to be collected, the URL including the specified character string pattern is not collected.

In each recollection mode, the "collect all files newly" mode is the U
It means a mode of recollecting all RL. That is, regardless of whether it has been acquired in the past, it is newly collected from the origin URL. The “collect only uncollected files” mode is a mode in which previously collected URLs are not collected.
The "collect only updated files (with deletion function)" mode is a mode in which only the document files that have been updated from the previous collection point are collected. The document whose update is to be checked is the URL collected in the previous collection more than the specified number of days before the current collection. The value entered in the "Update check frequency" field is the specified number of days. The default number of designated days is "1 day". If the URL collected in the previous collection more than the specified number of days before the present is deleted from the corresponding information site on the Internet / Intranet 30 and the document file cannot be acquired from the URL, The registration information corresponding to the document file is deleted from the knowledge DB 131.

FIG. 11 shows an example of the status screen. For example, selecting the “status” anchor 503 on the list screen or the setting screen causes a transition to the status screen of the currently displayed task. The "setting name", "knowledge database name", "state", etc. of the task are displayed on this status screen. Next to the “setting name”, a button 701 for instructing start / stop (restart / interrupt) of collection is displayed. In "Status", the status of the task (collecting / finished / not collected / suspended / abnormally terminated) is displayed, and further, the starting URL of the task, the number of link stages collected (current / maximum / upper limit), and the number of files The status information 702 such as (current / total / upper limit) is also displayed.

When the status of the task is not collected, finished, or suspended, the button 701 displays "collection started", as shown in FIG. "Start collection" button 70
When 1 is selected, the execution of the task is started by the web collection module 111. If the current state is ended or suspended, recollection will be started. If the current status of the task is being collected,
As shown in FIG. 12, the button 701 displays “collection interruption”. If you select the “Collect collection” button 701,
Collection tasks that are running are suspended. The list of URLs collected before the interruption is held in the results file 204.

Next, with reference to FIG. 13 to FIG. 15, the interruption and resumption (recollection) operation of the collection process by the button operation on the status screen will be specifically described.

The suspend / resume function is a function used when the Web collection processing in progress is temporarily stopped and restarted at a later date. W by Web collection module 111
The eb collection process is performed by the internet / intranet 30
Each time a document file is acquired from the URL, the URL, a status code indicating success or failure, and the like are written in the result file 204 as the progress status information. If the acquired document file has a URL of the link destination, the URL is added to the result file 204, and the process is repeated until there is no uncollected URL in the result file 204. If the "collection suspension" button 701 is pressed during this collection processing, the Web collection task in execution is suspended at that point. Then, later, when the "collection start" button 701 is pressed with the interrupted Web collection task selected, the interrupted Web collection task is restarted and recollection is started. For example, when the “collect only uncollected files” mode is selected as the recollection mode, the uncollected URs held in the result file 204 are collected.
The collection process will be restarted from L.

FIG. 14 shows an example of a URL list held as progress status information in the result file 204. As shown in the figure, in the result file 204, in addition to the “URL” of each URL to be acquired, “collection date”, “collection date and time”, “status code”, “starting URL”, and “number of link stages” are included. Will be recorded. "Number of links" is
Indicates the number of links in the acquisition target URL when viewed from the starting URL. For the uncollected URLs, the information of “collection date”, “collection date and time”, and “status code” is blank.
This makes it possible to detect the restart point when the Web collection process is interrupted midway. The "status code" indicates the success or failure of the collection process, which is an HTTP status code returned from the partner web server from which the web page is acquired. The type of this HTTP status code is "200" indicating that the file acquisition was successful (normal end), "403" indicating an error due to the access authority, and the corresponding file cannot be found (for the request to acquire the deleted document file). “Response”) includes an error “404”, a timeout error “408”, and the like. In the present embodiment, “40
When a "4" error occurs, "404" error is described in the "status code" column, and the description is referred to in the "404" error recollection process described later.

The flowchart of FIG. 15 shows a series of processing procedures from the start of Web collection to the interruption thereof.

The Web collection module 111 first
The document file specified by the origin URL is acquired (step S121), and the result (collection date, collection date and time, status code, origin URL, link stage number, URL (same as origin URL) is registered in the result file 204. The progress status information is updated (step S122) If the acquisition is successful, the knowledge DB 13 is acquired from the collected document files.
The attribute information to be registered in 1 is acquired (step S12
2) Next, text extraction processing (text extraction and detection of link destination URL in the case of HTML file) is executed (step S124). When the link destination URL is detected, the URL, the starting point URL, and the link stage number described above are additionally registered in the URL list of the result file 204 (step S125). After that, a process of describing the attribute information and the text data on the registration file is performed (step S126). Here, it is determined whether or not there is an interruption instruction (step S127). If there is no interruption instruction (NO in step S127), it is determined whether or not the uncollected URL is registered in the result file 204.
It is determined whether the collection process is completed (step S
128).

If uncollected URLs are registered in the result file 204 (NO in step S128), the processes from step S121 described above are repeated. That is, the document file specified by the URL added to the result file 204 is acquired (step S121),
The result (collection date, collection date and time, status code) is registered in the result file 204 to update the progress status information (step S122). If the acquisition is successful, the attribute information to be registered in the knowledge DB 131 is acquired from the collected document file (step S122), and then the text extraction process (text extraction and detection of link destination URL in the case of HTML file) is executed. (Step S12
4). If the linked URL is detected, its UR
L, the starting URL and the number of links are the result file 20
4 is additionally registered in the URL list (step S12).
5). After that, a process of writing the attribute information and the text data on the registration file 114 is performed (step S12).
6). Such Web collection processing is performed by the setting file 20.
It is recursively executed until the collection is completed within the collection range of the Web collection condition set to 5. When the "collection suspension" button 701 is pressed during this Web collection, the management interface 112 is accessed through the control file 202.
Issues an interruption instruction to the Web collection module 111 (YES in step S127), and the Web collection task being executed by the Web collection module 111 is interrupted.

FIG. 16 shows a description example of the part described in the environment setting file 206 according to the present embodiment.
In the environment setting file 206, as described above, W
eb collection environment setting information (Web collection module 111
Information that specifies the detailed operation of “not found” is described as environment setting information that specifies execution / non-execution of recollection for the “404” error in the present embodiment.
= “0” ”or“ not found = “1” ”is described. When "not found =" 0 "" is described, the collection mode (hereinafter referred to as the normal collection mode) will not be collected even if the content that has been lost once is restored, and the "404" error has occurred as before. No collection will be done. When “not found =“ 1 ”” is described, a collection mode (hereinafter “40
4) error recollection processing mode), and the URL for which the “404” error occurred last time is tried to collect again. This environment setting file 206 can be rewritten with a text editor,
For example, “not f
From “ound =“ 0 ”” to “not found =”
The change to "1" and the change from "not found =" 1 "" to "not found =" 0 "can be arbitrarily performed.

Here, the collection processing in the recollection processing mode of the above-mentioned “404” error will be described with reference to FIGS. 17 and 18.

FIG. 17 is a flowchart showing the processing procedure of the “404” error check executed by the Web collection module 111, and FIG. 18 is the above-mentioned “404”.
It is a flow chart which shows an error recollection processing procedure.

Upon receiving the collection request from the management interface 112, the Web collection module 111 starts collecting the origin URL and the URL of the linked link destination.
In this collection process, the "404" error check process shown in FIG. 17 is executed. In this process, first, the setting file 205 is referenced (step S281), the starting point URL is read (step S282), and the starting point URL is read.
To start collecting from. In this collection, "404"
When no error occurs (NO in step S283),
The collection process shown in FIG. 15 is executed (step S
284), if the "404" error does not occur for each of the origin URL and the link destination URL to be traced (NO in step S283), the collection processing described above is recursively executed (NO in step S256, link destination URL in S282). Read, NO in S283, S284, ..., S2
86 YES).

If a "404" error occurs during the above collection (YES in step S283),
“404” error ”is described in the“ status code ”column of the result file 204 shown in FIG. 14, and the registration file (XML) 1 for deletion is further stored in the registration directory 113.
14 is created (step S285).

Next, with reference to FIG. 18, the recollection processing of the "404" error will be described. In this process, when the document information (document file) is collected, the result file 20
The URLs to be collected are read out from step 4 (step S29
1), "404" in the corresponding "Status Code" column
It is checked whether or not an error is described (step S292 in the figure).

Here, in the "Status code" column, "4
If the "04" error is not described, the above-described collection processing is executed (step S294 in the figure). If the "404" error is not described in the "status code" column for each of the subsequent URLs, the process shown in FIG. 15 is performed. The collection processing shown is recursively executed (NO in step S295, S2).
91 link destination URL reading, S292 NO, S2
94, ..., YES in S295).

In the "Status code" column, "40
4 "error is described, environment setting file 2
06, "not found =" 1 "", "not found"
It is checked which of “found =“ 1 ”” is described (step S293).

Here, "not found =" 0 ""
When it is described (NO in step S293), it recognizes that it is in the setting state of the normal collection mode, which does not collect even if the lost content is restored, and collects for the “404” error as before. Do not do.

In addition, in the environment setting file 206, "no
When “tfound =“ 1 ”” is described (YES in step S293), even if the content is once lost, it is collected if it has been recovered, and the state of the “404” error recollection processing mode is set. Recognizing that there is a URL, an attempt is made again to collect the URL for which the "404" error occurred last time (step S294).

By preparing the environment setting mode for the "404" error as described above, the not found
By setting = 1, it is possible to reduce the frequency of occurrence of annoying "404" errors, and in an environment where the above-mentioned recollection is frequently performed, By setting not_found = 0, it is possible to select a collection mode in which content is not collected even when the content is restored as before, and thus it is possible to flexibly deal with the occurrence of the “404” error.

Since all the functions of the knowledge information collecting system of this embodiment are realized by a computer program, the computer program is stored in a computer-readable storage medium, and the computer program is stored through the storage medium. It is possible to easily obtain the same effect as that of the present embodiment only by installing the program in a normal computer that can be connected to a computer network and executing the program.

Further, the present invention is not limited to the above-described embodiment, and can be variously modified at the stage of implementation without departing from the spirit of the invention. Furthermore, the embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiment, the problem described in the section of the problem to be solved by the invention can be solved, and the effect described in the section of the effect of the invention can be solved. When the above is obtained, the configuration in which this constituent element is deleted can be extracted as the invention.

[0091]

As described above, according to the present invention,
By providing an environment setting mode for "404" error and setting an arbitrary mode (normal collection mode / recollection processing mode of "404" error) according to the usage pattern, usage environment, etc. It is possible to efficiently collect various types of document files of different types as knowledge information.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of a knowledge information collecting system according to an embodiment of the present invention.

FIG. 2 is an exemplary block diagram showing the functional configuration of a Web collection module provided in the knowledge information collection system of the same embodiment.

FIG. 3 is a diagram showing an example of an output format of a registration file used in the knowledge information collecting system of the embodiment.

FIG. 4 is an exemplary view showing another example of an output format of a registration file used in the knowledge information collecting system of the embodiment.

FIG. 5 is an exemplary flowchart showing a processing procedure of a Web collection module provided in the knowledge information collection system of the embodiment.

FIG. 6 is an exemplary flowchart showing a processing procedure of a registration module provided in the knowledge information collecting system of the embodiment.

FIG. 7: We in the knowledge information collecting system of the same embodiment
The figure which shows the relationship between b content and its registration processing.

FIG. 8 is a diagram for explaining three screens provided by the knowledge information collecting system of the embodiment and their transitions.

FIG. 9 is an exemplary view showing an example of a list screen used in the knowledge information collecting system of the embodiment.

FIG. 10 is an exemplary view showing an example of a setting screen used in the knowledge information collecting system of the embodiment.

FIG. 11 is an exemplary view showing an example of a situation screen used in the knowledge information collecting system of the embodiment.

FIG. 12 is an exemplary view showing an example of a situation screen used in the knowledge information collecting system of the embodiment.

FIG. 13 is an exemplary view for explaining a suspending / resuming operation of the Web collecting process in the knowledge information collecting system of the embodiment.

FIG. 14 is a diagram showing an example of contents of a result file provided in the knowledge information collecting system of the embodiment.

FIG. 15 is an exemplary flowchart showing a processing procedure from the start to the interruption of the Web collection processing in the knowledge information collection system of the embodiment.

FIG. 16 is an exemplary view showing a description content of an environment setting file in the knowledge information collecting system of the embodiment.

FIG. 17 is an exemplary flowchart showing a processing procedure of a “404” error check executed by the Web collecting module in the knowledge information collecting system of the embodiment.

FIG. 18 is an exemplary flowchart showing the procedure of recollecting the “404” error in the knowledge information collecting system of the embodiment.

[Explanation of symbols]

11 ... Web collection system 12 ... Registration module 13 ... Knowledge engine 30 ... Internet / Intranet 111 ... Web collection module 112 ... Management interface 113 ... Registration directory 131 ... Knowledge database 132 ... Search engine 201 ... Lock file 202 ... control file 203 ... log file 204 ... Result file 205 ... Setting file 206 ... Environment setting file 301 ... Collection control unit 302 ... Attribute extraction unit 303 ... Text extraction unit 304 ... Format conversion unit 607 ... "Collection site" setting form

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Koji Shioda             2-9 Suehiro-cho, Ome City, Tokyo Stock Market             Company Toshiba Ome Factory F term (reference) 5B075 ND03 NK44 QP01

Claims (2)

[Claims] 1. A knowledge information collecting method for collecting document information to be registered in a knowledge database from a network by referring to information set in an environment setting file and information recorded in a result file, wherein: In the environment setting file, information that specifies whether or not to recollect an error that does not exist in the content is set, and in the result file, for each of the starting point URL and the link to follow when collecting the document information. , Describing the collection status including the occurrence of an error that the content does not exist, and referring to the result file for each of the starting URL and the link to follow when collecting the document information,
When the occurrence of an error that the content does not exist is described in the file, the environment setting file is referenced, and when the information that specifies the recollection processing is set in the file, the collection processing of the corresponding URL A method for collecting knowledge information, characterized by executing. 2. In a knowledge information collecting system for collecting document information to be registered in a knowledge database from a network, an environment in which information designating execution / non-execution of recollection when an error occurs in which a content does not exist is set. A setting file, a result file in which collection status information including occurrence of an error that content does not exist is described for each of a starting URL and a link to follow when collecting the document information, and a result file when collecting the document information For each of the starting point URL and the link to be followed, refer to the result file, and when the occurrence of an error that the content does not exist is described in the file, refer to the environment setting file and perform the recollection process in the file. When the specified information is set, execute the collection processing of the corresponding URL A knowledge information collecting system comprising: a collecting means.