JP3725837B2 - Knowledge information collecting system and knowledge information collecting method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a knowledge information collection system and a knowledge information collection method used in a knowledge management system, and more particularly to a knowledge information collection system and a knowledge information collection method for collecting document information to be registered in a knowledge database from a network.
[0002]
[Prior art]
In recent years, introduction of groupware for sharing information among a plurality of users has been promoted mainly by companies. As typical groupware, an e-mail system, a workflow system, and the like are known, but recently, a knowledge management system for supporting sharing of knowledge information has begun to be developed.
[0003]
This knowledge management system is for accumulating and managing knowledge information such as personal know-how in a knowledge database. By combining it with a search function such as natural language search, it is possible to efficiently use the accumulated knowledge information. It becomes possible.
[0004]
By the way, in such a knowledge management system, how to efficiently collect and accumulate knowledge information is an important point. In particular, since Web information on the Internet is a treasure trove of knowledge, a mechanism for efficiently collecting necessary information from the Internet is required.
[0005]
[Problems to be solved by the invention]
However, recently, not only text-based document files such as HTML documents but also document files of various file formats created by various application programs have begun to be used for homepages. In this case, if the document file acquired from the Internet is directly registered in the knowledge database, it becomes difficult to search for the document file on the knowledge database, and the knowledge cannot be used effectively.
[0006]
The present invention has been made in consideration of such circumstances, and a knowledge information collection system and knowledge information collection capable of efficiently collecting various types of document files having different file formats as knowledge information from the Internet. It aims to provide a method.
[0007]
[Means for Solving the Problems]
In order to solve the above-described problems, the present invention provides a starting URL for collecting document information to be registered in the knowledge database in a knowledge information collecting system for collecting document information to be registered in the knowledge database from the network. And a setting screen display means for displaying a setting screen including setting items of the collection file format that can be selected from a plurality of types of document file formats that can be collected, and a setting displayed by the setting screen display means Setting file creation means for creating a setting file described based on the setting contents on the screen, and document information to be registered in the knowledge database based on the description contents of the setting file created by the setting file creation means Information collecting means for collecting from the network, and the information collecting means is recorded in the setting file. When the document information is acquired from the starting URL and the link destination to be traced, it is checked whether or not the document information matches the collected file format described in the setting file, and the collected file format described in the setting file is checked. And collecting only document information that matches the document information from the network , converting each collected document information into a predetermined file format for registration, and registering it in the knowledge database in a common format To do.
[0008]
In this knowledge information collection system, the collection of document files from the network is executed by the information collection means based on the setting information that specifies the conditions for collecting information from the network. The information collecting means checks whether or not the document information matches the collected file format described in the setting file when acquiring the document information from the starting URL described in the setting file and the link destination to be traced. Only document information that matches the collection file format described in the setting file is selected and collected from the network. As a result, the file format of the document information to be collected can be selectively specified as needed, the document information of the desired file format can be selectively specified and collected, and knowledge information on the network can be efficiently collected it can. Furthermore, since the desired knowledge information can be more appropriately narrowed down and collected, a more advanced knowledge database can be constructed easily, quickly and efficiently.
[0009]
The present invention also relates to a knowledge information collecting method for collecting document information to be registered in the knowledge database from the network, a starting URL setting field for collecting the document information to be registered in the knowledge database, and collection A setting screen display step for displaying a setting screen including setting items of a collection file format presented so that a plurality of possible document file formats can be selected, and setting contents on the setting screen displayed by the setting screen display step A configuration file creation step for creating a configuration file originally described, and document information to be registered in the knowledge database is collected from the network based on the description contents of the configuration file created in the configuration file creation step. A collecting step, wherein the collecting step is a startup described in the configuration file. When document information is acquired from the URL and the link destination to be traced, it is checked whether the document information matches the collected file format described in the setting file, and matches the collected file format described in the setting file. It comprises a processing step of selecting only document information, converting it into a predetermined registration file format, and registering it in a common format in the knowledge database .
[0010]
According to this knowledge information collection method, only document information that matches the file format set on the setting screen can be collected from the network, and the file format of the document information to be collected can be selectively selected as necessary. It is possible to selectively collect document information of an arbitrary file format. Thereby, desired knowledge information can be more appropriately narrowed down and collected, and a more advanced knowledge database can be easily and quickly constructed.
[0011]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 shows the overall configuration of a knowledge management system using a knowledge information collection system according to an embodiment of the present invention. This knowledge management system is for providing services such as collection, analysis, and search of knowledge information, and includes a Web collection system 11, a registration module 12, a knowledge engine 13, and the like. The Web collection system 11, the registration module 12, and the knowledge engine 13 are realized as programs that are executed on a server computer. The Web collection system 11 and the registration module 12 constitute a knowledge information system for collecting knowledge information used in the knowledge management system. This knowledge information collection system executes a process of collecting documents such as Web pages from the Internet / intranet 30 and registering the contents of the documents in the knowledge database (knowledge DB) 131 of the knowledge management system. The web collection system 11 includes a web collection module 111, a management interface 112, and a registration directory 113.
[0012]
The Web collection module 111 is a program for collecting various document files on the Internet / intranet 30 and outputting them in a format that can be registered in the knowledge DB 131. The Web collection module 111 has a multi-thread structure, and can perform document file acquisition processing from the Internet / intranet 30 using HTTP (HyperText Transfer Protocol) in parallel. HTTP is a communication protocol for acquiring a document described in HTML (Hyper Text Markup Language) from each Web server 31 that is an information site on the Internet / intranet 30. The Web collection module 111 acquires a URL (Uniform Resource Locator) document file specified by setting information for specifying Web collection conditions from the Internet / intranet 30, and the acquired document file includes a link destination URL. Then, a recursive process of further acquiring a document file of the URL is performed, and related document file groups are sequentially acquired from the Internet / intranet 30. In this case, in order to detect link information (URL) to another document file, it is necessary to analyze the acquired document file. However, in the analysis process, not only link information but also knowledge Extraction of text data to be registered in the DB 131 is also executed.
[0013]
Document files to be collected are not only hypertext files (HTML files) that can describe URLs that are link information to other files, but also text files (plane text), and various types of files created by various application programs. File format file (for example, Portable Document Format file, document file created with word processor software, file created with spreadsheet software, presentation data file created with presentation data creation software, etc. Attached files in the format are also collected.
[0014]
The Web collection module 111 acquires attribute information to be registered in the knowledge DB 131 and extracts the text data described above for each document file collected from the Internet / intranet 30. The attribute information acquired from each document file is, for example, the URL of the document file, the file creation date (update date), and the like. Then, the Web collection module 111 creates a registration file in which the attribute information and text data of each document file are described, and outputs it to the registration directory 113. Here, the registered file means a file having a predetermined description format as a common interface for registering document information of each of a plurality of types of document files having different file formats in the knowledge DB 131. This registration file is used for registering document information of each of a plurality of types of document files in the knowledge DB 131 in a common format. In the present embodiment, XML (eXtensible Markup Language) is used as the registration file.
[0015]
The management interface 112 is a program for setting the contents of each Web collection process (hereinafter referred to as a Web collection process task) to be executed by the Web collection module 111. The management interface 112 sets the Web collection condition for each Web collection process task and It has functions such as management, control of starting / stopping the Web collection module 111, and management / presentation of the collection status of each Web collection processing task. The management interface 112 is implemented as an external program (CGI program) that the Web server 22 can start through the CGI (Common Gateway Interface) so that the administrator user can perform necessary operations from the Web browser 21 of the user terminal. ing.
[0016]
Further, the Web collection system 11 includes a lock file 201, a control file 202, a setting file 203, a result file 204, and a log file 205 as files for managing and controlling the operation of the Web collection module 111, as shown in the figure. Is provided.
[0017]
The lock file 201 is an exclusive control file for preventing double activation of the Web collection module 111. The control file 202 is a file used by the management interface 112 to stop the Web collection module 111. For example, in response to an instruction from an administrator user, a web collection processing task being executed is interrupted. Used for The designation of interruption during the execution of collection can be performed by the administrator user through the management interface 112.
[0018]
The setting file 203 is a file for describing setting information for designating the Web collection condition for each Web collection processing task, and holds a plurality of setting information corresponding to each of the plurality of Web collection processing tasks. be able to. The Web collection condition for each Web collection processing task is set by the administrator user. Web collection conditions include at least origin address information (origin URL) that indicates the location of a document file that should be the origin of information collection from the Internet / intranet 30, and a URL that designates or collects the collection range only within the domain. Includes multiple types of collection range information that specifies the pattern of URLs, URL patterns that are not collected, the maximum number of document files or links to be collected, and the file format to be collected It is. The web collection operation of the web collection module 111 is controlled based on the web collection condition. That is, the Web collection module 111 sequentially collects document file groups related to the document file specified by the origin URL within the range defined by the collection range information. A plurality of types of collection range information described in the setting file 203 and its setting means will be described later with reference to the GUI screen.
[0019]
The Web collection condition also includes a knowledge database name for designating a registration knowledge database. That is, in the knowledge DB 131, a plurality of knowledge databases having different types of stored documents are defined, and the plurality of knowledge databases are managed by the respective knowledge database names. By specifying the name of the registration knowledge database for each Web collection task by the setting information in the setting file 203 described above, the collected document information can be registered in an arbitrary knowledge database in the knowledge DB 131.
[0020]
Furthermore, it is possible to set a collection condition when performing recollection as the collection range information. Here, recollection refers to, for example, when the web collection module 111 periodically executes the same web collection processing task, or when the web collection processing task that has been interrupted is resumed. Refers to Web collection processing.
[0021]
The result file 204 is a file for managing a list of document files collected as a result of the web collection process for each web collection task. Here, a list of URLs obtained as a result of the web collection is obtained. The number of document files and the like are output for each Web collection processing task. The result file 204 is used for the purpose of presenting the collection status of each Web collection processing task to the user and for the purpose of improving the efficiency of recollection. By using this result file 204, a restart point when a document file deleted from an information site on the Internet / intranet 30 is detected among the previously collected document files or when the Web collection processing is interrupted in the middle is detected. Can also be detected.
[0022]
The log fill 205 is a file that manages the success or failure of the acquisition of the document file by the Web collection module 111 and the type of error.
[0023]
The registration directory 113 is a storage area for outputting document contents to be registered in the knowledge DB 113. Here, the above-described registration file 114 in which attribute information and text data extracted from each document file by the Web collection module 111 are described is output. The Web collection module 111 has first and second processing modes for extracting text data.
[0024]
In the first processing mode, analysis processing is performed on document files of all file formats to extract text data and further detect link information (URL) (URL detection is only for HTML files). In the second processing mode, only HTML files and text files (PlainText) are analyzed, text data is extracted, and link information (URL) is detected (URL detection is only for HTML files). ) For example, analysis processing is not performed on a document file in another file format (hereinafter referred to as a content file) such as a file in a Portable Document Format format.
[0025]
When the second processing mode is used, the text data and attribute information of HTML and PlainText are described on the registration file 114 and output on the registration directory 113. On the other hand, for files in other formats, such as files in the Portable Document Format format, the content file 115 is output as it is on the registration directory 113, and the attribute information of the file and the path name of the content file 115 are displayed on the registration file 114. Described. In the second processing mode, it is important to extract text data in the analysis process for a file that requires an analysis process for detecting link information. Text data may be extracted only for HTML files.
[0026]
In both the first and second processing modes, the Web collection module 111 issues a registration request file to the registration module 12 when processing of the Web collection task being executed is completed, and registers the document. To the registration module 12. In the registration request file, the file name of the registration file, the knowledge database name of the registration destination, and the like are described.
[0027]
The registration module 12 registers the attribute information and text data of each document file collected by the Web collection module 111 in the knowledge DB 131. That is, the registration module 12 acquires the corresponding registration file 114 in response to the registration request from the Web collection module 111, and extracts the attribute information and text data described for each document file on the registration file 114. , It is registered in the registration destination knowledge database of the knowledge DB 131. The registration module 12 also has a text data extraction function. When the path name of the content file 115 is described in the registration file 114, the registration module 12 selects the content file specified by the path name. Text data is extracted from 115 and registered in the corresponding registration destination knowledge database of the knowledge DB 131.
[0028]
Furthermore, the registration module 12 also executes registration processing for documents collected by other collection modules other than the Web collection module 111. Other collection modules include, for example, a file collection module that collects document information from a file server of an electronic filing system, an RDB collection module that collects document information managed from an RDB database server, and an electronic bulletin board format community. There are a community collection module that collects document information posted there, a user collection module that a user uses to convert document files of various application programs into an output format of a registration file (XML), and the like. The registration module 12 is used in common by each of these collection modules. For any collection module, the registration file in the XML format described above is used as an interface to the registration module 12.
[0029]
The knowledge engine 13 performs a knowledge analysis process for utilizing information stored in the knowledge DB 131. In this knowledge analysis process, a process of extracting important words by analyzing features of a large amount of document information stored in the knowledge DB 131, a clustering process of classifying and organizing into a plurality of categories according to features, and the like are performed. The knowledge engine 13 is provided with a knowledge engine 132 for performing a natural language search, and each user accesses the knowledge engine 132 from the web browser 41 of the user terminal through the web server 42, thereby The knowledge stored in the knowledge DB 131 can be searched. Since the attribute information of each document stored in the knowledge DB 131 includes the URL of the document, the original document can be acquired from the corresponding information site and displayed on the Web browser 41. It should be noted that the search process can be performed by selecting a certain knowledge database name and targeting only the selected knowledge database, or all the knowledge databases in the knowledge DB 131.
[0030]
Next, the functional configuration of the Web collection module 111 will be described with reference to FIG.
The Web collection module 111 includes a collection control unit 301, an attribute extraction unit 302, a text extraction unit 303, and a format conversion unit 304, as illustrated. The collection control unit 301 sequentially executes the Web collection tasks specified by the setting information one by one for all the setting information held in the setting file 203. For each Web collection task, related document file groups are sequentially collected from the Internet / intranet 30 starting from the starting URL specified in the setting information. The URL to the link destination document included in the acquired document file is added to the URL list 305, and the collection process is recursively executed while acquiring the URL from the URL list 305. As the URL list 305, the above-described result file 204 can be used. The collection range is limited by the collection conditions held in the setting file 203.
[0031]
Each collected document file is first sent to the attribute extraction unit 302 where attribute information of each document file is acquired. Next, the text extraction unit 303 performs document analysis processing, and extracts text data to be registered in the knowledge DB 131 and a link destination URL to be acquired next. For example, for HTML files, text data is extracted by extracting portions other than HTML tags. The extracted text is divided into a title and a body (body). Acquisition of the link destination URL is performed by acquiring the HREF value of the A tag, the AREF tag, the SRC value of the FRAME tag, the IFRAME tag, the LAYER tag, and the REFRESH value of the META tag. When operating in the above-described second processing mode, the processing of the text extraction unit 303 is performed only for HTML files and text files, and for document files of other file formats, such as Portable Document Format format files. Is not done.
[0032]
The attribute information and text data acquired from each document file are sent to the format conversion unit 304, where they are formatted into an output format that conforms to the description format of the XML registration file, and output on the registration directory 113. In one registration file 114, for example, attribute information and text data for about 1000 document files are described. For document files that have not been subjected to text extraction, attribute information and path names are described on the registration file 114.
[0033]
Next, an example of the output format of the registration file 114 will be described with reference to FIGS.
[0034]
FIG. 3 shows an example of an output format when the text extraction unit 303 performs text extraction. The first tag <? Xml version = "1.0" encoding = "Shift_JIS"?> And <KnowledgeSystem> of the file indicate the beginning of the file, and the last tag </ KnowledgeSystem> indicates the end of the file.
[0035]
In each record surrounded by <RECORD> and </ RECORD>, attribute information and text data of one document file are described. The meaning of tags in each record is as follows.
[0036]
MODE: Mode information that specifies the operation mode of the registration module 12. This mode information instructs registration or deletion of document information (text data and attributes) with respect to the knowledge DB 131 for each document file. Either 2 = register (overwrite) or 0 = delete. When deleting, only the TYPE and UNIQUE of the following tags are not output.
[0037]
TYPE: Indicates the type of collection. In this example, it is always “Web collection”.
UNIQUE: A unique key for identifying the document file registered in the knowledge DB 131. Usually, the URL of the document is used as a unique key.
[0038]
ORGDATE: Indicates the creation date (or update date) of the document file.
TITLE: Indicates the title of the document file. The text data extracted from the title part of the HTML file becomes the title. No title is output for files other than HTML files. This title is output as the title of the document file on the search screen.
[0039]
AUTHOR: Describes the host name (host address of URL) of the information site that owns the document file.
DATE: Describe the date part of ORGDATE above.
URL: URL of the document file. Same value as UNIQUE.
BODY: Text data extracted from the document file is described.
[0040]
FIG. 4 shows an example of an output format related to a document file that has not been subjected to text extraction by the text extraction unit 303.
[0041]
Text data is not described in BODY, but the path name to the content file 115 output on the registration directory 113 is described in PATH1 in an area surrounded by <BDYFILE></BDYFILE>. DEL = 1 instructs the registration module 12 to delete the original file on the registration directory 113 after extracting text data from the content file 115. When the Web collection module 111 operates in the second processing mode, text data is described in BODY for HTML files and plain text, and content files are stored in BDYFILE for document files (content files) in other file formats. Will be described.
[0042]
Next, a series of processing procedures executed in the Web collection module 111 on the collected document file will be described with reference to the flowchart of FIG.
[0043]
First, attribute information (URL, AUTHOR, ORGDATE, DATE) to be registered in the knowledge DB 131 is acquired from the collected document file (step S101). The attribute information can be acquired using a value returned from the information site by HTTP, a value added in the collected document file, or the like. Thereafter, in the second processing mode, the file type is determined based on the extension of the collected document file, and the file is an HTML file, a plain text file, or other file format. Is determined (steps S102 and S103). If the collected document file is an HTML file or a plain text file (YES in step S103), the above-described text extraction process (text extraction and link URL detection in the case of an HTML file) is executed (step S104). Then, the process of describing the attribute information and the text data on the registration file 114 in the above-described format (inserting the text data into BODY) is performed (step S105). On the other hand, if the file has a file format other than the HTML file or the plain text file (NO in step S103), the file is output as it is on the registration directory 113 (step S106), and then the attribute information and the file A process of describing the path name on the registration file 114 (describes the path name in BDYFILE) is performed (step S107).
[0044]
If the document file of the target URL cannot be obtained from the Internet / intranet 30, the mode information (MODE) = 0 on the condition that the content of the document file is already registered in the knowledge DB 131. (Delete) is described on the registration file 114.
[0045]
In the first processing mode, the processes of steps S102 and S103 are not performed, and the processes of steps S104 and S105 are executed for all the acquired files.
[0046]
Next, the procedure of the registration process executed by the registration module 12 will be described with reference to the flowchart of FIG.
[0047]
Upon receiving a registration request from the Web collection module 111, the registration module 12 extracts the records one by one from the registration file (XML file) specified in the registration request file from the Web collection module 111, and performs the following processing for each record. Do. First, it is checked whether the mode information in the processing target record is MODE = 0 or MODE = 2 (steps S111 and S112).
[0048]
When MODE = 2, according to each tag in the record, the registration module 12 stores the data items described in the tag (the contents of TYPE, UNIQUE, ORGDATE, TITLE, AUTHOR, DATE, URL, BODY), Registration is performed in the registration destination knowledge database in the knowledge DB 131 specified in the registration request file (step S113). If there is no text data in the BODY tag, the text data is not registered. Next, it is determined whether or not a path name is described in the tag of BDYFILE (step S114). When the path name is described (YES in step S114), the corresponding content file is acquired from the storage area specified by the path name (step S115), and the process of extracting text data from the content file is performed. Performed (step S116). Then, the extracted contents are registered in the registration destination knowledge database as text data of the corresponding document file (step S117).
[0049]
When MODE = 0, the attribute information and text data of the registered document file specified by UNIQUE in the record are searched from the knowledge DB 131, and the registered contents are deleted from the registration destination knowledge database (step S118).
[0050]
FIG. 7 shows the relationship between the state of a document file (Web content) on the Internet / intranet 30 and registration / deletion processing to be performed on the knowledge DB 131. In the present embodiment, the following processing is performed based on a policy that the state of the latest Web content on the Internet / intranet 30 is reflected in the knowledge DB 131 as much as possible.
[0051]
(1) The updated Web content is overwritten on the knowledge DB 131 (MODE = 2). When registering the updated Web content attribute information and text data, the UNIQUE is the same as the UNIQUE of the pre-update Web content already registered in the knowledge DB 131. Therefore, by MODE = 2, the attribute information and text data of the Web content before update already registered in the knowledge DB 131 are updated (overwritten) to the attribute information and text data of the updated Web content.
[0052]
(2) The added Web content is additionally registered in the knowledge DB 131 (MODE = 2). When registering the attribute information and text data of the added Web content, the UNIQUE is not registered in the knowledge DB 131. Therefore, the attribute information and text data of the added Web content can be additionally registered in the knowledge DB 131 by MODE = 2.
[0053]
(3) The deleted Web content is also deleted from the knowledge DB 131 (MODE = 0).
[0054]
Next, functions provided by the management interface 112 will be specifically described. The management interface 112 is a user interface for setting, managing, and executing a Web collection task as described above. 1) A function for creating a plurality of Web collection task settings and storing them in the setting file 204. 2) Collection Administrator user with functions to perform management operations such as list and deletion of settings, 3) function to start and stop (suspend and resume) collection, and 4) function to check the status and results of collection in real time To provide.
[0055]
As shown in FIG. 8, the screen displayed on the administrator user's Web browser 21 by the management interface 112 includes three screens, a “list screen”, a “setting screen”, and a “status screen”. When is started, the list screen is displayed first. The three screens change in accordance with operations performed by the administrator user on each screen. The list screen displays a list of currently created settings and the latest collection results for each setting. For example, when a created setting is selected on this list screen and the selected setting is changed to a setting screen or status screen related to the setting, the setting content can be added / deleted on the setting screen. On the status screen, it is possible to specify activation and termination (interruption and resumption) related to the setting. Also, when a new setting creation is selected on the list screen, the screen transitions to the setting screen, and a process for creating a new setting can be performed on the setting screen. Hereinafter, each of the list screen, setting screen, and status screen will be described in detail.
[0056]
FIG. 9 shows an example of a list screen. As shown in the figure, a “list” anchor 501, a “setting anchor” 502, and a “situation” anchor 503 are displayed on the list screen. These “list” anchor 501, “setting anchor” 502, and “situation” anchor 503 are links to the list screen, setting screen, and status screen, respectively. When the “list” anchor 501 is clicked on the list screen, the display content of the list screen is updated to the latest state. Further, a “new setting” field 504, a “create” button 505, and a status list display unit 506 are provided on the list screen. When a setting name of a Web collection task to be newly created is input in the “new setting” field 504 and a “create” button 505 is clicked, a setting screen is displayed and setting information relating to the setting name can be newly created. .
[0057]
The status list display unit 506 displays a list of collection statuses for each Web collection task that has already been created and registered based on the contents held in the result file 204 described above. The following items are displayed for each Web collection task.
[0058]
"Setting name": Name of the registered setting "Knowledge database name": Database name where the collected data is registered "Status": Current status of the setting (Non-collecting / Collecting / Ending / Suspended / Abnormal end Or)
"Start": Last start date and time "End": Last end date and time "Files": Number of files collected in the previous task execution (this time) and the total number of files collected so far Maximum value (whole)
Thus, in this embodiment, a plurality of Web collection tasks can be set, and the collection status of each of the plurality of Web collection tasks can be confirmed on the list screen.
[0059]
Next, an example of a setting screen will be described with reference to FIG. On the setting screen, as shown in the figure, in addition to the “list” anchor 501, “setting anchor” 502, “status” anchor 503, a “setting name” field 601, a “knowledge database name” field 602, “proxy” A “server” setting form 603, a “collection file format” setting form 604, an “acquisition request” setting form 605, an “authentication information” setting form 606, a “collection site” setting form 607 constituting a collection site setting screen, and the like are provided. These can be used to perform the following setting editing operations.
[0060]
(1) Setting Name: The “setting name” field 601 displays a setting name designated by creating a new setting or selecting an existing setting on the list screen.
(2) DB selection: By inputting the knowledge database name in the “knowledge database name” field 602, the registration destination knowledge database in which the collected data is registered is selected.
[0061]
(3) Proxy: A proxy server used for collection is set using a “proxy server” setting form 603.
(4) File format to be collected: Select a file format to be collected on the “collected file format” setting form 604. In this embodiment, HTML and TEXT files are selected by default (the specification of HTML and TEXT files cannot be canceled). In addition, for example, a file in a Portable Document Format format, a document file created by word processor software Attached files of various general-purpose file formats such as files created by spreadsheet software and presentation data files created by presentation data creation software can be selected as collection targets. The present embodiment shows an example in which the file formats of collectable document files are six types, and each can be arbitrarily designated by a check box. For example, by designating all six types of file formats (HTML, PDF, PPT, DOC, XS, JBW) with check boxes, document files of all the designated file formats can be collected. Further, by deselecting a document file with a file format that is no longer necessary with a check box, the document file with that file format is deleted from the knowledge DB 131.
[0062]
(5) Acquisition request: On the “acquisition request” setting form 605, the time-out time and the number of retries at the time of URL acquisition are designated.
(6) Authentication information: On the “authentication information” setting form 606, authentication information necessary for accessing a URL that requires user authentication is set.
[0063]
(7) Collection site: On the “collection site” setting form 607, the following items are set.
[0064]
-Origin URL setting: Specifies the URL that is the origin of collection. In the present embodiment, it is possible to input up to 1000 characters excluding single-byte characters and special characters on the setting field of the starting URL.
[0065]
-Maximum number of link steps: Specify the maximum number of link steps from the starting URL to be collected.
-Maximum number of files: Specify the maximum number of files to collect.
-Collection range: When "Collect only within the domain" is specified, collection is performed only within the domain of the origin URL. The default is not specified. When this “collect only within the domain” is specified by a check box, the document information from the corresponding information site on the Internet / intranet 30 is set only for the origin URL set in the origin URL setting field. To collect.
[0066]
URL pattern to be collected: A character string pattern of a URL to be collected is designated. Of the URLs to be collected, only the URL files including the designated character string pattern are collected. In the present embodiment, a maximum of 1000 bytes can be set, one pattern per line, in the setting field for setting the URL pattern to be collected.
[0067]
URL pattern not to be collected: Specifies a URL character string pattern to be excluded from the collection target. Among the URLs to be collected, URL files including the designated character string pattern are not collected. In this embodiment, it is possible to set a maximum of 1000 bytes, one pattern per line, in the setting field for setting URL patterns not to be collected.
[0068]
If there is a URL that matches both the pattern to be collected and the pattern that is not collected, the pattern that is not collected has priority. That is, if there is a URL that matches both the pattern to be collected and the pattern not to be collected, the file of the URL is not collected.
[0069]
Re-collection setting: Specifies the behavior when the collection task to be set performs re-collection from the same origin URL. There are three modes for re-collection ("Collect all files", "Collect only uncollected files", "Collect only updated files (with delete function)"). Can be specified. Here, the “collect all files newly” mode means a mode in which all URLs to be collected are collected again. In other words, regardless of whether or not it was acquired in the past, new collection is performed again from the starting URL. Further, the “collect only uncollected files” mode is a mode in which previously collected URLs are not collected. The “collect only updated files (with delete function)” mode is a mode in which only document files that have been updated since the previous collection point are collected. The document to be checked for the presence or absence of update is a URL collected in the previous collection more than the specified number of days before the current collection. The value entered in the “Update Check Frequency” field is the specified number of days. The default specified number of days is “1 day”. In the previous collection, if the URL collected more than the specified number of days before the current date has been deleted from the relevant information site on the Internet / intranet 30, and the document file could not be obtained from the URL, Registration information corresponding to the document file is deleted from the knowledge DB 131.
[0070]
FIG. 11 shows an example of a status screen. For example, when the “status” anchor 503 is selected on the list screen or the setting screen, the screen is shifted to the currently displayed task status screen. On this status screen, the “setting name”, “knowledge database name”, “status”, and the like of the task are displayed. A button 701 for instructing start / stop (resumption / interruption) of collection is displayed next to “setting name”. In “Status”, the status of the task (collecting / finished / not collected / suspended / abnormal termination) is displayed, and the starting URL of the task, the number of collected link stages (current / maximum / upper limit), and the number of files Status information 702 such as (current / whole / upper limit) is also displayed.
[0071]
When the state of the task is uncollected, ended, or interrupted, “collection start” is displayed on the button 701 as shown in FIG. When the “collection start” button 701 is selected, execution of the task is started by the web collection module 111. If the current state is terminated or interrupted, recollection will begin. If the current state of the task is being collected, as shown in FIG. 12, “Collecting collection” is displayed on the button 701. When the “collection interruption” button 701 is selected, the collection task in operation is interrupted. A list of URLs collected before the interruption is maintained in the result file 204.
[0072]
Next, with reference to FIG. 13 to FIG. 15, the operation of interrupting and resuming (recollecting) collection processing by operating a button on the status screen will be specifically described.
[0073]
This interruption / resumption function is a function used when the Web collection process being executed is temporarily stopped and resumed at a later date. The Web collection processing by the Web collection module 111 is executed while writing a URL, a status code indicating success or failure, etc. as progress status information in the result file 204 every time a document file is acquired from the Internet / intranet 30. If there is a link destination URL in the acquired document file, the URL is added to the result file 204, and the process is repeated until there are no uncollected URLs in the result file 204. When the “collection interruption” button 701 is pressed in the middle of the collection process, the web collection task being executed is interrupted at that time. Then, when the “start collection” button 701 is pressed while the interrupted Web collection task is selected, the interrupted Web collection task is resumed and recollection is started. For example, when the “collect only uncollected files” mode is selected as the recollection mode, the collection process is resumed from the uncollected URL held in the result file 204.
[0074]
FIG. 14 shows an example of a URL list held in the result file 204 as progress status information. As shown in the figure, the result file 204 includes “collection date”, “collection date / time”, “status code”, “origin URL”, and “link stage number” in addition to the “URL” for each URL to be acquired. To be recorded. “Number of link steps” indicates the number of links of the acquisition target URL when viewed from the starting URL. For uncollected URLs, information on “collection date”, “collection date / time”, and “status code” is blank. Thereby, it is possible to detect a restart point when the Web collection process is interrupted in the middle. “Status code” indicates success or failure of the collection process, and this is an HTTP status code returned from the Web server from which the Web page is acquired. The type of HTTP status code includes “200” indicating that the file acquisition was successful (normal end), “403” indicating an error due to access authority, and the corresponding file is not found (in response to the request for acquiring the deleted document file) Response) error "404", timeout error "408" and the like are included.
[0075]
The flowchart in FIG. 15 shows a series of processing procedures from when Web collection is started until it is interrupted.
[0076]
The Web collection module 111 first acquires a document file specified by the origin URL (step S121), and the result (collection date, collection date, status code, origin URL, link stage number, URL (same as origin URL). ) Is registered in the result file 204 to update the progress status information (step S122) If the acquisition is successful, attribute information to be registered in the knowledge DB 131 is acquired from the collected document file (step S122). Next, text extraction processing (text extraction in the case of an HTML file and detection of a link destination URL) is executed (step S124) If a link destination URL is detected, the URL, the above-described starting URL, and the number of link stages are described. Are additionally registered in the URL list of the result file 204 (step S125). Thereafter, a process for describing the attribute information and text data on the registration file is performed (step S126), where it is determined whether or not there is an interruption instruction (step S127). NO), it is determined whether or not the Web collection process is completed based on whether or not an uncollected URL is registered in the result file 204 (step S128).
[0077]
If an uncollected URL is registered in the result file 204 (NO in step S128), the processing from step S121 described above is repeated. That is, the document file specified by the URL added to the result file 204 is acquired (step S121), and the result (collection date, collection date and time, status code) is registered in the result file 204 to update the progress status information. Is performed (step S122). If the acquisition is successful, attribute information to be registered in the knowledge DB 131 is acquired from the collected document file (step S122), and then text extraction processing (text extraction and link destination URL detection in the case of an HTML file) is executed. (Step S124). When the link destination URL is detected, the URL, the starting URL, and the number of link steps are additionally registered in the URL list of the result file 204 (step S125). Thereafter, a process for describing the attribute information and text data on the registration file 114 is performed (step S126). Such Web collection processing is recursively executed until collection is completed within the collection range of the Web collection condition set in the setting file 203. When the “collection interruption” button 701 is pressed at the time of this web collection, a suspension instruction is issued from the management interface 112 to the web collection module 111 through the control file 202 (YES in step S127) and executed by the web collection module 111. The web collection task inside is interrupted.
[0078]
Here, FIG. 16 shows Web collection processing executed by the Web collection module 111 in accordance with the collection target file format set on the “collection file format” setting form 604 of the setting screen shown in FIG. 10 as the collection range. This will be described with reference to the flowchart shown.
[0079]
When “setting anchor” 502 is clicked on the list screen shown in FIG. 9, as shown in the screen transition diagram of FIG. 8, the list screen shown in FIG. 9 is changed to the setting screen shown in FIG. By setting the “collection file format” setting form 604, the “collection site” setting form 607, etc. on the screen, the setting file 203 including the setting contents of each setting form is created (steps S241 and S242). ).
[0080]
In the present embodiment, the file format of document information (document file) to be collected is set in the “collected file format” setting form 604. Here, when there are document file formats to be collected other than the default HTML and TEXT files among the six types of document file formats (HTML, PDF, PPT, DOC, XS, JBW) that can be collected, By specifying the document file format to be collected with a check box, the document files of all the designated file formats can be collected.
[0081]
Further, in the “collection site” setting form 607 on the setting screen, the starting URL of the site to be collected is set in the starting URL setting field. At this time, the URL pattern to be collected, the URL pattern not to be collected, the setting for collecting only within the domain, the upper limit value setting for the number of document files to be collected, the upper limit value setting for the number of link steps, are set simultaneously with the setting of the starting URL In addition to the document file format to be collected, the URL pattern to be collected, the URL pattern not to be collected, the collection only within the domain, Each filtering (filtering) function such as the upper limit of the number of document files and the upper limit of the number of link stages can be performed. However, in the present embodiment, the setting of the collection range other than the setting of the file format to be collected is not particularly mentioned.
[0082]
After setting the “collection file format” setting form 604 and the “collection site” setting form 607 as described above on the setting screen shown in FIG. 10, click the “status anchor” 503 on the setting screen. Then, as shown in the screen transition diagram of FIG. 8, the setting screen shown in FIG. 10 is changed to the status screen shown in FIG. 11 (step S243), and the “start collection” button 701 is selected on the status screen. Then, the Web collection process shown in FIG. 15 is started, and in the Web collection process, the Web collection can be performed by narrowing the collection range according to the setting contents of the “collection file format” setting form 604.
[0083]
In order to narrow down the collection range, in the present embodiment, according to the file format set in the “collection file format” setting form 604, the collection processing is performed by selecting the document information (document file) to be collected.
[0084]
In this collection process, referring to the setting file 203, first the starting URL is read (steps S244 to S246), and the site designated by the URL, that is, the Web server 31 is accessed to make a file request. In response to this request, the Web server 31 returns the content type of the collection target file.
[0085]
When the Web collection module 111 acquires the content type from each Web server 31 specified by the URL, the content type is checked (step S247), and the document file to be collected is set to the “collection file format”. It is determined whether or not the file format set in the form 604 matches (step S248).
[0086]
If the document file to be collected matches the file format set in the “collected file format” setting form 604 (YES in step S248), the collecting process is started (step S249).
[0087]
If the document file to be collected does not match the file format set in the “collected file format” setting form 604 (NO in step S248), the link destination is checked, and if there is a link destination (step No in S250), the same access as described above is performed for each linked Web server 31 (steps S246, S247,...).
[0088]
In this way, the Web collection module 111 sorts the document files to be collected according to the file format to be collected set in the setting file 203, and collects document information from the network. As a result, a document file having a required file format can be specified and collected from the network, and knowledge information on the network can be efficiently collected by narrowing down the document files to be collected.
[0089]
Furthermore, in addition to the specification of the file format to be collected as described above, the setting of the URL pattern to be collected, the setting of the URL pattern not to be collected, the setting of collecting only within the domain, and the upper limit of the number of document files to be collected It is possible to set multiple types of collection ranges at the same time, such as setting the upper limit value of the number of link stages, setting the file format to be collected, etc., and filtering the document file to be collected by combining these, the desired knowledge information By appropriately narrowing down and collecting data, a more advanced knowledge database can be constructed easily, quickly and efficiently. For example, by setting a combination of the file format to be collected, the URL pattern to be collected, and the URL pattern not to be collected as described above, the URL pattern to be collected and the URL URL to be collected in the specified file format are set. Desired knowledge information can be collected by designating a pattern. In addition, by combining and setting the file format to be collected and the collection only within the domain, knowledge information in the designated file format can be collected only within the domain of the starting URL. In addition to this, by setting an upper limit value of the number of document files, an upper limit value of the number of link stages, and the like, the number of link stages and the number of files to be collected can be arbitrarily regulated.
[0090]
Since all the functions of the knowledge information collection system of this embodiment are realized by a computer program, the computer program is stored in a computer-readable storage medium, and the computer program is stored in the computer network through the storage medium. The effect similar to that of the present embodiment can be easily obtained simply by introducing and executing the program on a connectable ordinary computer.
[0091]
Further, the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the scope of the invention in the implementation stage. Further, the above embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituent requirements are deleted from all the constituent requirements shown in the embodiment, the problem described in the column of the problem to be solved by the invention can be solved, and the effect described in the column of the effect of the invention Can be obtained as an invention.
[0092]
【The invention's effect】
As described above, according to the present invention, it is possible to efficiently collect various types of document files having different file formats from the Internet as knowledge information.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a knowledge information collection system according to an embodiment of the present invention.
FIG. 2 is an exemplary block diagram illustrating a functional configuration of a Web collection module provided in the knowledge information collection system according to the embodiment;
FIG. 3 is a view showing an example of an output format of a registration file used in the knowledge information collection system of the embodiment.
FIG. 4 is a view showing another example of an output format of a registration file used in the knowledge information collection system of the embodiment.
FIG. 5 is an exemplary flowchart illustrating a processing procedure of a Web collection module provided in the knowledge information collection system of the embodiment;
FIG. 6 is an exemplary flowchart illustrating a processing procedure of a registration module provided in the knowledge information collection system of the embodiment;
FIG. 7 is an exemplary view showing the relationship between the Web content and the registration process in the knowledge information collection system of the embodiment.
FIG. 8 is a view for explaining three screens provided by the knowledge information collection system of the embodiment and transitions thereof;
FIG. 9 is a view showing an example of a list screen used in the knowledge information collection system of the embodiment.
FIG. 10 is a view showing an example of a setting screen used in the knowledge information collection system of the embodiment.
FIG. 11 is a diagram showing an example of a situation screen used in the knowledge information collection system of the embodiment.
FIG. 12 is a diagram showing an example of a situation screen used in the knowledge information collection system of the embodiment.
FIG. 13 is an exemplary view for explaining a Web collection process interruption / resumption operation in the knowledge information collection system according to the embodiment;
FIG. 14 is a diagram showing an example of the contents of a result file provided in the knowledge information collection system of the embodiment.
FIG. 15 is an exemplary flowchart illustrating a processing procedure from the start to the interruption of the Web collection processing in the knowledge information collection system according to the embodiment;
FIG. 16 is an exemplary flowchart illustrating the setting of the file format to be collected and the procedure of the Web collection process according to the setting in the knowledge information collection system of the embodiment;
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 11 ... Web collection system 12 ... Registration module 13 ... Knowledge engine 30 ... Internet / intranet 111 ... Web collection module 112 ... Management interface 113 ... Registration directory 131 ... Knowledge database 132 ... Search engine 201 ... Lock file 202 ... Control file 203 ... Setting File 204 ... Result file 205 ... Log file 301 ... Collection control unit 302 ... Attribute extraction unit 303 ... Text extraction unit 304 ... Format conversion unit 604 ... "Collection file format" setting form 607 ... "Collection site" setting form

Claims (6)

In a knowledge information collection system that collects document information to be registered in the knowledge database from the network,
A setting screen including a setting field of a starting URL for collecting document information to be registered in the knowledge database and a setting item of a collection file format that presents a plurality of types of document file formats that can be collected is displayed. A setting screen display means;
Setting file creating means for creating a setting file described based on the setting contents on the setting screen displayed by the setting screen displaying means;
Information collecting means for collecting document information to be registered in the knowledge database based on the description contents of the setting file created by the setting file creating means from the network;
The information collecting means checks whether or not the document information matches the collected file format described in the setting file when acquiring the document information from the starting URL described in the setting file and the link destination to be traced. Selecting only document information that matches the collected file format described in the setting file, collecting it from the network, and converting each of the collected document information into a predetermined file format for registration Knowledge information collection system characterized by registering in a common format.   The information collecting means includes a processing mode for performing analysis processing on document files of all file formats to extract text data, and an analysis processing for files that require analysis processing for detecting link information. The knowledge information collecting system according to claim 1, further comprising a processing mode in which text data extraction is also performed.   The knowledge information collection system according to claim 1 or 2, wherein a file format that can be registered in the knowledge database is initially set in the setting item of the collection file format.   The document information of the canceled file format registered in the knowledge database is registered in the knowledge database when a file other than the default file format is canceled on the setting items of the collected file format. 4. The knowledge information collecting system according to claim 3, further comprising processing means for deleting from the information.   The setting screen displayed by the setting screen display means collects a setting item for setting the collection range within the domain, an item for setting the character string pattern for the URL to be collected, an item for setting the character string pattern for the URL not to be collected, 2. The method according to claim 1, further comprising at least one of an item for setting an upper limit of the link and an item for setting an upper limit of the number of files to be collected, wherein the setting content of each item is reflected in the setting content of the setting item of the collection file format. 2. Knowledge information collecting system described in 2. In a knowledge information collection method for collecting document information to be registered in a knowledge database from the network,
A setting screen including a setting field of a starting URL for collecting document information to be registered in the knowledge database and a setting item of a collection file format that presents a plurality of types of document file formats that can be collected is displayed. Setting screen display step,
A setting file creating step for creating a setting file described based on the setting contents on the setting screen displayed by the setting screen displaying step;
A collection step of collecting document information to be registered in the knowledge database from the network based on the description content of the configuration file created in the configuration file creation step;
The collecting step checks whether or not the document information matches the collected file format described in the setting file when acquiring the document information from the starting URL described in the setting file and the link destination to be traced; Selecting only document information that matches the collected file format described in the setting file, converting it to a predetermined file format for registration, and registering in a common format in the knowledge database. Knowledge information collection method.

Images