JP2006106911A - Knowledge information collecting system, knowledge information collecting method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To collect only information which is useful to users. <P>SOLUTION: A starting point URL, the number of stages of links whose information is to be collected and keywords representing unnecessary words are set in a setting file 13. A link label extraction module 113 extracts link character strings from page information collected from a network (the Internet/intranet) 20. A link determination module 114 determines whether the page information of a linking destination is useless from the extracted link character strings and the set keywords representing the unnecessary words. A collection control module 111 controls the collection of information from the network 20 by following the links from the starting point URL. The collection control module 111 does not collect the page information of the linking destination which is determined to be useless by the link determination module 114 even if the information is in a range of the number of the set stages. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a knowledge information collection system, a knowledge information collection method, and a program suitable for collecting information to be registered in a knowledge database used in a knowledge management system from a network.

  In recent years, knowledge management systems for supporting sharing of knowledge information have been developed. This knowledge management system is for accumulating and managing knowledge information such as personal know-how in a knowledge database. By combining it with a search function such as natural language search, the knowledge management system can be used efficiently. Is realized.

  In such a knowledge management system, how to efficiently collect knowledge information is an important point. Therefore, recently, a knowledge information collection system for efficiently collecting various types of document files having different file formats as knowledge information from a network such as the Internet has been developed (see, for example, Patent Document 1). In the knowledge information collection system described in Patent Document 1, document information is collected from the Internet as follows according to the knowledge information collection conditions (knowledge information collection conditions) set in the setting file.

First, the knowledge information collection system has a setting file and a web collection module. In the setting file, at least one of the number of links to be collected and the number of collected files for each link is set together with the origin URL (Uniform Resource Locator) by the operation of a user (for example, an administrative user who is an administrator). The When the starting URL and the number of links are set in the setting file, the Web collection module traces all the links within the range that does not exceed the upper limit of the number of links set from the starting URL, and retrieves document information ( Page information). When the number of collection files for each link is set in addition to the starting URL and the number of links, the Web collection module does not exceed the upper limit of the number of links set from the starting URL, and the set link Document information is collected from the Internet by following all links within a range that does not exceed the number of collected files.
JP 2003-303197 A (paragraphs 0008, 0010, 0086 to 0089)

  As described above, according to the knowledge information collection technique described in Patent Document 1 (hereinafter referred to as the prior art), the number of link stages or the like to be collected from the network can be arbitrarily set using the setting file. Is specified, the document information is collected by tracing all the links within the range not exceeding the upper limit of the number of stages of the specified links from the starting URL.

  However, in the above prior art, all information is collected from the designated URL within the range of the number of stages of the designated link regardless of the contents of the document information (page information) to be collected, which is unnecessary (unnecessary) for the user. There is a possibility that a lot of information is included. The prior art also has a problem in that even if there is useful (important) information for the user in the link beyond the specified number of steps, the information is not collected.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide a knowledge information collection system, a knowledge information collection method, and a program capable of efficiently collecting only information useful for a user.

  According to one aspect of the present invention, a knowledge information collection system that collects information to be registered in a knowledge database from a network is provided. This system sets the starting location information indicating the location of page information that is the starting point of information collection from the network and the number of link stages to be collected from the network, and also collects information from the network. Setting means for setting a word related to a link to be removed as a keyword representing an unnecessary word, link character string extracting means for extracting a link character string from page information collected from the network, and extracted link character Link determination means for determining whether the linked page information is useless from the column and the keyword indicating the unnecessary word set, and collecting information from the network by following the link from the set starting point location information Information collection control means to be controlled, and determined to be useless by the link determination means. Landing page information of includes an information collection controlling means be in the range of number of links set to exclude the collection.

  In the above configuration, it is determined whether the linked page is useless from the link character string in the page and the keyword indicating the set unnecessary word, and the page determined to be useless is set in advance. Even if it is within the range of the number of linked links, it is not collected. This prevents the collection of even unnecessary information that should not be registered in the knowledge database because it is within the preset range of the number of links, and only useful information that should be registered in the knowledge database. It can be collected efficiently.

  Here, instead of setting a word related to a link to be excluded from information collection from the network as a keyword representing an unnecessary word in the setting means, a link to be collected from the network. In addition to having the function of setting a phrase related to the keyword as a keyword representing an important word, instead of determining whether the link destination page information is useless, the link determination means sets the extracted link character string. It is preferable to provide a function for determining whether the linked page information is useful from the keyword representing the important word. In this case, the information collection control means may have a function of collecting the page information of the link destination determined to be useful even if it exceeds the set number of stages.

  In such a configuration, it is determined whether the linked page is useful from the link character string in the page and the keyword representing the set important word, and the page determined to be useful is set in advance. Even if the number of linked links is exceeded, it will be collected. This prevents the useful information that should be registered in the knowledge database from being collected because it is outside the preset number of link stages, and only the useful information that should be registered in the knowledge database is more efficient. Can be collected well.

  Further, the system searches the information collected in the knowledge database according to a given search expression, presents the search result to the user, and information of the information referred to according to the search result by the search means. Means for evaluating usability or uselessness by the user, the search formula used for the search by the search means, the number of times the information is collected for each piece of information collected in the knowledge database, Search log storage means for storing the user's evaluation result for the information as a search log, words and phrases appearing in the search formula stored in the search log storage means, the number of references for each information, and the evaluation result for each information Log statistics generation means for generating an evaluation value representing the degree to which the word is important or unnecessary for the user for each word appearing in the search expression; Based on the evaluation value for each phrase that is, it may be configured to the keyword generation unit for generating keywords that represent important words or unnecessary word that can be set by the setting means is added.

  In such a configuration, for each word / phrase that appears in the search expression, the word / phrase appears in the user based on the word / phrase that appears in the search expression used in the past search, the reference count for each information, and the evaluation result for each information. Is used as a keyword that represents an important word or an unnecessary word from the evaluation value for each word, that is, from the evaluation result on the usage status (search log) of the search system. Are automatically extracted. As a result, if a keyword representing an important word is automatically extracted, it is possible to save the user from setting an important keyword. In addition, if a keyword representing an unnecessary word is automatically extracted, it is possible to save the user from setting a keyword related to a link that should be excluded from information collection. In addition, keywords related to links that should not be collected (unnecessary keywords) are different from keywords related to links that should be collected (important keywords). However, it is difficult to find out without actually referring to it. Therefore, it is extremely useful for the user that keywords representing unnecessary words are automatically extracted.

  Here, instead of setting the keywords representing the automatically extracted important words or unnecessary words unconditionally by the setting means, a list of keywords representing the automatically extracted important words or unnecessary words is displayed to the user. It is good also as a structure which presents and makes a user select the keyword set by the said setting means from the list.

  In such a configuration, the user can determine whether a keyword representing an automatically extracted important word or unnecessary word is really important or unnecessary.

  According to the present invention, it is determined whether the linked page is useless from the link character string in the page and the keyword indicating the set unnecessary word, and the page determined to be useless is set in advance. Even if it is within the range of the number of steps of the link, it is excluded from the collection target, so since it is within the range of the number of steps of the link set in advance, even unnecessary information that should not be registered in the knowledge database is collected Can be prevented.

  Further, according to the present invention, it is determined whether the linked page is useful from the link character string in the page and the keyword representing the set important word, and the page determined to be useful is set in advance. Even if the number of linked links exceeds the specified number of links, the information is out of the preset number of links, so even useful information to be registered in the knowledge database is not collected. Can be prevented.

  Therefore, according to the present invention, it is possible to efficiently collect only useful information for the user.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of a knowledge management system that realizes a knowledge information collection system according to an embodiment of the present invention. This knowledge management system provides services such as collection, analysis and search of knowledge information. The knowledge management system includes a Web information collection system 11, a knowledge search system 12, a setting file 13, a keyword generation module 14, and a synonym dictionary 15. These Web information collection system 11, knowledge search system 12, setting file 13, keyword generation module 14 and synonym dictionary 15 collect knowledge information for collecting knowledge information which is one of the services provided by the knowledge management system. Build the system.

  The Web information collection system 11 collects Web information (page information) useful for the user from a network, for example, a Web server 21 on the Internet / intranet 20 and extracts a text portion. DB) 121. The Web information collection system 11 includes modules of a collection control module 111, a text extraction module 112, a link label extraction module 113, and a link determination module 114.

  The collection control module 111 controls collection of useful page information from the Internet / intranet 20 according to the setting contents of the setting file 13. The text extraction module 112 extracts text from the page information collected under the control of the collection control module 111 and stores it in the knowledge DB 121. The link label extraction module 113 extracts a link label, which is a link character string in the page, from the collected page information. The link determination module 114 determines whether to collect link destination pages from the extracted link labels and the setting contents of the setting file 13. The determination result is passed to the collection control module 111 and used for control of page information collection by the collection control module 111.

  The knowledge search system 12 includes a knowledge DB 121, a search engine 122, a search log 123, and a log statistics generation module 124. The knowledge DB 121 is used to store document information (text) extracted by the text extraction module 112 from the page information collected by the collection control module 111 in the Web information collection system 11. The search engine 122 searches the knowledge DB 121 for document information (text) that matches the search expression according to the search expression (search sentence, search condition) indicated by the search request input from the web browser 16 in response to the operation of the user 101. The search result is presented to the user 101 via the Web browser 16. The search engine 122 also presents the user 101 with document information selected from the search results presented to the user 101. As a result, the user 101 can refer to the target document information. The search log 123 is a history (log) of information search / reference by the search engine 122, for example, a search formula used for the search, a reference count of the search result, and an evaluation result by the user for each referred document information. Used to store information. The log statistic generation module 124 statistically analyzes the words and phrases appearing in the search expression, the number of reference times of the search result, and the user evaluation result for each referenced information in accordance with the search log 123, thereby For a word, information (log statistical information) necessary to determine whether the word is important to the user, unnecessary, or neither is generated.

  The setting file 13 includes conditions for collecting Web information, such as a starting URL, the number of stages of links (initial number of stages) n to be collected from the Internet / intranet 20, the upper limit of the number of collected pages, important words and unnecessary words. (Knowledge information collection conditions) are set and maintained. In the present embodiment, representative words registered in the synonym dictionary 15 are used as important words and unnecessary words set in the setting file 13. The setting file 13 is generated according to a setting request input from the web browser 17 in response to an operation of a user, for example, a management user (administrator) 102. In FIG. 1, one setting file 13 is shown. However, a plurality of setting files 13 are generated, and an arbitrary setting file 13 is designated by the user's operation, and Web information collection according to the conditions set in the designated setting file 13 is performed. It may be a configuration.

  The starting URL is location information indicating the location of page information that is the starting point when information collection from the Internet / intranet 20 is controlled by the collection control module 111. Important words and unnecessary words are keywords used to determine whether link destination page information is useful or useless from link labels (link character strings) in page information. The keyword generation module 14 generates important words and unnecessary words as keywords based on the statistical information (log statistical information) related to the search generated by the log statistical generation module 124. A synonym dictionary 15 is used for keyword generation by the keyword generation module 14. In the synonym dictionary 15, a set of phrases having similar meanings is registered in advance as synonyms. Each set of synonyms is associated with a representative word. For example, “price”, “price”, “list price”, and “charge” are registered in the synonym dictionary 15 as elements of a synonym group whose representative word is “price”.

Each module in the Web information collection system 11, the search engine 122 and the log statistics generation module 124 in the knowledge search system 12, and the keyword generation module 14 are
This is realized by reading and executing a special software program (knowledge management program) installed in the computer by the computer (CPU in the computer). This program is stored in advance in a computer-readable storage medium (a magnetic disk typified by a floppy (registered trademark) disk, a CD-ROM, an optical disk typified by a DVD, a semiconductor memory typified by a flash memory, etc.). Can be distributed. Further, this program may be downloaded (distributed) via a network.

Next, Web information collection processing in the knowledge management system of FIG. 1 will be described with reference to the flowcharts of FIGS.
First, it is assumed that the management user 102 wants to collect Web information (page information) to be registered in the knowledge DB 121 from a Web server on the Internet / intranet 20 using the knowledge management system (internal Web information collection system 11) of FIG. . In this case, the management user 102 calls the Web information collection system 11 by operating an input unit (not shown) such as a keyboard. Then, the collection control module 111 in the Web information collection system 11 requests the management user 102 to perform an input operation for designating the setting file 13. When an arbitrary setting file 13 is specified by an input operation of the management user 102, the collection control module 111 responds from the Internet / intranet 20 using the setting contents of the specified setting file 13 as a Web information collection condition. Collect web information as follows:
First, the collection control module 111 in the Web information collection system 11 sets the number of links Z, which is the target of information collection starting from the starting page, to an initial value (initial number of stages) n (step S1). In the present embodiment, it is assumed that the initial value n is 2. The starting page URL (starting URL) and the initial value n of the number of steps Z are set in the setting file 13 designated by the management user 102. Next, the collection control module 111 collects the origin page (page information thereof) using, for example, HTTP (Hyper Text Transfer Protocol) (step S2).

  Next, the collection control module 111 checks whether there is an unprocessed collected page (step S3). If there is an unprocessed collected page, one unprocessed collected page is selected (step S4). The text extraction module 112 extracts text from the page (page information) selected by the collection control module 111 (step S5).

  The link label extraction module 113 checks whether an unprocessed link exists in the text extracted by the text extraction module 112 (step S6). If there is an unprocessed link, the link label extraction module 113 selects one unprocessed link (step S7).

  The link label extraction module 113 determines the type of the selected link (step S8). If the link type is a character link, the link label extraction module 113 extracts a character string (link character string) surrounded by character links (tags <A> and </A>) as a link label. (Step S9). On the other hand, if the link type is an image link, the link label extraction module 113 extracts the character string (link character string) of the alt property in the tags <A> and </A> as a link label (step S10). ).

  The link label extraction module 113 extracts the link character string (link label) for all links included in the page selected in step S4 (steps S7, S8, S9 or steps S7, S8, S10). ) Is repeated (step S6). When the operation of extracting the link character string (link label) is executed for all the links in the selected page, control is passed from the link label extraction module 113 to the link determination module 114.

  Then, the link determination module 114 determines whether a link character string including the important word set in the setting file 13 exists in all the link character strings extracted from the page selected in step S4. (Step S11). In the determination in step S11, the synonym dictionary 15 is referred to, and synonyms of the important words set in the setting file 13 are also handled as important words. For example, if the important word set in the setting file 13 is “price” described above, “price”, “list price”, and “fee”, which are synonyms for “price”, are also set as important words in the setting file 13. Are treated as being. It should be noted that “price”, “list price”, and “charge” may be set as important words together with “price” when the setting file 13 is generated. In this case, it is not necessary to refer to the synonym dictionary 15 in the determination in step S11.

  If there is a link character string including an important word (step S11), the link determination module 114 determines whether the currently set link collection stage number Z for the corresponding link series exceeds n (initial stage number). Determination is made (step S12). If Z does not exceed n, that is, if Z matches n, the link determination module 114 updates Z to Z + m, that is, the number of stages n + m that is larger than the initial stage number n (step S13). On the other hand, if Z exceeds n, the link determination module 114 determines whether the number of link stages from the starting page to the link destination is within the currently set number Z of link collection stages (step) S14). If the number of link stages up to the link destination exceeds Z, Z is updated to the number of stages Z + 1 which is 1 greater than the current number of stages (step S15).

The link determination module 114 proceeds to step S16 after executing step S13 or S15. On the other hand, if the number of stages of link collection up to the link destination is within Z (step S14), the link determination module 114 proceeds directly to step S16 without updating Z. In step S16, the keyword generation module 14 checks whether an unprocessed link character string exists in the link character string extracted from the page selected in step S4. If there is an unprocessed link character string as in this example, the link determination module 114 selects one unprocessed link character string (step S17).
The link determination module 114 determines whether the character string (link label) of the link selected in step S17 includes the important word set in the setting file 13 (step S18). In the determination in step S18, the synonym dictionary 15 is referred to, and the synonyms of the important words set in the setting file 13 are also treated as important words.

  If the link character string includes an important word (step S18), the link determination module 114 does not depend on the current stage number Z of link collection related to the link string including the corresponding page (the page selected in step S4). Determine (decide) to collect linked pages. Then, the collection control module 111 collects linked pages (step S21). The “page” (page information) collected by the collection control module 111 is stored in the knowledge DB 121 in combination with a “link source label (link character string)”.

  On the other hand, if the link character string does not include an important word (step S18), the link determination module 114 determines whether the character string includes an unnecessary word set in the setting file 13 (step S19). ). Also in the determination in step S19, the synonym dictionary 15 is referred to in the same manner as in step S11 or S18, and synonyms of unnecessary words set in the setting file 13 are also handled as unnecessary words. Here, if the link character string includes an unnecessary word (step S19), the link determination module 114 determines not to collect the link destination page regardless of the current stage number Z of link collection regarding the link string including the corresponding page. (decide. In this case, the collection control module 111 refrains from collecting the link destination page (step S23). In the present embodiment, when both important words and unnecessary words exist in one link, the important words are prioritized as apparent from the flowchart of FIG. 3 (step S18). ".

  On the other hand, if the link character string does not include an unnecessary word (step S19), that is, if the link character string includes neither an important word nor an unnecessary word (steps S18 and S19), the link determination module 114 links from the starting page to the link destination. It is determined whether the number of link stages up to is within the current set stage number Z (S20). When the link character string does not include an important word or an unnecessary word (steps S18 and S19), the link determination module 114 is limited to the case where the number of link stages from the start page to the link destination is within the current set stage number Z. (Step S20), the collection control module 111 is requested to collect linked pages. The collection control module 111 collects linked pages in response to a request from the link determination module 114 (step S21). On the other hand, if the link character string does not include an important word or an unnecessary word (steps S18 and S19), and the number of links from the starting page to the link destination exceeds the currently set number Z (step S20). The link determination module 114 requests the collection control module 111 not to collect the link destination page. Thereby, the collection control module 111 refrains from collecting linked pages (step S23). In this case, control is passed from the collection control module 111 to the link determination module 114, and the determination in step S16 is performed.

  When the collection control module 111 collects the link destination page (page information) in step S21, the collection control module 111 determines whether the total number of collection pages of the corresponding link exceeds the set upper limit of the number of collection pages (step S22). . If the total number of collection pages does not exceed the upper limit of the set number of collection pages, control is passed from the collection control module 111 to the link determination module 114, and the determination in step S16 is performed.

  In this way, all link character strings extracted from the page selected in step S4 are linked based on link determination processing for whether important words, unnecessary words or other words are included, and the determination results. When collection control for collecting or not collecting the previous page is executed, the collection control module 111 performs the determination in step S3 again unless the total number of collection pages exceeds the set upper limit of the number of collection pages. . When an unprocessed collected page remains, the process starting from step S4 is performed on the page. That is, the processing starting from step S4 is repeated until the total number of collected pages exceeds the upper limit of the number of collected pages (step S22) or until all the links on the collected pages have been processed (step S3).

  As described above, in the present embodiment, when the link character string (link label) in the page includes an unnecessary word (step S19), the link destination page is likely to be useless for the user. Regardless of the set stage number Z and the number of link stages up to the link destination (that is, even if the link destination is within the range of the set stage number Z), collection of the link destination page is suppressed (step S23).

  Further, in the state where the current stage number Z of link collection is the initial stage number n, there are links including important words on the pages in the corresponding link string (that is, pages up to n stages) (steps S11 and S12). The number Z of link collection stages connected to the page is increased from Z = n to Z = Z + m = n + m (step S13). Similarly, when the current stage number Z of link collection exceeds the initial stage number n and there is a link including an important word on the Z-th page in the corresponding link string, that is, the number of stages to the link destination is When Z is exceeded (steps S11, S12, and S14), the number Z of link collections connected to the Z-th page is increased by 1 (step S15). As a result, when the link character string (link label) in the page includes an unnecessary word (step S18), the link destination page is likely to be useful to the user, so the link destination is the initial stage number n of link collection. However, the collection of linked pages is suppressed (step S21).

  Further, in this embodiment, when the link label extracted from the page does not include any keyword of the important word and the unnecessary word (steps S18 and S19), the number of link stages up to the link destination corresponds to that point in time. Only when it is within the set number Z of link collections to be performed (step S20), linked pages are collected (step S21). In other words, if the link label does not include either an important word or an unnecessary word (steps S18 and S19), and the number of link stages up to the link destination is not within the set number Z of the corresponding link collection at that time ( In step S20), the linked page is not collected (step S23).

  Next, a specific example of the above-described Web information collection processing will be described with reference to a diagram showing an example of link determination / page collection in FIG. FIG. 4 shows a starting page Ps specified by the starting URL set in the setting file 13. In this page Ps, there are links to pages P11, P12, and P13 in the first stage starting from the page Ps. The links (labels) to the pages P11, P12, and P13 include words indicated by symbols ◯, Δ, and X, respectively. In the setting file 13, words indicated by symbols ◯ and X are set as important words and unnecessary words, respectively. In the following description, for simplification, it is assumed that ◯ and X are important words and unnecessary words, respectively. Further, it is assumed that Δ is a word that is neither ○ nor X. In this case, of the first-stage pages P11, P12, and P13, the page P13 ahead of the link including x is within the range of the set stage number Z = 2 (n = 2) of the link collection at that time. Regardless, they are not collected. On the other hand, the page P12 ahead of the link including Δ is collected because it is within the range of the link collection setting stage number Z = 2 (n = 2). Further, the page P11 ahead of the link including the circle is collected regardless of the set number Z of link collections.

  The page P11 is linked to the second-stage pages P210 and P211 corresponding to ◯ and Δ. In this case, the number of links Z to be collected from the page Ps and the link sequence including the page P11 is increased from 2 (n) to 2 + 2 (n + m). As a result, even though the second page P210 and the third page P310 linked to the page P210 have only links including Δ, the third page after the link P310 and the fourth page P410 are collected. This page P410 has only a link including Δ. The page P510 of the fifth stage ahead of this link exceeds the number of linked stages Z = 4 at that time. For this reason, the page P510 is not collected. In the present embodiment, the number of links to be collected regarding the link string including the page P11 is increased only once.

  On the other hand, the third page P311 linked to the page P211 includes a link including Δ and a link including ○, and the fourth page P411 and P412 at the end of the link are collected together. There is only a link including Δ on page P411. The page P511 of the fifth stage ahead of this link exceeds the number of link stages Z = 4 to be collected at that time. For this reason, the page P511 is not collected.

  On the other hand, the page P412 includes a link including Δ and a link including ○ '. ○ ′ is a synonym for ○. That is, the page P412 has a link including an important word. Here, the page P412 is the fourth stage from the starting page Ps, and the number of stages to the link destination of the page P412 exceeds the number of links Z to collect. However, since there is a link including an important word on the page P412, Z is incremented by 1 to Z = 5, and the fifth page P513, P514 linked to the page P412 is collected. There is only a link including Δ on page P513. The page P613 at the sixth stage after this link exceeds the number of link stages Z to collect at that time. For this reason, the page P613 is not collected. On the other hand, the page P514 includes a link including “◯”. Therefore, the sixth page P614 at the end of this link is collected.

  In the above embodiment, there is a link including an important word on the pages from the starting page to n (= 2) stages, and the set number Z of link collections regarding the link string including the link does not exceed n ( When Z is the initial stage number n), Z is increased by m (= 2). However, when there is a link including an important word on a page in an arbitrary stage, and the number Z of link collection settings for the link string including the link is n or more, the Z may be increased by one. Absent.

Next, the operation of the knowledge search system 12 centering on the log statistics generation module 124 will be described.
When the search engine 122 in the knowledge search system 12 receives a search request according to the operation of the user 101 via the Web browser 16, the search engine 122 searches the knowledge DB 121 for document information (page information) that matches the search expression indicated by the search request. . At this time, the search engine 122 stores the search expression in the search log 123.

  When searching for document information that matches the search expression, the search engine 122 generates image information of a list of search results and presents it to the user 101 via the Web browser 16. This list of search results includes, for example, ID (information ID) of each document information that matches the search formula indicated by the search request. It is also possible to include a summary of each searched document information in the search result list. The user 101 selects the information ID of the document information that he / she wants to refer to from the list of search results. When the search engine 122 detects that the information ID is selected from the list of search results, the search engine 122 reads the document information indicated by the selected ID from the knowledge DB 121 and presents the document information to the user 101 via the Web browser 16. To do.

  The search log 123 includes a log table 123a having a data structure shown in FIG. Each entry in the log table 123a is linked to each document information (page information) stored in the knowledge DB 121 in combination with the document information ID (information ID) and the document information. The original label (link character string), the number of times the document information is referred to by the user (reference number), and the user's evaluation result regarding the document information are set. The evaluation result item includes items of an important evaluation count added by 1 when the document information is important (useful) and an unnecessary evaluation count added by 1 when the document information is unnecessary (useless).

  In the present embodiment, when document information (page information) is stored in the knowledge DB 121, the link source stored in the knowledge DB 121 is paired with the document information ID and the document information (page information). Entry information including a label (link character string) is generated and stored in the log table 123a. At this time, the reference count, the important evaluation count, and the unnecessary evaluation count in the entry information are all initialized to zero.

  The number of times of reference in the entry information of the log table 123a is obtained when the ID of the corresponding document information is selected by the user 101 from the list of search results, and the document information indicated by the selected ID is referred to by the user 101. Incremented by 1 by the search engine 122. When document information is referred to, the search engine 122 requests the user 101 to input an evaluation result indicating whether the document information is important (useful) or unnecessary (useless). If “important” is input (selected) as the evaluation result, the number of important evaluations in the corresponding entry information in the log table 123a is incremented by one. On the other hand, when “unnecessary” is input (selected) as the evaluation result, the number of unnecessary evaluations in the corresponding entry information in the log table 123a is incremented by one.

  The search log 123 also includes a log statistics table 123b having a data structure shown in FIG. Each entry in the log statistics table 123b is stored in the search log 123. For each word (phrase) that appears in the search formula used for searching the document information, the number of occurrences of the search formula, the importance evaluation ratio, and the unnecessary Each item includes an evaluation ratio, appearance frequency rank, important evaluation rank, unnecessary evaluation rank, importance, and determination result. The search expression appearance count indicates the number of times the corresponding word appears in the search expression. The important evaluation ratio and the unnecessary evaluation ratio indicate the ratio of the important evaluation count and the unnecessary evaluation count to the reference count of the corresponding word, respectively. The appearance frequency rank indicates the rank of the search expression appearance frequency. The important evaluation rank and the unnecessary evaluation rank indicate the ranks of the “important” evaluation ratio and the “unnecessary” evaluation ratio, respectively. The importance level indicates the importance level of the corresponding word. The determination result indicates whether the word is an “important word”, an “unnecessary word”, or neither, which is determined from the importance of the corresponding word.

  In the present embodiment, the log statistics table 123b is periodically generated by the log statistics generation module 124. Hereinafter, the log statistics table generation processing by the log statistics generation module 124 will be described with reference to the flowcharts of FIGS.

  First, the log statistics generation module 124 extracts one unprocessed search expression from the search expressions stored in the search log 123 (step S31). Next, the log statistics generation module 124 extracts unprocessed words (phrases) that appear in the retrieved search expression (Step S32). If a word not stored in the log statistics table 123b is extracted (step S33), the log statistics generation module 124 generates entry information of the log statistics table 123b including the word (step S34). At this time, the number of appearances of the search expression in the entry information is initialized to 1, and the other items are blank. If entry information including a word extracted from the search formula is already stored in the log statistics table 123b (step S33), the log statistics generation module 124 increments the search formula appearance count in the entry information by one. (Step S35). The search expression appearance count of each entry information in the log statistics table 123b is frequency information (statistical information) obtained by analyzing words appearing in the search expression used for the search.

  Next, the log statistic generation module 124 determines whether or not an unprocessed word exists in the search expression extracted in step S31 (step S36). If there is an unprocessed word, the log statistics generation module 124 returns to the process of step S32. On the other hand, if there is no unprocessed word, the log statistics generation module 124 determines whether there is an unprocessed search expression in the search log 123 (step S37). If there is an unprocessed search expression, the log statistics generation module 124 returns to the process of step S31.

  When the log statistics generation module 124 finishes processing all the search expressions stored in the search log 123 (step S37), the important evaluation ratio and the unnecessary evaluation ratio in each entry information in the log statistics table 123b The appearance frequency rank, the important evaluation rank, the unnecessary evaluation rank, the importance, and the determination result are determined as follows.

  First, the log statistics generation module 124 selects one unprocessed entry information from the log statistics table 123b (step S38). Next, the log statistics generation module 124 reads a word in the selected entry information (step S39). Next, the log statistics generation module 124 searches the log table 123a for entry information in which the link character string including the word read in step S39 is set, and sets the reference count, the important evaluation count, and the unnecessary evaluation count. Reference is made (steps S40 and S41). Then, the log statistics generation module 124 calculates the ratio (%) of each of the important evaluation count and the unnecessary evaluation count with respect to the reference count, and sets it in the corresponding entry information in the log statistics table 123b (step S42). Here, for the words that are commonly included in different link character strings ("price" in the example of FIG. 4), all the reference counts and the evaluation counts (important evaluation count and unnecessary evaluation count) are summed and referred to The ratio (%) of the number of important evaluations and the number of unnecessary evaluations to the number of times is calculated. The log statistics generation module 124 repeats the above-described processing of steps S38 to S43 for all entry information in the log statistics table 123b (step S44).

  Next, the log statistics generation module 124 ranks the search expression appearance counts so that all the entry information items in the log statistics table 123b are in descending order of the search expression appearance counts, and appears in all the entry information items. The frequency ranking is set (step S45). Similarly, the log statistics generation module 124 ranks the important evaluation ratios so that the importance evaluation ratios of all the entry information in the log statistics table 123b are in descending order, and sets the important evaluation ranks in all the entry information. Set (step S46). In this case, the higher the ranking, the higher the importance evaluation ratio. Similarly, the log statistics generation module 124 ranks the unnecessary evaluation ratios so that the unnecessary evaluation ratios of all the entry information in the log statistics table 123b are in ascending order, and sets the unnecessary evaluation ranks in all the entry information. Set (step S47). In this case, the lower the ranking, the higher the unnecessary evaluation ratio.

  When executing the above steps S45 to S46, the log statistics generation module 124 calculates the sum of the appearance frequency rank, the important evaluation rank, and the unnecessary evaluation rank for each entry information in the log statistics table 123b, and calculates the total value. It is set as “importance” in the entry information (step S48). In other words, the log statistics generation module 124 comprehensively evaluates the appearance frequency rank, the important evaluation rank, and the unnecessary evaluation rank, and determines the “importance” of the corresponding word. Here, the greater the number of times the corresponding word appears in the search expression, the higher the importance evaluation rank, and the lower the unnecessary evaluation rank, the higher the “importance” (that is, the “unnecessity” becomes lower). . Similarly, the smaller the number of times the corresponding word appears in the search expression, the lower the importance evaluation rank, and the higher the unnecessary evaluation rank, the lower the “importance” (that is, the “unnecessity” becomes higher). . In this way, “importance” is equivalent to representing “unnecessity” in a different way of looking.

  Next, the log statistic generation module 124 determines “importance” in the entry information for each word in the log statistic table 123b, and sets words belonging to the higher X% (for example, 20%) as candidates for “important words”. The words belonging to the lower Y% (for example, 20%) are set as candidates for “unnecessary words” (step S49). The keyword generation module 14 executes a process for determining candidates for the “important words” and “unnecessary words” (step S49) and a process for calculating the “importance” for each word (step S48). It may be a configuration.

  As described above, in the present embodiment, (1) words / phrases appearing in a search expression, (2) the number of reference times of a search result, and (3) a user's evaluation for each referenced information is statistically analyzed and processed The “important word” candidates and the “unnecessary word” candidates are automatically determined. Here, words that are frequently used in search expressions, frequently referenced, or included in highly evaluated information are candidates for “important words” as keywords for collecting information needed by the user. The In addition, “unnecessary words” as keywords for preventing information that is not required by the user from being collected from words or phrases included in information with low frequency of appearance and low reference frequency or low evaluation. Candidate When generating the log statistics table 123b, the synonym dictionary 15 may be used to replace the synonym with a representative word, for example, and merge it into one entry information.

  The keyword generation module 14 generates the setting file 13 from the management user 102 periodically or from the management user 102 with the list of “importance” candidates and the list of “unnecessary words” candidates determined by the log statistics generation module 124. Is requested to the management user 102 via the web browser 17 and the management user 102 selects “importance” and “unnecessary words” from the list, thereby “importance”. And “unnecessary words” are determined. The keyword generation module 14 generates the setting file 13 in which the selected “importance” and “unnecessary words” are set. Of course, the “importance” candidate and the “unnecessary word” candidate may be automatically determined as “importance” and “unnecessary word”, respectively.

  FIG. 1 shows only one knowledge DB 121. However, the knowledge search system 12 has a plurality of knowledge DBs 121 in many cases. In this case, the setting file 13 may include information specifying the knowledge collection destination knowledge DB 121. In the setting file 13, “character string pattern of URL to be collected” and “character string pattern of URL not to be collected” described in Patent Document 1 are set. May be specified.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment.

The block diagram which shows the structure of the knowledge management system which implement | achieves the knowledge information collection system which concerns on one Embodiment of this invention. The figure which shows a part of flowchart for demonstrating the web information collection process in the embodiment. The figure which shows the remainder of the flowchart for demonstrating the web information collection process in the embodiment. The figure which shows an example of link determination / page collection in Web information collection processing. The figure which shows the data structure example of the log table 123a. The figure which shows the example of a data structure of the log statistics table 123b. The figure which shows a part of flowchart for demonstrating the log statistics table production | generation process in the embodiment. The figure which shows the remainder of the flowchart for demonstrating the log statistics table production | generation process in the embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 11 ... Web information collection system, 12 ... Knowledge search system, 13 ... Setting file (setting means), 14 ... Keyword generation module, 15 ... Synonym dictionary, 20 ... Internet / intranet (network), 111 ... Collection control module (information) (Collection control means), 112 ... text extraction module, 113 ... link label extraction module (link character string extraction means), 114 ... link determination module, 122 ... search engine (search means, means for evaluation), 123 ... search log (search) Log accumulation means), 123a ... log table, 123b ... log statistics table, 124 ... log statistics generation module.

Claims (11)

In a knowledge information collection system that collects information to be registered in the knowledge database from the network,
Sets starting location information indicating the location of page information that is the starting point of information collection from the network, and the number of links to be collected from the network, and is not subject to information collection from the network A setting means for setting a word related to a link to be a keyword representing an unnecessary word,
Link character string extracting means for extracting a link character string from page information collected from the network;
Link determination means for determining whether or not link destination page information is useless from the extracted link character string and the keyword representing the set unnecessary word;
Information collection control means for controlling information collection from the network by following a link from the set origin location information, and the link destination page information determined to be useless by the link determination means, An information collection control means that excludes information from collection even within the set number of link stages. In a knowledge information collection system that collects information to be registered in the knowledge database from the network,
Setting the starting location information indicating the location of page information that is the starting point of information collection from the network and the number of links to be collected from the network, and the information collection target from the network A setting means for setting a word related to a link to be used as a keyword representing an important word,
Link character string extracting means for extracting a link character string from page information collected from the network;
Link determination means for determining whether link destination page information is useful from the extracted link character string and the keyword representing the set important word;
Information collection control means for controlling information collection from the network by following a link from the set origin location information, the page information of the link destination determined to be useful by the link determination means, A knowledge information collection system comprising: information collection control means for collection even if the range of the set number of stages is exceeded. In a knowledge information collection system that collects information to be registered in the knowledge database from the network,
Setting the starting location information indicating the location of page information that is the starting point of information collection from the network and the number of links to be collected from the network, and the information collection target from the network Setting means for setting each word related to a link to be used as a keyword representing an important word, and each word related to a link to be excluded from information collection from the network as a keyword representing an unnecessary word;
Link character string extracting means for extracting a link character string from page information collected from the network;
From the extracted link character string and the keyword representing the set important word, it is determined whether link destination page information is useful, and the extracted link character string and the set unnecessary word are determined. A link determination means for determining whether or not link destination page information is useless from
Information collection control means for controlling information collection from the network by following a link from the set origin location information, the page information of the link destination determined to be useful by the link determination means, Even if the set number of steps is exceeded, the page information of the link destination determined to be useless by the link determining means is collected even if it is within the set number of steps. A knowledge information collection system comprising: an information collection control unit that is not a target. Search means for searching the information collected in the knowledge database according to a given search formula, and presenting the search results to the user;
Means for allowing the user to evaluate the usefulness or uselessness of the information collected in the knowledge database referred to according to the search result by the search means;
The search formula used for the search by the search means, the number of times the information is collected for each piece of information collected in the knowledge database, and the user's evaluation result for the information for each piece of information collected in the knowledge database And a search log storage means for storing as a search log,
Analyzing words and phrases appearing in the search formula stored in the search log storage means, the number of references for each information stored in the search log storage means, and the evaluation results for each information stored in the search log storage means Log statistic generation means for generating an evaluation value representing a degree to which the word is important or unnecessary for the user for each word appearing in the search expression;
A keyword generation unit that generates a keyword representing an unnecessary word that can be set by the setting unit based on the evaluation value for each of the phrases generated by the log statistics generation unit. The knowledge information collection system according to 1 or 3.   Further comprising means for presenting a list of keywords representing unnecessary words generated by the keyword generating means to the user so that the user can select keywords representing unnecessary words set by the setting means from the list. The knowledge information collecting system according to claim 4. Search means for searching the information collected in the knowledge database according to a given search formula, and presenting the search results to the user;
Means for allowing the user to evaluate the usefulness or uselessness of the information collected in the knowledge database referred to according to the search result by the search means;
The search formula used for the search by the search means, the number of times the information is collected for each piece of information collected in the knowledge database, and the user's evaluation result for the information for each piece of information collected in the knowledge database And a search log storage means for storing as a search log,
Analyzing words and phrases appearing in the search formula stored in the search log storage means, the number of references for each information stored in the search log storage means, and the evaluation results for each information stored in the search log storage means Log statistic generation means for generating an evaluation value representing a degree to which the word is important or unnecessary for the user for each word appearing in the search expression;
The method further comprises: keyword generating means for generating a keyword representing an important word that can be set by the setting means based on the evaluation value for each of the phrases generated by the log statistics generating means. The knowledge information collection system according to 2 or 3.   Further comprising means for presenting a list of keywords representing the important words generated by the keyword generating means to the user so that the user can select keywords representing the important words set by the setting means from the list. The knowledge information collection system according to claim 6. A knowledge information collecting method applied to a knowledge information collecting system for collecting information to be registered in a knowledge database from a network,
The starting location information indicating the location of page information that is the starting point of information collection from the network, the number of links to be collected from the network, and the information that is not collected from the network Generating a configuration file in which keywords representing unnecessary words related to power links are set;
Collecting information from the network by following links from the set origin location information;
Extracting a link character string from page information collected from the network;
Determining whether the link destination page information is useless from the extracted link character string and a keyword representing an unnecessary word set in the setting file,
The step of collecting the information includes the step of excluding the link destination page information determined to be useless even if the link destination page information is within the set number of stages. Knowledge information collection method characterized by A knowledge information collecting method applied to a knowledge information collecting system for collecting information to be registered in a knowledge database from a network,
The starting location information indicating the location of page information that is the starting point of information collection from the network, the number of link stages that are the target of information collection from the network, and the information collecting target from the network Generating a configuration file with keywords representing key words related to the link;
Collecting information from the network by following links from the set origin location information;
Extracting a link character string from page information collected from the network;
Determining whether the link destination page information is useful from the extracted link character string and a keyword representing an important word set in the setting file,
The step of collecting the information includes a step of collecting the linked page information determined to be useful even if the linked page information exceeds the set number of steps. Characteristic knowledge information collection method. A program executed by a knowledge information collection system that collects information to be registered in the knowledge database from the network,
On the computer,
The starting location information indicating the location of page information that is the starting point of information collection from the network, the number of links to be collected from the network, and the information that is not collected from the network Generating a configuration file in which keywords representing unnecessary words related to power links are set;
Extracting a link character string from page information collected from the network by following a link from the set origin location information;
Determining whether or not link destination page information is useless from the extracted link character string and a keyword representing an unnecessary word set in the setting file;
The step of collecting information from the network by following links from the set origin location information, wherein the link destination page information determined to be useless is the set page information. A program that executes steps that are not subject to collection even within the range of the number of steps. A program executed by a knowledge information collection system that collects information to be registered in the knowledge database from the network,
On the computer,
The starting location information indicating the location of page information that is the starting point of information collection from the network, the number of link stages that are the target of information collection from the network, and the information collecting target from the network Generating a configuration file with keywords representing key words related to the link;
Extracting a link character string from page information collected from the network by following a link from the set origin location information;
Determining whether link destination page information is useful from the extracted link character string and a keyword representing an important word set in the setting file;
Collecting information from the network by following links from the set origin location information, wherein the link destination page information determined to be useful is the set page information. A program for executing the steps to be collected even if the number of steps exceeds the range.

Images