JP4128212B1 - Relevance calculation system between keywords and relevance calculation method - Google Patents

Abstract

A system capable of efficiently calculating the degree of association between all types of keywords based on the co-occurrence between keywords.
A keyword extracting unit that extracts keywords from a plurality of document files, and a degree of association between a pair of keywords is calculated for every keyword combination based on the appearance frequency of each keyword in each document file. The relevance calculation system 10 between keywords provided with the relevance calculation part 18 stored in relevance table DB26. The degree-of-relevance calculation unit 18 calculates the appearance frequency of keywords that have appeared in document file units, calculates the square value of the appearance frequency of each keyword, totals the square values over all document files, Calculate the product value of the appearance frequency between a pair of keywords in units, total the product values across all document files, calculate the square root of the sum of the square values of each keyword, add both square roots, and add the sum The degree of relevance is calculated by dividing the sum of product values between the keywords.
[Selection] Figure 1

Description

  The present invention relates to a relevance calculation system and a relevance calculation method between keywords, in particular, to extract a term that is closely related to an input search term, or a company or product that is closely related to an extracted term, The present invention relates to a technique for calculating the degree of association between keywords, which is indispensable for realizing an associative search for presenting people and the like.

A search system is used to extract necessary information from a vast amount of information. In the case of a general search system, there is a mechanism for extracting information that contains the same or similar concept as the input search term. ing. For example, if a search term “Fuji” is given to a database that stores information on a large number of companies, the search system can accurately output a list of companies that include the character string “Fuji” in the name. If you enter "environmental problem" at a search site on the Internet, a list of Web pages that contain the text "environmental problem" is displayed on the display.
As a result, the user can reach the target information, but the search results there are only in the expected range, and even if you look at the search result list, you can expect unexpected discoveries. There wasn't. Of course, new knowledge can be obtained in the process of examining details of individual data in the search result list, but information including other terms closely related to the search term cannot be extracted directly.

In this regard, in the case of the “associative search system” disclosed in Patent Document 1, the related term storage means that stores the related terms of each term and the co-occurrence with each term (appear in the same document) It has a co-occurrence company name storage means that stores company names (high probability). When a search term is entered, it extracts terms related to it and extracts company names with high co-occurrence for each term. It has a mechanism to do.
JP 2004-110386 A

  As a result, when users enter "environmental problems" as a search term, it becomes possible to directly list the names of companies that often appear in documents related to environmental problems. Recognize and connect with investment behavior.

  However, in the case of this associative search system, the object of associative search is limited to the company name (including the related company name).

  That is, in this conventional associative search system, after extracting keywords from text information such as securities reports and newspaper articles, whether or not each keyword corresponds to a company name with reference to the company information storage unit. In the case of a company name, the company name is stored in the co-occurrence company name storage means as being related to other keywords appearing in the same text information.

For this reason, if you enter the search term Internet, you can pick up a company name that has a strong co-occurrence with "Internet", but pick up people, regions, technologies, etc. that have a strong co-occurrence with "Internet". I can't.
Patent Document 1 shows an example in which search results of “broadband”, “network”, and “e-mail” can be obtained in response to input of “Internet”, but these are related as “Internet” thesaurus to the last. It is prepared in advance in the term storage means and is not extracted based on the co-occurrence with the “Internet”.

  The present invention has been devised to solve the above problem, and based on the co-occurrence between all keywords including a company name, a system capable of efficiently calculating the degree of association between a pair of keywords. It is intended to be realized.

In order to achieve the above object, a relevance calculation system between keywords described in claim 1 extracts a plurality of keywords from the document storage means storing a plurality of document files and each of the document files. Based on the keyword extraction means stored in the storage means and the frequency of appearance of each keyword in each document file, the degree of association between a pair of keywords is calculated for all keyword combinations and stored in the keyword relevance degree storage means A degree calculating means, wherein the relevance degree calculating means (1) detects a keyword having a history of appearance in the document file in units of document files, and calculates the frequency of appearance thereof (2 ) The process of calculating the square value of the appearance frequency of each keyword, and (3) totaling the square value of the appearance frequency of each keyword, and summing up the total over all document files Processing for output, (4) in the document file units, the pair of keywords, the product of the frequency of occurrence of each keyword, the process of calculating a product value of the frequency between a pair of keywords (5) between the keywords The product of the appearance frequencies of the document, and calculating the sum over all document files; (6) the process of calculating the square root of the sum of (3) above; and (7) the above (6) of the pair of keywords. And calculating the degree of relevance between the two keywords by adding the square roots of each other and dividing the sum of (5) by the sum.
Note that the order of the processes (1) to (7) may be appropriately changed as long as no logical contradiction occurs.

The relevance calculation system between keywords described in claim 2 is a relevance calculation system between keywords comprising a management server, a plurality of first distributed processing servers, and a second distributed processing server, The management server distributes a plurality of document files stored in the document storage means to each first distributed processing server, and stores the keyword transmitted from each first distributed processing server in the keyword storage means. Means for transmitting all the keywords stored in the keyword storage means to the first distributed processing server, and a plurality of appearance frequency square value files transmitted from the first distributed processing servers, A means for transmitting to the second distributed processing server, and a plurality of types of combination frequency product value files transmitted from each of the first distributed processing servers according to the type of the first Means for distributing and distributing to the distributed processing server; means for storing, in the keyword frequency total table storage means, a sum total of all squares of the appearance frequency of each keyword transmitted from the second distributed processing server; Means for storing the sum of the product values of the appearance frequencies between the keywords transmitted from each first distributed processing server over all document files in the keyword combination frequency sum table storage means; From the keyword retrieval means, the keyword combination frequency sum table storage means , the means for retrieving the sum of the product values of the appearance frequencies between the keywords for the pair of keywords, and the keyword frequency sum table storage means from the pair of keywords . for keywords, and means for taking the sum of the square of the frequency of occurrence of each keyword, the square root of this sum it Means for calculating the degree of association between the two keywords by adding both square roots and dividing the sum of the product values of the appearance frequencies between the keywords by the sum. The keyword extraction means for the server to extract keywords from the responsible document file distributed by the management server, the means for transmitting each keyword to the management server, and when all keywords are transmitted from the management server, Means for detecting the presence / absence of a keyword for each document file, means for calculating a square value of the appearance frequency of a keyword with an appearance record, and describing for each document file in the appearance frequency square value file, and a pair of keywords with an appearance record Means for generating keyword combinations in which the first character code of the first character is placed first, and for each combination For each pair of keywords, means for calculating the product of the appearance frequencies of each keyword as a product value of the appearance frequencies between the pair of keywords, the character code of the first character of the first keyword, and a character code range assigned in advance Means for identifying the combination frequency product value file to be described by comparing the assigned character code ranges of the plurality of combination frequency product value files, and the product value in the corresponding combination frequency product value file for each document file. Means for transmitting the appearance frequency square value file and the plurality of types of combination frequency product value files to the management server, and when a plurality of the same type of combination frequency product value files are transmitted from the management server, The means for concatenating the combination frequency product file and the combination of keywords described in the concatenation file correspond to the character code of each keyword. The second distributed processing server, and means for summing up product values in units of combinations of the same keywords, calculating a sum total over all document files, and means for transmitting the sum to the management server. However, when a plurality of appearance frequency square value files are transmitted from the management server, the means for concatenating the appearance frequency square value files and the keywords described in the concatenation file are sorted according to their character codes. And a means for calculating the sum of squares of appearance frequencies in units of the same keyword, calculating a sum total over all document files, and a means for transmitting the sum to the management server.
The system according to claim 2 includes a plurality of distributed processes including a keyword extraction process, a keyword appearance frequency square value file generation process, an appearance frequency product value file generation process between keywords, and a sum calculation process for all appearance frequency product values over all documents. It is characterized in that it is distributed by the server and the summation calculation process for all the documents of the appearance frequency square value is processed by a single distributed processing server.

The relevance calculation system between keywords described in claim 3 is a relevance calculation system between keywords comprising a management server and a plurality of distributed processing servers, wherein the management server is stored in the document storage means. Means for distributing the plurality of document files to a plurality of first distributed processing servers comprising at least a part of the plurality of distributed processing servers, and a plurality of keywords stored in the keyword storage means. Means for transmitting to each of the distributed processing servers, and a plurality of types of combination frequency product value files transmitted from each of the first distributed processing servers; Between each keyword transmitted from each second distributed processing server and means for distributing and distributing to each distributed processing server according to the type in charge of each of the two distributed processing servers Means for storing the sum of product values of appearance frequencies over all document files in the keyword combination frequency sum table storage means, and a plurality of document files stored in the document storage means in the plurality of distributed processing servers. Means for distributing to at least a portion of a plurality of third distributed processing servers, means for transmitting a plurality of keywords stored in the keyword storage means to each of the third distributed processing servers, and each third A plurality of types of appearance frequency square value files transmitted from the plurality of distributed processing servers are classified into a plurality of fourth distributed processing servers consisting of at least a part of the plurality of distributed processing servers according to their respective responsibilities. The distribution sum for each keyword and the sum total over all document files of the square value of the frequency of appearance of each keyword transmitted from each fourth distributed processing server Means for storing the de-frequency sum table storage means, and means for taking out the keyword from the storage means of a pair keywords from the keyword combination frequency sum table storage means for the pair of keywords, the product value of the frequency between the keywords And a means for extracting the sum of squares of the appearance frequency of each keyword for the pair of keywords from the keyword frequency total table storage means , calculating a square root of the sum, and calculating both square roots. Means for calculating the relevance between the two keywords by adding and dividing the sum of the product values of the appearance frequencies between the keywords by the sum, and each of the first distributed processing servers is configured by the management server For the distributed document file, there is a means for detecting the existence of each keyword for each document file and the appearance history. Means for generating a combination of keywords in which the first character code of the first character is arranged first among a pair of keywords, and the product of the appearance frequency of each keyword for each pair of keywords A means for calculating a product value of appearance frequencies between a pair of keywords, a character code of the first character of the first keyword, and a character code range in charge of a plurality of combination frequency product value files to which a character code range is assigned in advance For identifying a combination frequency product value file to be described, a means for describing the product value in a corresponding combination frequency product value file for each document file, and a plurality of types of combination frequency product value files Each of the second distributed processing servers includes a plurality of similar combination frequency product value files from the management server. Is transmitted, means for concatenating each combination frequency product value file, means for sorting the keyword combinations described in the concatenated file according to the character code of each keyword, and the same keyword combination unit Each of the above-mentioned third distributed processing servers is in charge of distribution by the management server, each of which has means for summing up product values and calculating a sum total over all document files and means for transmitting the sum to the management server. For a document file, a means for detecting the presence or absence of each keyword for each document file, a means for calculating a square value of the appearance frequency of a keyword with an appearance record, a character code of each keyword, and a character code range are assigned in advance. A method for identifying the appearance frequency square value file to be described by comparing the character code ranges of multiple appearance frequency square value files. And means for describing the square value for each document file in the corresponding appearance frequency square value file, and means for transmitting the plurality of types of appearance frequency square value files to the management server, respectively. When a plurality of the same type appearance frequency square value files are transmitted from the management server, the distributed processing server in FIG. 6 connects the appearance frequency square value files and the keywords described in the concatenation file with respective characters. A means for sorting according to the code, a means for calculating the square value of the appearance frequency in the same keyword unit, calculating a sum total over all document files, and a means for transmitting the sum to the management server. It is a feature.
The system according to claim 3 includes an appearance frequency product value file generation process between keywords, a sum calculation process for all appearance frequency product values, a keyword appearance frequency square value file generation process, and an appearance frequency square value all documents. Each of the total sum calculation processes is characterized in that it is distributed by a plurality of distributed processing servers.
The first to fourth distributed processing servers are logical divisions focusing on the respective functions, and when the distributed processing servers are physically independent, of course, they overlap each other. There is also a possibility. If there is a physical overlap, sending and receiving files between servers need only be performed as much as necessary. For example, when the first distributed processing server and the third distributed processing server are configured by a common server machine, the management server only needs to distribute the responsible document file to the first server, and the third distributed processing Duplicate delivery of the same document file to the server can be omitted (the same applies to the inventions of claims 4 to 6).

The inter-keyword relevance calculation system according to claim 4 is a keyword relevance calculation system comprising a management server and a plurality of distributed processing servers, wherein the management server is stored in the document storage means. Means for distributing the plurality of document files to a plurality of first distributed processing servers comprising at least a part of the plurality of distributed processing servers, and a plurality of keywords stored in the keyword storage means. The means for transmitting to each of the distributed processing servers and the combined frequency product value file transmitted from each of the first distributed processing servers are sent to the second distributed processing server that is one of the plurality of distributed processing servers. The sum of the product values of the appearance frequencies between the keywords transmitted from the second distributed processing server and all document files is displayed as a keyword combination frequency sum total. Means for storing in the means, means for distributing the plurality of document files stored in the document storage means to a plurality of third distributed processing servers comprising at least a part of the plurality of distributed processing servers, Means for transmitting a plurality of keywords stored in the keyword storage means to each of the third distributed processing servers, and an appearance frequency square value file transmitted from each of the third distributed processing servers. A means for transmitting to a fourth distributed processing server that is one of the servers, and a sum of the square values of the appearance frequencies of the keywords transmitted from the fourth distributed processing server over all document files means for storing the sum table storage means, and means for retrieving a pair of keywords from the keyword storage unit, from the keyword combination frequency sum table storage means, the For pairs of keywords, and means for taking the sum of the product values of frequency between each keyword from the keyword frequency summation table storage means for the pair of keywords, and means for taking the sum of the square of the frequency of occurrence of the keywords And calculating the relevance between the two keywords by calculating the square root of each sum, adding both square roots, and dividing the sum of the product values of the appearance frequencies between the keywords by this sum, Each of the first distributed processing servers is configured to detect the presence or absence of each keyword for each document file with respect to the document file distributed by the management server, and the first character between a pair of keywords having a history of appearance. Means for generating a combination of keywords in which the younger code is placed first, and a pair of keywords for each combination , Means for calculating a product of appearance frequencies of each keyword as a product value of appearance frequencies between a pair of keywords, means for describing the product value in a combination frequency product value file for each document file, and the combination frequency product Means for transmitting the value file to the management server,
When the second distributed processing server transmits a plurality of combination frequency product value files from the management server, a means for concatenating each combination frequency product value file, and a combination of keywords described in the concatenation file, Means for sorting according to the character code of each keyword, means for totalizing product values in combination units of the same keyword, calculating a sum total over all document files, means for transmitting this sum to the management server, Each of the third distributed processing servers calculates means for detecting the presence or absence of each keyword for each document file for the document file distributed by the management server, and calculates the square value of the appearance frequency of the keyword having the appearance record. Means for describing the square value for each document file in the appearance frequency square value file, and the appearance frequency square value file is stored in the management server. And means for connecting the appearance frequency square value files when the fourth distributed processing server transmits a plurality of appearance frequency square value files from the management server, and is described in the connection file. A means for sorting the generated keywords according to their character codes, a means for calculating the square value of the appearance frequency in the same keyword unit, calculating a sum total over all document files, and transmitting the sum to the management server. And a means.
In the system according to claim 4, the appearance frequency product value file generation processing between keywords and the keyword appearance frequency square value file generation processing are distributed by a plurality of distributed processing servers, and the sum of the appearance frequency product values over all documents is calculated. The processing is characterized in that the sum calculation processing for all documents of the processing and the appearance frequency square value is processed by a single distributed processing server.

The relevance calculation system between keywords described in claim 5 is a relevance calculation system between keywords comprising a management server and a plurality of distributed processing servers, wherein the management server is stored in the document storage means. Means for distributing the plurality of document files to a plurality of first distributed processing servers comprising at least a part of the plurality of distributed processing servers, and a plurality of keywords stored in the keyword storage means. Means for transmitting to each of the distributed processing servers, and a plurality of types of combination frequency product value files transmitted from each of the first distributed processing servers; Between the two distributed processing servers, the means for distributing and distributing each type according to each charge, and the keywords transmitted from the second distributed processing server Means for storing the sum of all current product values of all product files in the keyword combination frequency sum table storage means; and a plurality of document files stored in the document storage means in the plurality of distributed processing servers. Means for distributing to at least a portion of a plurality of third distributed processing servers, means for transmitting a plurality of keywords stored in the keyword storage means to each of the third distributed processing servers, and each third Means for transmitting the appearance frequency square value file transmitted from the distributed processing server to a fourth distributed processing server which is one of the plurality of distributed processing servers, and the fourth distributed processing server. A means for storing the sum of the squares of the appearance frequency of each keyword over all document files in the keyword frequency sum table storage means, and a pair of keys from the keyword storage means. Means for taking out over de, from the keyword combination frequency sum table storage means for the pair of keywords, and means for taking the sum of the product values of frequency between each keyword from the keyword frequency summation table storage means, the pair For each keyword, a means to extract the sum of the squares of the appearance frequency of each keyword, calculate the square root of this sum, add both square roots, and use this sum to divide the sum of the product values of the appearance frequency between keywords. The first distributed processing server detects the presence or absence of each keyword for each document file with respect to the assigned document file distributed by the management server. Key with the younger character code of the first character placed between the pairing keyword and the keyword with the appearance record Means for generating a combination of words, means for calculating the product of the appearance frequencies of each keyword for each pair of keywords as a product value of the appearance frequencies between the pair of keywords, and the first keyword Means for comparing the character code of the first character with the assigned character code range of a plurality of combination frequency product value files to which a character code range has been assigned in advance to identify a combination frequency product value file to be described; For each document file in the corresponding combination frequency product value file, and means for transmitting the plurality of types of combination frequency product value files to the management server, and each of the second distributed processing servers includes: When a plurality of similar combination frequency product value files are transmitted from the management server, means for concatenating each combination frequency product value file and Means for sorting the keyword combinations according to the character code of each keyword, means for summing up the product values in the same keyword combination unit, calculating the sum over all document files, and this sum to the management server Means for detecting each keyword file for the document file distributed by the management server for each document file, and the appearance of a keyword with a history of appearance. Means for calculating a square value of the frequency, means for describing the square value for each document file in the appearance frequency square value file, and means for transmitting the appearance frequency square value file to the management server. The distributed processing server, when a plurality of appearance frequency square value files are transmitted from the management server, means for concatenating the appearance frequency square value files A means for sorting the keywords described in the concatenated file according to each character code; a means for calculating the sum of squares of appearance frequencies in units of the same keyword and calculating a total over all document files; and Means for transmitting to the management server.
In this system, the appearance frequency product value file generation process between keywords, the summation calculation process of all the appearance frequency product values over all documents, and the keyword appearance frequency square value file generation process are distributed by a plurality of distributed processing servers. Further, the present invention is characterized in that the sum total calculation process over all documents of the appearance frequency square value is processed by a single distributed processing server.

The relevance calculation system between keywords described in claim 6 is a relevance calculation system between keywords comprising a management server and a plurality of distributed processing servers, wherein the management server is stored in the document storage means. Means for distributing the plurality of document files to a plurality of first distributed processing servers comprising at least a part of the plurality of distributed processing servers, and a plurality of keywords stored in the keyword storage means. The means for transmitting to each of the distributed processing servers and the combined frequency product value file transmitted from each of the first distributed processing servers are sent to the second distributed processing server that is one of the plurality of distributed processing servers. The sum of the product values of the appearance frequencies between the keywords transmitted from the second distributed processing server and all document files is displayed as a keyword combination frequency sum total. Means for storing in the means, means for distributing the plurality of document files stored in the document storage means to a plurality of third distributed processing servers comprising at least a part of the plurality of distributed processing servers, Means for transmitting a plurality of keywords stored in the keyword storage means to the third distributed processing server, and a plurality of types of appearance frequency square value files transmitted from the respective third distributed processing servers; To each of the plurality of fourth distributed processing servers, and to each of the fourth distributed processing servers. The sum of the squares of the appearance frequency of each keyword over all document files is stored in the keyword frequency total table storage means, and a pair of keys from the keyword storage means. Means for retrieving a word from the keyword combination frequency sum table storage means for the pair of keywords, and means for taking the sum of the product values of frequency between each keyword from the keyword frequency summation table storage means, the pair For a keyword, a means for extracting the sum of squares of the appearance frequency of each keyword, and calculating the square root of this sum, adding both square roots, and dividing the sum of the product values of appearance frequencies between keywords by this sum. Accordingly, the first distributed processing server detects the presence / absence of each keyword for each document file in the assigned document file distributed by the management server. The key with the youngest character code of the first character placed between the means and a pair of keywords that have a history of appearance. A means for generating a combination of words, a means for calculating , for each pair of keywords , a product of appearance frequencies of each keyword as a product value of appearance frequencies between a pair of keywords, and the product value, Means for describing each document file in the combination frequency product value file, and means for transmitting the combination frequency product value file to the management server, wherein the second distributed processing server receives a plurality of combination frequency product values from the management server. A means for concatenating each combination frequency product file when the file is transmitted, a means for sorting the keyword combinations described in the concatenated file according to the character code of each keyword, and a combination unit of the same keyword Means for calculating the sum of the product values and calculating the sum over all document files, and means for transmitting the sum to the management server. Each of the distributed processing servers 3 detects the presence / absence of each keyword in the document file distributed by the management server for each document file, calculates a square value of the appearance frequency of the keyword having the appearance record, Means for comparing the character code of the keyword with the assigned character code range of a plurality of appearance frequency square value files to which a character code range is assigned in advance, and specifying the appearance frequency square value file to be described; , A means for describing each document file in a corresponding appearance frequency square value file, and a means for transmitting the plurality of types of appearance frequency square value files to the management server. When a plurality of similar appearance frequency square value files are transmitted from the server, means for concatenating each appearance frequency square value file and the concatenation file Means for sorting the keywords described in the file according to their character codes, means for summing up the squares of the appearance frequency in the same keyword unit, calculating the sum over all document files, and managing the sum And a means for transmitting to.
In the system according to claim 6, the appearance frequency product value file generation processing between keywords, the keyword appearance frequency square value file generation processing, and the summation calculation processing over all the documents of the appearance frequency square values are distributed by a plurality of distributed processing servers. In addition, the present invention is characterized in that the sum total calculation process of all appearance frequency product values over all documents is processed by a single distributed processing server.

  The system for calculating the degree of association between keywords described in claim 7 is the system according to claims 3 to 6, wherein the management server further converts the plurality of document files stored in the document storage means into the plurality of distributed processes. Means for instructing keyword extraction in advance to a plurality of distributed processing servers comprising at least some of the servers, and means for storing the keywords transmitted from each distributed processing server in the keyword storage means Each of the distributed processing servers includes a keyword extracting unit that extracts a keyword from a document file distributed by the management server, and a unit that transmits the keyword to the management server.

The method for calculating the degree of association between keywords according to claim 8 is a method for calculating the degree of association between keywords based on cooperation between a management server, a plurality of first distributed processing servers, and a second distributed processing server. The management server distributes a plurality of document files stored in the document storage means to each first distributed processing server, and each first distributed processing server sends the responsible document file transmitted by the management server. Extracting keywords from the server and transmitting the keywords to the management server, and after the management server stores the keywords transmitted from the first distributed processing servers in the keyword storage means, all the keywords are sent to the first distributed processing server. And the first distributed processing server that has received the request for each document file detects the presence or absence of each keyword for the assigned document file. The step of calculating the square value of the appearance frequency of the keyword with the appearance record, describing the document frequency in the appearance frequency square value file for each document file, and the character code of the first character between the pair of keywords with the appearance record A step of generating a combination of keywords in which the younger one is arranged first, and for each combination, the product of the appearance frequencies of each keyword is calculated as a product value of the appearance frequencies between the pair of keywords for each pair of keywords. A combination frequency product value file to be described by comparing the character code of the first character of the first keyword with the character code ranges in charge of a plurality of combination frequency product value files to which a character code range is assigned in advance And the product value of the appearance frequency between keywords related to each combination is stored in the corresponding combination frequency product value file as a document file. A step of describing each file, a step of transmitting the appearance frequency square value file and a plurality of types of combination frequency product value files to the management server, and a plurality of occurrences transmitted from each first distributed processing server by the management server A step of transmitting a frequency square value file to the second distributed processing server and a plurality of types of combination frequency product value files transmitted from each first distributed processing server for each type of combination frequency product value file. A distribution process to the first distributed processing server determined, and a second distributed processing server to which the plurality of appearance frequency square value files are transmitted from the management server connect the appearance frequency square value files; Sorting each keyword described in this concatenated file according to each character code The step of calculating the square value of the current frequency, calculating the total over all document files, the step of transmitting the total to the management server, and the management server of the appearance frequency transmitted from the second distributed processing server The step of storing the sum of the square values in the keyword frequency sum table storage means, and the step of the first distributed processing server to which the plurality of combination frequency product value files are transmitted from the management server connecting each combination frequency product value file And sorting the combinations of the keywords described in the linked file according to the character codes of the keywords, summing up the product values of the appearance frequencies in units of combinations of the same keywords, and summing up all the document files. A step of calculating, a step of transmitting the sum to the management server, and the management server sending the product value transmitted from each first distributed processing server The sum, and storing the keyword combination frequency sum table storage means, retrieving a pair of keywords from the keyword storage unit, from the keyword combination frequency sum table storage means for the pair of keywords, the appearance between the keywords A step of taking out a sum of product values of frequencies, a step of taking out a sum of squares of appearance frequencies of each keyword for the pair of keywords from the keyword frequency sum table storing means , and calculating a square root of the sum, respectively. And calculating the degree of association between the two keywords by adding the square roots and dividing the sum of the product values of the appearance frequencies between the keywords by this sum.
Note that the order of the above steps may be appropriately changed as long as no logical contradiction occurs.

  According to the relevance calculation system between keywords described in claim 1, first, in a document file unit, keywords that do not need to be calculated with other keywords because the appearance frequency is zero are excluded, and keywords that have a history of appearance After calculating the square value of the appearance frequency and the product value of the combination frequency, which are the basis for calculating the relevance level, the method of summing up all document files is used, so it appears in all document files. The degree of association between a large number of keywords can be calculated very efficiently. As a result, as in the search system of Patent Document 1, it is possible to calculate the degree of association between all types of keywords without limiting one of the keyword combinations to the company name prior to the calculation of the degree of association. Become.

In addition, even when a new document file is added to the document storage means, the processing of (1), (2), (4) is performed for each new document file, and the calculation results are converted into (3) and (5). After adding to the existing total value (sum), it is only necessary to redo the calculations of (6) and (7), and the recalculation processing of the relevance when adding a document file is facilitated.
In addition, when it is necessary to eliminate the influence of an old document file, the values of (2) and (4) related to the old document file are subtracted from the total value (sum) of (3) and (5). After that, since it is only necessary to redo the calculations of (6) and (7), it becomes easy to keep the degree of association between keywords up to date.

  According to the relevance calculation system between keywords described in claims 2 to 7 and the relevance calculation method between keywords described in claim 8, keyword extraction processing and appearance of each keyword are performed by a plurality of distributed processing servers. At least one of frequency square value file generation processing, appearance frequency square value summation calculation processing over all documents, appearance frequency product value file generation processing between keywords, and appearance frequency product value summation calculation processing over all documents As a result, the overall calculation processing related to the calculation of the degree of association between keywords can be greatly speeded up.

  FIG. 1 is a block diagram showing the functional configuration of the first keyword relevance calculating system 10 and the first search system 11 having the same according to the present invention. The document DB 12, the keyword extracting unit 14, Keyword DB 16, relevance calculation unit 18, keyword co-occurrence frequency table DB 20, keyword combination frequency sum table DB 22, keyword frequency sum table DB 24, keyword relevance table DB 26, proper noun DB 28, and search processing unit 30 And.

  The keyword extraction unit 14, the relevance calculation unit 18, and the search processing unit 30 are realized by the CPU of the computer executing necessary processing according to the OS and a dedicated application program.

The document DB 12, the keyword DB 16, the keyword co-occurrence frequency table DB 20, the keyword combination frequency sum table DB 22, the keyword frequency sum table DB 24, the keyword relevance table DB 26, and the proper noun DB 28 are stored in the hard disk of the computer.
A large number of document files (text data) such as newspaper articles, academic journals, and papers are stored in the document DB 12 in advance. In the proper noun DB 28, a number of proper nouns such as company names, product names, service names, and person names are registered for each category.

  As shown in FIG. 2, the keyword extraction unit 14 includes a dependency expression extraction filter 32, a delimiter extraction filter 34, a character string frequency statistical filter 36, a TermExtract filter 38, and a majority decision filter 40.

Next, the keyword extraction process by the keyword extraction unit 14 will be described with reference to the flowchart of FIG.
First, the keyword extraction unit 14 applies a dependency expression extraction filter 32 to each document file stored in the document DB 12, and extracts a character string having a predetermined dependency expression from each document file (S10).
That is, the dependency expression extraction filter 32 is provided with a large number of dependency expression patterns “XX manufacturer”, “XX is the main force”, and “XX is produced” in advance. After the expression pattern that applies to is detected, a character string corresponding to “XX” is extracted as a keyword candidate.

  Next, the keyword extraction unit 14 applies a delimiter extraction filter 34 to each document file, such as “XX”, “XX”, (XX), [XX], XX, and so on. The part of XX surrounded by delimiters such as parentheses, spaces, tabs, etc. is extracted as a keyword candidate (S12).

Next, the keyword extraction unit 14 applies the character string frequency statistical filter 36 to each document file, and counts how many times each character string included in each document file appears, including other documents. A character string having an appearance frequency is extracted as a keyword candidate (S14).
First, as shown in FIG. 4, the character string frequency statistical filter 36 pays attention to a noun (here, “DVD”) in the document, and the attention word “DVD” appears in each document file stored in the document DB 12. Add up the number you want. Next, the character string frequency statistical filter 36 expands the range to the morpheme before and after this attention word, totals the frequency that appears in all the documents, and the appearance frequency becomes less than a certain value (for example, 20 or less). Stop character range expansion at this point.

  For example, since the appearance frequency of “done DVD” including the previous morpheme of the DVD is as low as “2”, the range is not expanded to the previous morpheme. On the other hand, since the appearance frequency of “DVD recorder” including the next morpheme of DVD is as many as “862”, the appearance frequencies of “DVD recorder” including the next morpheme are tabulated. Since the appearance frequency is as low as “5”, the expansion of the range to subsequent morphemes is stopped.

Next, the character string frequency statistical filter 36 extracts “DVD” and “DVD recorder” as keyword candidates because they have an appearance frequency within a predetermined range (for example, 20 to 5,000). On the other hand, “done DVD” and “in the DVD recorder” are out of the above range, and are excluded from keyword candidates.
This is because, if the frequency of occurrence is less than 20 in all documents, it is not an important word in the first place, and if it exceeds 5,000, it is considered to be a general word or general word without features. The amount is adjusted as appropriate according to the amount of use and the purpose of the search system.

  By the way, since it takes a lot of time to count the appearance frequency of each character string included in a large amount of document files stored in the document DB 12, as shown in FIG. An index (so-called transposed index) is generated that summarizes whether each morpheme appearing in all document files exists in each document file. For this reason, the keyword extracting unit 14 can acquire the appearance frequency in a relatively short time by referring to the index.

Next, the keyword extraction unit 14 applies the TermExtract filter 38 to the document files stored in the document DB 12, and extracts a character string having a score equal to or higher than a predetermined value from each document file as a keyword candidate (S16).
This TermExtract is a string extraction algorithm devised to automatically extract technical terms from a specialized corpus (a huge text data consisting mainly of natural language sentences collected mainly for research purposes). A function is provided for extracting single nouns and compound nouns from the document file as candidate words and calculating the respective importance based on the appearance frequency and the connection frequency of each candidate word. Since this TermExtract itself is a known technique, further explanation is omitted.

Next, the keyword extraction unit 14 inputs each keyword candidate extracted by the dependency expression extraction filter 32, the delimiter extraction filter 34, the character string frequency statistical filter 36, and the TermExtract filter 38 to the majority filter 40, and narrows down the keywords.
The majority decision filter 40 matches the keyword candidates listed by each filter, recognizes those listed as keyword candidates by two or more filters as final keywords, and stores them in the keyword DB 16 (S18).

  As described above, by using the four filters of the dependency expression extraction filter 32, the delimiter extraction filter 34, the character string frequency statistical filter 36, and the TermExtract filter 38, important words are leaked when keywords are extracted from the document file. In addition to this, it is possible to prevent unnecessary keywords (noise) from being mixed by narrowing down using the majority filter 40.

As described above, the keyword candidate selected by two or more of the four filters is recognized as an official keyword, and selection by three or more filters may be a requirement for keyword recognition. it can.
Further, the number of filters is not limited to the above, and other effective keyword candidate extraction filters may be provided in the keyword extraction unit 14.

Next, according to the flowchart of FIG. 6, the relevance calculation process between the keywords by the relevance calculation unit 18 will be described.
First, the relevance calculating unit 18 generates a keyword co-occurrence frequency table by counting the co-occurrence frequencies of each keyword in each document file, and stores the keyword co-occurrence frequency table in the keyword co-occurrence frequency table DB 20 (S20).
FIG. 7 shows a specific example of the keyword co-occurrence frequency table stored in the keyword co-occurrence frequency table DB20. For each document D1-Dn stored in the document DB 12, each keyword KW-1 to n is stored. Appearance frequency is described.

Here, the degree of association between a certain keyword X and Y can be theoretically calculated by substituting the appearance frequency of X and Y described in the keyword co-occurrence frequency table DB20 into i of Equation 1. is there.

  Since the numerator of Equation 1 means the sum of the products of the appearance frequencies of the keywords X and Y for all documents, the value increases as the frequency of occurrence of X and Y in the same document increases. However, if the absolute number of occurrence frequencies of X and Y in a specific document is large, the value of the numerator increases accordingly, and it does not necessarily indicate the high co-occurrence of X and Y. . On the other hand, the denominator is obtained by adding the square roots of the sums of all the squares of the appearance frequencies of the keywords X and Y for each document, and the value increases as the appearance frequency in the specific document of X and Y increases. Become. For this reason, by dividing the numerator value by the denominator value, the influence of the large number of occurrence frequencies of X and Y in a specific document is eliminated, and the co-occurrence between X and Y is increased. It is possible to derive the degree of relevance based on it.

However, in the method of simply performing the calculation of Equation 1, if the amount of document files and the total number of keywords are large, a huge amount of calculation occurs, and a lot of processing time is required.
Therefore, in this embodiment, the calculation process is simplified by generating the keyword combination frequency summation table and the keyword frequency summation table based on the keyword co-occurrence frequency table.

FIG. 8 illustrates the procedure. In this case, the keyword co-occurrence frequency table describes the appearance frequencies of the keywords KW-1 to KW-5 in the document D1, but the KW-3 and KW-4 appearance frequencies are 0. The combination of keywords for which the degree of relevance should be calculated is the following three patterns.
(KW-1, KW-2), (KW-1, KW-5), (KW-2, KW-5)
Next, the relevance calculation unit 18 generates a keyword combination frequency sum table describing values multiplied by the appearance frequency for each combination, and a keyword frequency sum table describing values obtained by squaring the appearance frequency of each keyword. The combined frequency total table DB22 and the keyword frequency total table DB24 are stored (S22, S24).

In the keyword combination frequency summation table of FIG. 8, only the value for the document D1 is described. However, the same processing is executed for each document, and the values are added based on the result. Is equivalent to the numerator of Equation 1.
Similarly, in the keyword frequency total table of FIG. 8, only the value for the document D1 is described, but the value obtained by squaring the appearance frequency of each keyword in each document is aggregated, and the final value ( By calculating the square root of (sum), a value corresponding to the denominator of Equation 1 is obtained.

  Finally, as shown in FIG. 9, the degree-of-relevance calculation unit 18 reads the sum of the combination frequencies of the keywords X and Y from the keyword combination frequency sum table DB22, and the sum of the square values of the keywords X from the keyword frequency sum table DB24. After reading the sum of the square values of the keyword Y and calculating the square root of the sum of the square values, substituting these values into Equation 1 calculates the relevance between the keywords X and Y, and the keyword relevance table Store in DB26 (S26). The degree-of-association calculation unit 18 repeats the process until the process is completed for all keyword combinations.

As described above, the combination pattern between each keyword is extracted for each document file, the product value and the square value of each keyword are obtained, and then the value of each document file is added, so that the appearance frequency This makes it possible to omit the calculation processing related to the keyword with 0.
For this reason, it is practical to calculate the relevance between all keywords without being limited to the company name as in the search system of Patent Document 1.

In addition, when a new document file is added to the document DB 12, values related to each keyword in the new document file are stored in the existing total values stored in the keyword combination frequency summation table DB22 and the keyword frequency summation table DB24. By adding to, it is possible to easily recalculate the degree of association between keywords.
Even when the influence of an obsolete document file is excluded, the value related to each keyword in the document file is subtracted from the existing total value stored in the keyword combination frequency summation table DB22 and the keyword frequency summation table DB24. Thus, it is possible to easily maintain the degree of association between keywords in the latest state.

Next, a search processing procedure in the system 10 will be described with reference to the flowchart of FIG.
First, when the user inputs a search word from the terminal device α, the search processing unit 30 that has received the search word (S40) refers to the keyword relevance table DB 26 as shown in FIG. A keyword having a similarity is specified, and a list of keywords having a predetermined degree of relevance to the keyword is extracted (S42).
Next, the search processing unit 30 refers to, for example, the company name DB in the proper noun DB 28 and extracts the company name included in the list (S44).
The extracted list of company names is transmitted to the terminal device α as a company list closely related to the search term (S46).

As a result, the user can recognize a company closely related to the input search word (for example, current affair term), and can use it for the judgment of investment behavior.
If a person name DB is designated as the proper noun DB 28, a person closely related to the input search word can be picked up.

However, a list of keywords closely related to the search term may be returned to the terminal device α as it is without matching with the company name DB or the person name DB.
Thereafter, when the user designates a specific keyword in the keyword list as a search word, the search processing unit 30 further extracts a list of keywords having a predetermined relationship with the keyword and transmits it to the terminal device α. .
As a result, the user can expand the search range in a chain from related words to related words, and can be expected to arrive at an unexpected keyword.

When the user designates a specific keyword in the search result list and requests the presentation of a document as the basis thereof (S48), the search processing unit that accepts the request (S48) assigns the search word and the keyword as shown in FIG. Based on this, the keyword co-occurrence frequency table DB 20 is searched to generate a list of document numbers in which co-occurrence occurs between the two (S50).
Next, the search processing unit 30 searches the document DB 12 based on the document number list, generates a list of document texts, and transmits it to the terminal device α (S52, S54).
As a result, the number, title, abstract, date, etc. of the document in which the search word and the keyword appear simultaneously are displayed in a list on the display of the terminal device α.

When the user selects one of these, the search processing unit 30 extracts the corresponding document file from the document DB 12 and transmits it to the terminal device α.
As a result, the user can browse the contents of the document file and individually confirm the relevance between the search word and the keyword.

  FIG. 13 is a conceptual diagram showing a second search system 52 provided with a system for calculating the degree of association between second keywords according to the present invention. The relationship calculation system 50 between the second keywords includes a management server 54, three first distributed processing servers 56a to 56c, and a second distributed processing server 57. It is intended to make the creation of the keyword relevance table more efficient and faster by cooperating with the distributed processing servers 56a to 56c and the second distributed processing server 57.

The management server 54 includes a document DB 12, a keyword DB 16, a keyword combination frequency sum table DB 22, a keyword frequency sum table DB 24, a keyword relevance table DB 26, and a proper noun DB 28.
A Web server 58 is connected to the management server 54 via a network, and a plurality of terminal devices α are connected to the Web server 58 via the Internet 60.

A procedure for generating a keyword relevance table in the second relevance calculation system 50 will be described below with reference to the flowcharts of FIGS.
First, as shown in FIG. 16, the management server 54 divides and distributes a large number of document files 62 stored in the document DB 12 to the first distributed processing servers 56a to 56c (S60 in FIG. 14). At this time, the management server 54 adjusts the data amounts of the document files 62a to 62c to be distributed to the first distributed processing servers 56a to 56c so that the processing loads on the first distributed processing servers 56a to 56c become substantially equal.

Next, in the first distributed processing servers 56a to 56c, keyword extraction processing is performed on the distributed document files 62a to 62c by the keyword extraction processing units 64a to 64c (S61).
In the keyword extraction process, as described above, the dependency expression extraction filter 32, the separator character extraction filter 34, the character string frequency statistical filter 36, the TermExtract filter 38, the majority filter provided in each of the keyword extraction processing units 64a to 64c. By using 40, keywords in an appropriate range from which noise is eliminated are extracted.

Regarding the application of the character string frequency statistical filter 36, since it is necessary to refer to the appearance frequency of the attention word in the document in charge of other distributed processing servers, the first distributed processing servers 56a to 56c The management server 54 is inquired about this.
Receiving this, the management server 54 refers to the morpheme index (transposed index) provided in the document DB 12 to acquire the appearance frequency of the attention word in all documents, and the result is the first query source. To the distributed processing server 56.

The first distributed processing servers 56a to 56c that have completed the keyword extraction processing for the assigned document files 62a to 62c assigned to themselves transmit the extracted keywords to the management server 54 (S62).
The management server 54 registers the keywords received from the first distributed processing servers 56a to 56c in the keyword DB 16 (S63). At this time, when the same keyword is repeatedly transmitted from the first distributed processing servers 56a to 56c, one of them is registered in the keyword DB 16.

  Next, as shown in FIG. 17, the management server 54 transmits the data 66 of all the keywords registered in the keyword DB 16 to the first distributed processing servers 56a to 56c (S64).

In FIG. 17, for convenience of illustration, a state in which the keyword data 66 is transmitted to the first distributed processing server 56 b is illustrated, but the same keyword data is also applied to the other first distributed processing servers 56 a and 56 c. 66 is sent.
Similarly, in the following, various processes will be described with a focus on the first distributed processing server 56b, but similar processes are also executed in the other first distributed processing servers 56a and 56c.

  The file generation unit 68b of the first distributed processing server 56b that has received the keyword data 66 from the management server 54 first detects the presence / absence of the appearance of individual keywords for each document file 62b allocated to itself, and The appearance frequency of a certain keyword is calculated for each document file (S65).

Next, the file generation unit 68b calculates a value obtained by squaring the appearance frequency of each keyword, and describes it in the appearance frequency square value file 70b (S66).
For example, when “Sakura” appears 3 times, “Spring” 5 times, and “鶯” appears 6 times in a document file, the file generation unit 68b displays “Sakura, 9”, “Spring, 25”, “鶯, 36 ", the appearance frequency square value data composed of the combination of the keyword and the square value of the appearance frequency is described in the appearance frequency square value file 70b in units of one line.

  Next, the file generating unit 68b generates a combination of two keywords for all keywords that have appeared in the document file (S67). At this time, the file generation unit 68b arranges the keyword with the younger character code of the first character among the pair of keywords in the first (left side).

  For example, assuming that there are three keywords “Sakura”, “Spring”, and “鶯” in a document file, the file generation unit 68b takes into account the shift JIS code of each leading character, and “Sakura, Spring”. Generates a combination of “Sakura, Samurai” and “Spring, Samurai”. Incidentally, the character code of “Sa” in “Sakura” is “82B3”, the character code of “Spring” is “8F74”, and the character code of “鶯” is “E9F2”.

  Next, the file generation unit 68b multiplies the appearance frequency between the keywords of each combination, and generates combination frequency product value data in which the combination of the keyword and its product value are set (S68).

Next, the file generation unit 68b describes this combination frequency product value data in a combination frequency product value file corresponding to the character code of the first character of the first keyword (S69).
That is, as this combination frequency product value file, three types of files corresponding to the number of first distributed processing servers 56a to 56c (first combination frequency product value file 72b, second combination frequency product value file 74b, A third combination frequency product value file 76b) is generated on the disk by the file generation unit 68b.

  In addition, character code ranges are allocated in advance to the combination frequency product value files 72b, 74b, and 76b. For example, assuming the Shift JIS code system shown in FIG. 18 (a), as shown in FIG. 18 (b), the first combination frequency product value file 72b has a character whose first byte is 20 to DF. A code range is assigned. The second combination frequency product value file 74b has a character code range of 81 to 8E in the first byte, and the third combination frequency product value file 76b has characters in the first byte of 8F to 9F and E0 to EF. Each code range is allocated.

Therefore, the combination frequency product value data of “Sakura, Spring, 15” and “Sakura, Samurai, 18” is “82B3” in the character code of “Sa”, and therefore is stored in the second combination frequency product value file 74b. Described.
On the other hand, the combination frequency data of “spring, 鶯, 30” is described in the third combination frequency product value file 76b because the character code of “spring” is “8F74”.
When the combination frequency product value data “PCT, patent, 20” is generated, it is described in the first combination frequency product value file 72b in charge of the half-width alphabet.

The file generation unit 68b executes the above-described processing of S65 to S69 on all assigned document files 62b assigned to itself (S70).
In this process, it is expected that a large number of appearance frequency square value data relating to the same keyword and combination frequency product value data relating to the same keyword combination will occur between different document files. The values of each data are not added up, and each data is added to the appearance frequency square value file and the corresponding combination frequency product value file in the order of generation.

  The above processing is executed independently in the first distributed processing servers 56a to 56c, and when the processing for the document files 62a to 62c in charge of each is completed, the processing is managed from the first distributed processing servers 56a to 56c. For the server 54, keyword appearance frequency square value files 70a to 70c, first keyword combination frequency product value files 72a to 72c, second keyword combination frequency product value files 74a to 74c, and third keyword combination frequency product value file 76a to 76c are transmitted (S71 in FIG. 15).

  On the other hand, the management server 54 sends the first keyword combination frequency product value files 72a to 72c, the second keyword combination frequency product value files 74a to 74c, and the third information transmitted from the first distributed processing servers 56a to 56c. The keyword combination frequency product value files 76a to 76c are distributed and distributed to the first distributed processing servers 56a to 56c according to their respective responsibilities (S72).

  For example, as shown in FIG. 19, since the second distributed frequency product value file is assigned in advance to the first distributed processing server 56b, the second distributed processing servers 56a to 56c generate the second Combination frequency product value files 74 a to 74 c are distributed from the management server 54.

Similarly, first keyword combination frequency product value files 72a to 72c generated by the first distributed processing servers 56a to 56c are assigned to the first distributed processing server 56a to which the first combination frequency product value file is assigned. However, the third combination frequency product value files 76a to 76c generated by the first distributed processing servers 56a to 56c are distributed to the distributed processing server 56c to which the third combination frequency product value file is assigned. .
Hereinafter, although the description will be focused on the processing in the first distributed processing server 56b, the same processing is executed in the other distributed processing servers 56a and 56c.

First, in the first distributed processing server 56b, the three combination frequency product value files 74a to 74c are combined by the file combining unit 78b (S73).
Next, the sort processing unit 80b is activated, and the keyword combinations (X, Y) described in the combined file 82b are arranged in the order of their character codes (S74). As a result, a sorted file 86b in which a plurality of combination frequency product value data having the same keyword combination is arranged, such as “Sakura, Spring, 20”... “Sakura, Spring, 32”. Generated.
Next, the addition processing unit 84b is activated, so-called control break processing is performed on the sorted file 86b, and product values are tabulated in units of the same keyword combination (S75).

As a result, the sum (corresponding to the numerator in Equation 1) of the product values of the appearance frequencies of the keywords X and Y in each document file is obtained over all document files.
The calculation result file 88b is transmitted from the first distributed processing server 56b to the management server 54 (S76).
On the other hand, the management server 54 extracts the data of the calculation result file 88b and registers it in the keyword combination frequency sum table DB22 (S77). That is, when the combination value of the same keyword X, Y already exists in the keyword combination frequency summation table, the management server 54 adds the result value to the existing value, and when the existing value does not exist, the keyword X, Y and its value are newly added.

  In parallel with the above, the second distributed processing server 57 also executes predetermined processing. That is, as shown in FIG. 20, the appearance frequency square value files 70a to 70c transmitted from the first distributed processing servers 56a to 56c are transmitted from the management server 54 to the second distributed processing server 57 ( S78).

In the second distributed processing server 57 that has received this, the file combination unit 90 combines the three appearance frequency square value files 70a to 70c (S79).
Next, the sort processing unit 91 is activated, and the keywords and square values described in the combined file 92 are arranged in the order of character codes (S80). As a result, a sorted file 93 in which a plurality of the same keywords are arranged, such as “Sakura, 16”... “Sakura, 9”.
Next, the addition processing unit 94 is activated, and the square values are totaled in the same keyword unit (S81).

As a result, the total sum of the square values of the frequency frequencies of the keywords in each document over all the documents is obtained.
The calculation result file 95 is transmitted from the second distributed processing server 57 to the management server 54 (S82).
In response to this, the management server 54 extracts the result data in the calculation result file 95 and registers it in the keyword frequency summation table DB 24 (S83). That is, when the value of the same keyword already exists in the keyword frequency summary table, the management server 54 adds the result value to the existing value, and adds the keyword and its value when the existing value does not exist. .

  Finally, as shown in FIG. 9, the management server 54 reads the sum of the combination frequencies of the keywords X and Y from the keyword combination frequency summation table DB22, and the sum of the square values of the keyword X and the keyword from the keyword frequency summation table DB24. After reading the sum of the squared values of Y and obtaining the square root of the sum of the squared values, substituting these values into Equation 1 calculates the relevance between the keywords X and Y, and the keyword relevance table DB26 (S84). The management server 54 repeats the processing until the processing is completed for all keyword combinations.

  In the case of the second relatedness calculation system 50 between keywords, as described above, the keyword extraction process, the keyword combination frequency summation calculation process and the keyword frequency summation calculation process which are the premise of the relevance calculation are the first distributed process. Since the servers 56a to 56c and the second distributed processing server 57 are executed in parallel, the generation speed of the keyword relevance table can be dramatically improved.

  In addition, the first distributed processing servers 56a to 56c store the calculation result data in a file format, and writing to the database does not occur each time the data is stored, so that the overall processing speed is increased. be able to.

Hereinafter, the search processing procedure in the second search system 52 will be described with reference to the flowchart of FIG.
First, when the user inputs a search word from the terminal device α, the management server 54 that has received the search word via the Web server 58 (S90) refers to the keyword relevance table DB 26 as shown in FIG. A keyword having the same or similar similarity within a certain range is specified, and a list of keywords having a predetermined degree of relevance to the keyword is extracted (S91).

Next, the management server 54 refers to, for example, the company name DB in the proper noun DB 28 and extracts the company name included in the list (S92).
The extracted list of company names (list of companies closely related to the search term) is transmitted to the terminal device α via the Web server 58 (S93).

As a result, the user can recognize a company closely related to the input search word (for example, current affair term), and can use it for the judgment of investment behavior.
If a person name DB is designated as the proper noun DB 28, a person closely related to the input search word can be picked up.

Note that the function allocation to each distributed processing server is not limited to the method of the second relevance calculation system 50 described above.
For example, in the above case, the first distributed processing server 56 performs keyword extraction processing, keyword appearance frequency square value file generation processing, inter-keyword combination frequency product value file generation processing, and all combination frequency product value documents. Although it is in charge of the total calculation process over the file, each process can be distributed to a group of a plurality of other distributed processing servers.

  Also, a plurality of second distributed processing servers 57 can be provided, and the sum calculation processing over all document files of the appearance frequency square value of each keyword can be distributed. In this case, as a precondition, it is necessary to generate a plurality of types of keyword appearance frequency square value files corresponding to the same number of character code ranges as the number of second distributed processing servers 57. Naturally, it is possible to assign to the distributed processing server 57.

It is a block diagram which shows the function structure of the association degree calculation system between the 1st keywords and 1st search system which concern on this invention. It is a block diagram which shows the function structure of a keyword extraction part. It is a flowchart which shows a keyword extraction process. It is explanatory drawing which shows operation | movement of a character string frequency statistical filter. It is explanatory drawing which shows a mode that the morpheme index is formed in document DB. It is a flowchart which shows the related degree calculation process between keywords. It is explanatory drawing which shows an example of a keyword co-occurrence frequency table. It is explanatory drawing which shows the method of simplifying a relevance calculation process. It is explanatory drawing which shows a mode that a keyword relevance table is produced | generated based on a keyword combination frequency total table and a keyword frequency total table. It is a flowchart which shows the procedure of a search process. It is explanatory drawing which shows the point which extracts a company name list based on a search term. It is explanatory drawing which shows the point which shows the basis of the relevance degree between a search word and a specific keyword. It is a block diagram which shows the function structure of the related degree calculation system between the 2nd keywords which concerns on this invention, and a 2nd search system. It is a flowchart which shows the related degree calculation process between keywords. It is a flowchart which shows the related degree calculation process between keywords. It is a schematic diagram which shows a mode that a document file is divided and delivered to a 1st distributed processing server from a management server. It is a schematic diagram which shows a mode that the appearance frequency square value file of each keyword and the combination frequency product value file between keywords are produced | generated by the 1st distributed processing server. It is explanatory drawing which shows the example which allocated the character code range to each combination frequency product value file. It is a schematic diagram which shows a mode that the sum total of the appearance frequency product value between keywords is calculated by the 1st distributed processing server. It is a schematic diagram which shows a mode that the sum total of the appearance frequency square value of each keyword is calculated by the 2nd distributed processing server. It is a flowchart which shows the procedure of a search process.

Explanation of symbols

10 Relevance calculation system between first keywords
11 First search system
12 Document DB
14 Keyword extractor
16 Keyword DB
18 Relevance calculator
20 Keyword co-occurrence frequency table DB
22 Keyword combination frequency summary table DB
24 Keyword Frequency Summation Table DB
26 Keyword relevance table DB
28 proper noun DB
30 Search processing section
32 Dependency Expression Extraction Filter
34 Delimiter extraction filter
36 String frequency statistics filter
38 TermExtract filter
40 Majority filter
50 Relevance calculation system between second keywords
52 Second search system
54 Management server
56a-56c First distributed processing server
57 Second distributed processing server
58 Web server
60 Internet
62a-62c Document file in charge
64a-64c Keyword extraction processing unit
68b File generator
70a-70c Keyword appearance frequency square value file
72a to 72c Combination frequency product file
74a to 74c Combination frequency product value file
76a to 76c Combination frequency product file
66 All keyword data
78b File connection
80b Sort processing part
82b Combined file
84b Addition processing unit
86b Sorted files
88b Calculation result file
90 File connection
91 Sort processing section
92 Combined files
93 Sorted files
94 Addition processing section
95 Calculation result file α Terminal device

Claims (8)

Document storage means for storing a plurality of document files;
A keyword extracting means for extracting a plurality of keywords from each document file and storing the extracted keywords in the keyword storage means;
The system includes a relevance calculation unit that calculates the relevance between a pair of keywords for all combinations of keywords based on the appearance frequency of each keyword in each document file and stores the relevance in a keyword relevance storage unit. And
The relevance calculation means is
(1) In a document file unit, a process for detecting keywords that have appeared in the document file and calculating their appearance frequency;
(2) a process of calculating the square value of the appearance frequency of each keyword;
(3) The process of calculating the sum of the squares of the appearance frequency of each keyword and calculating the total over all document files;
(4) A process of calculating , for each pair of keywords , a product of the appearance frequencies of each keyword as a product value of the appearance frequencies between the pair of keywords in a document file unit;
(5) Totaling the product values of the appearance frequency between each keyword and calculating the total over all document files;
(6) A process for calculating the square root of the sum of (3) above;
(7) A process of calculating the degree of association between both keywords by adding the square roots of (6) above of a pair of keywords and dividing the sum of the above (5) by the sum,
This is a system for calculating the degree of association between keywords. A relevance calculation system between keywords comprising a management server, a plurality of first distributed processing servers, and a second distributed processing server,
Means for distributing a plurality of document files stored in the document storage means to each first distributed processing server;
Means for storing the keyword transmitted from each first distributed processing server in the keyword storage means;
Means for transmitting all the keywords stored in the keyword storage means to the first distributed processing server,
Means for transmitting a plurality of appearance frequency square value files transmitted from each first distributed processing server to the second distributed processing server;
Means for distributing and distributing a plurality of types of combination frequency product values files transmitted from each first distributed processing server to the first distributed processing server to be responsible according to the type;
Means for storing, in the keyword frequency sum total table storage means, the sum total of all the document files of the square values of the appearance frequencies of the keywords transmitted from the second distributed processing server;
Means for storing sum totals over all document files of product values of appearance frequencies between the keywords transmitted from the respective first distributed processing servers in the keyword combination frequency sum table storage means;
Means for retrieving a pair of keywords from the keyword storage means;
Means for taking out a sum of product values of appearance frequencies between the keywords for the pair of keywords from the keyword combination frequency sum table storage means;
From the keyword frequency summation table storage means for the pair of keywords, and means for taking the sum of the square of the frequency of occurrence of each keyword,
A means for calculating the degree of relevance between the two keywords by calculating the square root of each sum, adding both square roots, and dividing the sum of the product values of the appearance frequencies between the keywords by this sum,
The first distributed processing server, a keyword extracting means for extracting a keyword from a responsible document file distributed by the management server;
Means for sending each keyword to the management server;
Means for detecting the presence or absence of each keyword for the document file in charge for each document file when all keywords are transmitted from the management server;
Means for calculating a square value of an appearance frequency of a keyword having an appearance record, and describing each document file in an appearance frequency square value file;
A means for generating a combination of keywords in which the first character code of the first character is arranged first among a pair of keywords having a history of appearance;
For each pair, for each pair of keywords, means for calculating the product of the appearance frequencies of each keyword as the product of the appearance frequencies between the pair of keywords;
Means for identifying the combination frequency product value file to be described by comparing the character code of the first character of the first keyword with the assigned character code ranges of a plurality of combination frequency product value files to which character code ranges are assigned in advance When,
Means for describing the product value for each document file in a corresponding combination frequency product value file;
Means for transmitting the appearance frequency square value file and a plurality of types of combination frequency product value files to the management server;
Means for linking each combination frequency product file when a plurality of similar combination frequency product files are transmitted from the management server;
Means for sorting the keyword combinations described in the concatenated file according to the character code of each keyword;
A means of calculating product values in combination units of the same keyword and calculating a sum total over all document files;
Means for transmitting this sum to the management server,
Means for connecting the appearance frequency square value files when the second distributed processing server transmits a plurality of appearance frequency square value files from the management server;
Means for sorting the keywords described in the concatenated file according to their character codes;
Means for calculating the sum of the squares of the appearance frequency in the same keyword unit and calculating the total over all document files;
A system for calculating a degree of association between keywords, comprising means for transmitting the sum to a management server. A relevance calculation system between keywords comprising a management server and a plurality of distributed processing servers,
Means for distributing the plurality of document files stored in the document storage means to a plurality of first distributed processing servers comprising at least a part of the plurality of distributed processing servers;
Means for transmitting each of the plurality of keywords stored in the keyword storage means to the first distributed processing server;
A plurality of types of combination frequency product files transmitted from each first distributed processing server are assigned to each of a plurality of second distributed processing servers including at least a part of the plurality of distributed processing servers. Means to distribute and distribute according to the type,
Means for storing sum totals over all document files of product values of appearance frequencies between the keywords transmitted from each second distributed processing server in the keyword combination frequency sum table storage means;
Means for distributing a plurality of document files stored in the document storage means to a plurality of third distributed processing servers comprising at least a part of the plurality of distributed processing servers;
Means for transmitting each of the plurality of keywords stored in the keyword storage means to the third distributed processing server;
A plurality of types of appearance frequency square value files transmitted from each third distributed processing server are assigned to each of a plurality of fourth distributed processing servers including at least a part of the plurality of distributed processing servers. Means to distribute and distribute according to the type,
Means for storing, in the keyword frequency total table storage means, the sum total over all document files of the square value of the appearance frequency of each keyword transmitted from each fourth distributed processing server;
Means for retrieving a pair of keywords from the keyword storage means;
Means for taking out a sum of product values of appearance frequencies between the keywords for the pair of keywords from the keyword combination frequency sum table storage means;
Means for extracting , from the keyword frequency sum table storage means , a sum of square values of appearance frequencies of the keywords for the pair of keywords ;
A means for calculating the degree of relevance between the two keywords by calculating the square root of each sum, adding both square roots, and dividing the sum of the product values of the appearance frequencies between the keywords by this sum,
Each of the first distributed processing servers detects a presence or absence of each keyword for each document file in the document file distributed by the management server;
A means for generating a combination of keywords in which the first character code of the first character is arranged first among a pair of keywords having a history of appearance;
For each pair, for each pair of keywords, means for calculating the product of the appearance frequencies of each keyword as the product of the appearance frequencies between the pair of keywords;
Means for identifying the combination frequency product value file to be described by comparing the character code of the first character of the first keyword with the assigned character code ranges of a plurality of combination frequency product value files to which character code ranges are assigned in advance When,
Means for describing the product value for each document file in a corresponding combination frequency product value file;
Means for transmitting these multiple types of combination frequency product files to the management server,
When each of the second distributed processing servers transmits a plurality of similar combination frequency product value files from the management server, means for connecting the combination frequency product value files;
Means for sorting the keyword combinations described in the concatenated file according to the character code of each keyword;
A means of calculating product values in combination units of the same keyword and calculating a sum total over all document files;
Means for transmitting this sum to the management server,
Each of the above third distributed processing servers detects, for each document file, the presence or absence of each keyword in the document file distributed by the management server;
Means for calculating a square value of the appearance frequency of a keyword with an appearance record;
Means for comparing the character code of each keyword with the assigned character code range of a plurality of appearance frequency square value files to which a character code range is assigned in advance, and specifying an appearance frequency square value file to be described;
Means for describing the square value for each document file in the corresponding appearance frequency square value file;
Means for transmitting these multiple types of appearance frequency square value files to the management server,
Each of the fourth distributed processing servers, when a plurality of similar appearance frequency square value files are transmitted from the management server, means for connecting the appearance frequency square value files;
Means for sorting the keywords described in the concatenated file according to their character codes;
Means for calculating the sum of the squares of the appearance frequency in the same keyword unit and calculating the total over all document files;
A system for calculating the degree of association between keywords, comprising: a means for transmitting the sum to the management server. A relevance calculation system between keywords comprising a management server and a plurality of distributed processing servers,
Means for distributing the plurality of document files stored in the document storage means to a plurality of first distributed processing servers comprising at least a part of the plurality of distributed processing servers;
Means for transmitting each of the plurality of keywords stored in the keyword storage means to the first distributed processing server;
Means for transmitting the combination frequency product value file transmitted from each first distributed processing server to a second distributed processing server that is one of the plurality of distributed processing servers;
Means for storing sum totals over all document files of product values of appearance frequencies between keywords transmitted from the second distributed processing server in keyword combination frequency sum table storage means;
Means for distributing a plurality of document files stored in the document storage means to a plurality of third distributed processing servers comprising at least a part of the plurality of distributed processing servers;
Means for transmitting each of the plurality of keywords stored in the keyword storage means to the third distributed processing server;
Means for transmitting an appearance frequency square value file transmitted from each third distributed processing server to a fourth distributed processing server that is one of the plurality of distributed processing servers;
Means for storing, in the keyword frequency total table storage means, the sum total over all document files of the square value of the appearance frequency of each keyword transmitted from the fourth distributed processing server;
Means for retrieving a pair of keywords from the keyword storage means;
Means for taking out a sum of product values of appearance frequencies between the keywords for the pair of keywords from the keyword combination frequency sum table storage means;
Means for extracting , from the keyword frequency sum table storage means , a sum of square values of appearance frequencies of the keywords for the pair of keywords ;
A means for calculating the degree of relevance between the two keywords by calculating the square root of each sum, adding both square roots, and dividing the sum of the product values of the appearance frequencies between the keywords by this sum,
Each of the first distributed processing servers detects a presence or absence of each keyword for each document file in the document file distributed by the management server;
A means for generating a combination of keywords in which the first character code of the first character is arranged first among a pair of keywords having a history of appearance;
For each pair, for each pair of keywords, means for calculating the product of the appearance frequencies of each keyword as the product of the appearance frequencies between the pair of keywords;
Means for describing the product value for each document file in the combination frequency product value file;
Means for transmitting this combined frequency product value file to the management server,
The second distributed processing server, when a plurality of combination frequency product value files are transmitted from the management server, means for concatenating each combination frequency product value file;
Means for sorting the keyword combinations described in the concatenated file according to the character code of each keyword;
A means of calculating product values in combination units of the same keyword and calculating a sum total over all document files;
Means for transmitting this sum to the management server,
Each of the above third distributed processing servers detects, for each document file, the presence or absence of each keyword in the document file distributed by the management server;
Means for calculating a square value of the appearance frequency of a keyword with an appearance record;
Means for describing the square value for each document file in an appearance frequency square value file;
Means for transmitting the appearance frequency square value file to the management server,
Means for connecting the appearance frequency square value files when the fourth distributed processing server transmits a plurality of appearance frequency square value files from the management server;
Means for sorting the keywords described in the concatenated file according to their character codes;
Means for calculating the sum of the squares of the appearance frequency in the same keyword unit and calculating the total over all document files;
A system for calculating a degree of association between keywords, comprising means for transmitting the sum to a management server. A relevance calculation system between keywords comprising a management server and a plurality of distributed processing servers,
Means for distributing the plurality of document files stored in the document storage means to a plurality of first distributed processing servers comprising at least a part of the plurality of distributed processing servers;
Means for transmitting each of the plurality of keywords stored in the keyword storage means to the first distributed processing server;
A plurality of types of combination frequency product files transmitted from each first distributed processing server are assigned to each of a plurality of second distributed processing servers including at least a part of the plurality of distributed processing servers. Means to distribute and distribute according to the type,
Means for storing sum totals over all document files of product values of appearance frequencies between keywords transmitted from the second distributed processing server in keyword combination frequency sum table storage means;
Means for distributing a plurality of document files stored in the document storage means to a plurality of third distributed processing servers comprising at least a part of the plurality of distributed processing servers;
Means for transmitting each of the plurality of keywords stored in the keyword storage means to the third distributed processing server;
Means for transmitting an appearance frequency square value file transmitted from each third distributed processing server to a fourth distributed processing server that is one of the plurality of distributed processing servers;
Means for storing, in the keyword frequency total table storage means, the sum total over all document files of the square value of the appearance frequency of each keyword transmitted from the fourth distributed processing server;
Means for retrieving a pair of keywords from the keyword storage means;
Means for taking out a sum of product values of appearance frequencies between the keywords for the pair of keywords from the keyword combination frequency sum table storage means;
Means for extracting , from the keyword frequency sum table storage means , a sum of square values of appearance frequencies of the keywords for the pair of keywords ;
A means for calculating the degree of relevance between the two keywords by calculating the square root of each sum, adding both square roots, and dividing the sum of the product values of the appearance frequencies between the keywords by this sum,
Each of the first distributed processing servers detects a presence or absence of each keyword for each document file in the document file distributed by the management server;
A means for generating a combination of keywords in which the first character code of the first character is arranged first among a pair of keywords having a history of appearance;
For each pair, for each pair of keywords, means for calculating the product of the appearance frequencies of each keyword as the product of the appearance frequencies between the pair of keywords;
Means for identifying the combination frequency product value file to be described by comparing the character code of the first character of the first keyword with the assigned character code ranges of a plurality of combination frequency product value files to which character code ranges are assigned in advance When,
Means for describing the product value for each document file in a corresponding combination frequency product value file;
Means for transmitting these multiple types of combination frequency product files to the management server,
When each of the second distributed processing servers transmits a plurality of similar combination frequency product value files from the management server, means for connecting the combination frequency product value files;
Means for sorting the keyword combinations described in the concatenated file according to the character code of each keyword;
A means of calculating product values in combination units of the same keyword and calculating a sum total over all document files;
Means for transmitting this sum to the management server,
Each of the above third distributed processing servers detects, for each document file, the presence or absence of each keyword in the document file distributed by the management server;
Means for calculating a square value of the appearance frequency of a keyword with an appearance record;
Means for describing the square value for each document file in an appearance frequency square value file;
Means for transmitting the appearance frequency square value file to the management server,
Means for connecting the appearance frequency square value files when the fourth distributed processing server transmits a plurality of appearance frequency square value files from the management server;
Means for sorting the keywords described in the concatenated file according to their character codes;
Means for calculating the sum of the squares of the appearance frequency in the same keyword unit and calculating the total over all document files;
A system for calculating a degree of association between keywords, comprising means for transmitting the sum to a management server. A relevance calculation system between keywords comprising a management server and a plurality of distributed processing servers,
Means for distributing the plurality of document files stored in the document storage means to a plurality of first distributed processing servers comprising at least a part of the plurality of distributed processing servers;
Means for transmitting each of the plurality of keywords stored in the keyword storage means to the first distributed processing server;
Means for transmitting the combination frequency product value file transmitted from each first distributed processing server to a second distributed processing server that is one of the plurality of distributed processing servers;
Means for storing sum totals over all document files of product values of appearance frequencies between keywords transmitted from the second distributed processing server in keyword combination frequency sum table storage means;
Means for distributing a plurality of document files stored in the document storage means to a plurality of third distributed processing servers comprising at least a part of the plurality of distributed processing servers;
Means for transmitting each of the plurality of keywords stored in the keyword storage means to the third distributed processing server;
A plurality of types of appearance frequency square value files transmitted from each third distributed processing server are assigned to each of a plurality of fourth distributed processing servers including at least a part of the plurality of distributed processing servers. Means to distribute and distribute according to the type,
Means for storing, in the keyword frequency total table storage means, the sum total over all document files of the square value of the appearance frequency of each keyword transmitted from each fourth distributed processing server;
Means for retrieving a pair of keywords from the keyword storage means;
Means for taking out a sum of product values of appearance frequencies between the keywords for the pair of keywords from the keyword combination frequency sum table storage means;
Means for extracting , from the keyword frequency sum table storage means , a sum of square values of appearance frequencies of the keywords for the pair of keywords ;
A means for calculating the degree of relevance between the two keywords by calculating the square root of each sum, adding both square roots, and dividing the sum of the product values of the appearance frequencies between the keywords by this sum,
Each of the first distributed processing servers detects a presence or absence of each keyword for each document file in the document file distributed by the management server;
A means for generating a combination of keywords in which the first character code of the first character is arranged first among a pair of keywords having a history of appearance;
For each pair, for each pair of keywords, means for calculating the product of the appearance frequencies of each keyword as the product of the appearance frequencies between the pair of keywords;
Means for describing the product value for each document file in the combination frequency product value file;
Means for transmitting this combined frequency product value file to the management server,
The second distributed processing server, when a plurality of combination frequency product value files are transmitted from the management server, means for concatenating each combination frequency product value file;
Means for sorting the keyword combinations described in the concatenated file according to the character code of each keyword;
A means of calculating product values in combination units of the same keyword and calculating a sum total over all document files;
Means for transmitting this sum to the management server,
Each of the above third distributed processing servers detects, for each document file, the presence or absence of each keyword in the document file distributed by the management server;
Means for calculating a square value of the appearance frequency of a keyword with an appearance record;
Means for comparing the character code of each keyword with the assigned character code range of a plurality of appearance frequency square value files to which a character code range is assigned in advance, and specifying an appearance frequency square value file to be described;
Means for describing the square value for each document file in the corresponding appearance frequency square value file;
Means for transmitting these multiple types of appearance frequency square value files to the management server,
Each of the fourth distributed processing servers, when a plurality of similar appearance frequency square value files are transmitted from the management server, means for connecting the appearance frequency square value files;
Means for sorting the keywords described in the concatenated file according to their character codes;
Means for calculating the sum of the squares of the appearance frequency in the same keyword unit and calculating the total over all document files;
A system for calculating a degree of association between keywords, comprising means for transmitting the sum to a management server. The management server distributes a plurality of document files stored in the document storage means in advance to a plurality of distributed processing servers including at least a part of the plurality of distributed processing servers, and instructs keyword extraction. Means to
Means for storing the keyword transmitted from each distributed processing server in the keyword storage means,
Each of the distributed processing servers described above extracts a keyword from a responsible document file distributed by the management server;
The system according to claim 3, further comprising means for transmitting each keyword to the management server. A method for calculating a degree of association between keywords based on cooperation between a management server, a plurality of first distributed processing servers, and a second distributed processing server,
The management server distributing a plurality of document files stored in the document storage means to each first distributed processing server;
Each first distributed processing server extracting a keyword from the responsible document file transmitted by the management server and transmitting the keyword to the management server;
The management server stores the keywords transmitted from each first distributed processing server in the keyword storage means, and then transmits all the keywords to the first distributed processing server;
Each of the first distributed processing servers receiving this detects the presence or absence of each keyword for the document file in charge for each document file;
Calculating a square value of an appearance frequency of a keyword having an appearance record and describing each document file in an appearance frequency square value file;
Generating a combination of keywords in which the first character code of the first character is arranged first among a pair of keywords having a record of appearance;
For each pair, for each pair of keywords , calculating the product of the appearance frequency of each keyword as the product value of the appearance frequency between the pair of keywords;
A step of identifying a combination frequency product value file to be described by comparing the character code of the first character of the first keyword with the assigned character code ranges of a plurality of combination frequency product value files to which a character code range is assigned in advance When,
A step of describing a product value of appearance frequencies between keywords related to each combination for each document file in a corresponding combination frequency product value file;
Transmitting the appearance frequency square value file and a plurality of types of combination frequency product value files to the management server;
A management server transmitting a plurality of appearance frequency square value files transmitted from each first distributed processing server to the second distributed processing server;
Distributing and distributing a plurality of types of combination frequency product value files transmitted from each first distributed processing server to the first distributed processing servers each assigned to each type of combination frequency product value file;
A second distributed processing server to which a plurality of appearance frequency square value files are transmitted from the management server, and concatenating each appearance frequency square value file;
Sorting each keyword described in the concatenated file according to each character code;
A step of calculating the square value of the appearance frequency in the same keyword unit and calculating a sum total over all document files;
Sending the sum to the management server;
The management server stores the sum of the squares of the appearance frequencies transmitted from the second distributed processing server in the keyword frequency sum table storage means;
A first distributed processing server to which a plurality of combination frequency product value files are transmitted from the management server, concatenating each combination frequency product value file;
Sorting each keyword combination described in the concatenated file according to the character code of each keyword;
A step of calculating product values of appearance frequencies in combination units of the same keyword and calculating a sum total over all document files;
Sending the sum to the management server;
A management server storing the sum of product values transmitted from each first distributed processing server in the keyword combination frequency sum table storage means;
Retrieving a pair of keywords from the keyword storage means;
Extracting a sum of product values of appearance frequencies between the keywords for the pair of keywords from the keyword combination frequency sum table storage unit;
Extracting the sum of squares of the appearance frequencies of the keywords for the pair of keywords from the keyword frequency total table storage means;
Calculating the relevance between both keywords by calculating the square root of each sum, adding the square root of both keywords, and dividing the sum of the product values of the appearance frequencies between the keywords by this sum;
A method for calculating the degree of association between keywords, comprising:

Images