JP2013077272A - Method, apparatus and computer program for obtaining keyword appearance frequency ranking - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method, apparatus and computer program for efficiently calculating keyword appearance frequency ranking in distributed processing with a plurality of computers.SOLUTION: Keywords which do not overlap each other are assigned to n computers, where n is a natural number. From the keywords assigned to each computer, keywords of the highest frequency ranks are acquired. From all the acquired keywords, k keywords of the highest overall frequency ranks are selected, where k is a natural number. In doing this, probability P(n, k, t) that the keywords of the highest frequency ranks in each computer are selected as the k highest overall frequency ranks is calculated, where t is a natural number satisfying t<k and represents the number of keywords acquired from each computer as the keywords of the highest frequency ranks. Then, t is estimated on the basis of the calculated probability P(n, k, t), t keywords of the highest frequency ranks are acquired from each computer, and k keywords of the highest overall frequency ranks are selected from all the acquired keywords.

Description

  The present invention relates to a method, an apparatus, and a computer program for efficiently obtaining a frequency order in which keywords appear when a frequency count of keywords appearing in a plurality of computers is distributed.

  Like so-called text mining, a list of k data (keyword in the case of text mining) and appearance frequency and a frequency of appearance are obtained from a large amount of data (document data in the case of text mining). In the system, the calculation processing load becomes excessive. In particular, when performing interactive text analysis, the next search keyword is selected from the top k keywords, and the top k keyword list is newly obtained for the search result based on the selected keyword. Since a process for obtaining k keywords is executed, how to shorten the response time is an important issue.

  When the number of documents is enormous, the processing is generally divided into a plurality of computers to perform parallel processing. For example, the document set is divided into small sets, each small set is arranged and stored in a plurality of computers, and the top k keywords in order of frequency are obtained by each computer. Then, the top k keywords in order of frequency are obtained by aggregating the appearance frequencies of the keywords obtained by all computers. That is, when the number of computers is n (n is a natural number), the top k in order of frequency are selected from k × n.

  However, in the above-described method, there is no guarantee that the top k keywords in the correct frequency order are obtained as a whole. Therefore, rather than dividing a document set into a plurality of computers and storing the information, for example, as in Patent Document 1, information relating to a keyword for determining an appearance frequency is divided into a plurality of computers and stored. Then, it is possible to obtain the top k keywords in the frequency order only for the stored keywords, and aggregate them to obtain the top k keywords in the correct frequency order as a whole.

Japanese Patent No. 4172801

  However, when distributed processing is performed with n (n is a natural number) computers, in order to calculate the top k keywords in the order of frequency, it is necessary to obtain the appearance frequency for n × k keywords. For example, when n = 32 and k = 100, the number of keywords to be finally obtained is 100, but it is necessary to obtain the appearance frequency of 32 × 100−100 = 3100 unnecessary keywords. Yes, efficiency is poor.

  The present invention has been made in view of such circumstances, and an object thereof is to provide a method, an apparatus, and a computer program for efficiently calculating the frequency order in which keywords appear when distributed processing is performed by a plurality of computers. .

  In order to achieve the above object, the method according to the first aspect of the present invention is a method that can be executed by a device that obtains a keyword in order of frequency using a plurality of computers, wherein the device has n (n is a natural number). ) Assigning non-overlapping keywords to the computers, and acquiring the top t keywords (t is a natural number, t <k) in order of frequency from the keywords assigned to each computer, and from all the acquired keywords A step of calculating a probability P (n, k, t) that the selected k items are the top k in the true frequency order when selecting k keywords in the order of frequency as a whole (k is a natural number); Based on the calculated probability P (n, k, t), the number t of keywords to be acquired from each computer as the top keyword in frequency order (t is a natural number, t <k) is estimated. That has a step, to obtain the frequency-ordered upper t keyword (s) from each computer, to select the frequency order top k keyword as a whole from all the keywords acquired.

  Further, in the method according to the second invention, in the first invention, the probability P (n, k, t) is “a computer to be selected when selecting k keywords in order of frequency from n × t keywords. The probability of selecting at most (t−1) keywords from any computer in the combination of the number of keywords for each.

  In the method according to the third invention, in the first invention, the probability P (n, k, t) is “when the top k keywords in the frequency order are selected from the n × t keywords including the frequency order”. In addition, in the combination of the number of keywords for each computer to be selected, the probability is that at most (t−1) keywords are selected from any computer.

  In the method according to the fourth invention, in the first invention, the probability P (n, k, t) is selected when “a combination that selects k keywords in order of frequency from n computers is selected”. The probability of selecting at most t keywords from any computer in the combination.

  Next, in order to achieve the above object, a method according to the fifth aspect of the present invention is a method that can be executed by a device that obtains a keyword in order of frequency using a plurality of computers, and the device includes n ( n is a natural number) assigning non-overlapping keywords to each computer, obtaining the top t (t is a natural number) keywords in order of frequency from each computer, and the order of frequency from the keywords assigned to each computer Keywords acquired as top keywords in order of frequency in each computer when top keywords are acquired and k keywords (k is a natural number, t <k) in order of frequency are selected from all the acquired keywords as a whole The number of keywords is (t-1) or less and the smallest of the keywords selected in each client And the step of determining whether or not the frequency of the t-th keyword of the client is different, and the number of keywords acquired as a top-order keyword in each computer is (t−1) or less. And when it is determined that the minimum frequency among the keywords selected in each client is different from the frequency of the t-th keyword of the client, among all the keywords acquired from each computer, -1) Select the top k keywords in order of frequency from the keywords.

  Next, in order to achieve the above object, an apparatus according to the sixth aspect of the present invention is an apparatus that uses a plurality of computers to obtain keywords in order of frequency and overlaps n (n is a natural number) computers. When a keyword assigning unit for assigning non-keywords and keywords assigned to each computer are acquired from the keywords assigned to each computer in the order of frequency, and k keywords (k is a natural number) as a whole are selected from all the acquired keywords. In addition, based on the probability calculation unit for calculating the probability P (n, k, t) that the top keyword in the frequency order of each computer is the top k in the frequency order, and the calculated probability P (n, k, t), An estimation unit for estimating the number of keywords t (t is a natural number, t <k) to be acquired from each computer as a keyword in order of frequency. To get the order of frequency of the upper t number of keywords from over data, to select the order of frequency of top k keyword as a whole from all of the keywords you get.

  According to a seventh aspect of the present invention, in the sixth aspect of the invention, the probability P (n, k, t) is “a computer to be selected when selecting the top k keywords in order of frequency from the n × t keywords. The probability of selecting at most (t−1) keywords from any computer in the combination of the number of keywords for each.

  In the device according to the eighth invention, in the sixth invention, the probability P (n, k, t) is “when the top k keywords in order of frequency are selected from the n × t keywords including the frequency order”. In addition, in the combination of the number of keywords for each computer to be selected, the probability is that at most (t−1) keywords are selected from any computer.

  In the device according to the ninth invention, in the sixth invention, the probability P (n, k, t) is selected when “a combination that selects k keywords in order of frequency from n computers is selected”. The probability of selecting at most t keywords from any computer in the combination.

  Next, in order to achieve the above object, an apparatus according to the tenth aspect of the present invention is an apparatus that uses a plurality of computers to obtain keywords in order of frequency and overlaps n (n is a natural number) computers. A keyword assigning unit that assigns no keywords, a keyword obtaining unit that obtains the top t (t is a natural number) keywords in order of frequency from each computer, and a keyword in the order of frequency obtained from the keywords assigned to each computer. In the case where the top k (k is a natural number, t <k) keywords in order of frequency are selected from all the keywords as a whole, the number of keywords acquired as the top keywords in frequency order by each computer is (t−1). The minimum frequency of keywords selected by each client and their clients A determination unit for determining whether or not the frequency of the t-th keyword is different, and the number of keywords acquired as top-ranked keywords in each computer is (t−1) or less, and each client If it is determined that the minimum frequency among the keywords selected in step 1 differs from the frequency of the t-th keyword of the client, the top (t−1) items in order of frequency of each computer out of all the keywords acquired from each computer The top k keywords in order of frequency are selected as a whole.

  Next, in order to achieve the above object, a computer program according to an eleventh aspect of the present invention is a computer program that can be executed by a device that obtains a keyword in order of frequency using a plurality of computers, Keyword assigning means for assigning non-overlapping keywords to n (n is a natural number) computers, keywords in higher frequency order are obtained from the keywords assigned to each computer, and the overall frequency order is obtained from all the obtained keywords. Probability calculating means for calculating the probability P (n, k, t) that the top keyword in the frequency order of each computer is k when selecting k (k is a natural number) keywords, the calculated probability Based on P (n, k, t), obtain from each computer as the top keyword in frequency order Estimating means for estimating the number of keywords t (t is a natural number, t <k), and the top t keywords in frequency order from each computer, and the top k keywords in frequency order as a whole from all the acquired keywords It functions as a means for selecting.

  In the computer program according to the twelfth aspect, in the eleventh aspect, the probability P (n, k, t) is selected when “k keywords having the highest frequency order are selected from n × t keywords”. The probability of selecting at most (t−1) keywords from any computer in the combination of the number of keywords for each computer.

  In the computer program according to the thirteenth aspect, in the eleventh aspect, the probability P (n, k, t) selects “k keywords including the frequency order from the n × t keywords in the order of frequency”. In this case, the probability of selecting at most (t−1) keywords from any computer in the combination of the number of keywords for each computer to be selected.

  Further, in the computer program according to the fourteenth invention, in the eleventh invention, the probability P (n, k, t) is “when a combination that selects k keywords in order of frequency from n computers is selected, It is the probability of selecting at most t keywords from any computer in the selected combination.

  Next, in order to achieve the above object, a computer program according to the fifteenth aspect of the present invention is a computer program that can be executed by a device that obtains keywords in order of frequency using a plurality of computers, Keyword assignment means for assigning non-overlapping keywords to n (n is a natural number) computers, keyword acquisition means for obtaining top t (t is a natural number) keywords in order of frequency from each computer, assigned to each computer When keywords with the highest frequency order are acquired from the keywords and k keywords with the highest frequency order (k is a natural number, t <k) are selected as a whole from all the acquired keywords, The number of acquired keywords is (t-1) or less, and The judging means for judging whether or not the minimum frequency among the keywords selected in the client is different from the frequency of the t-th keyword of the client, and the number of keywords acquired by each computer as the top keywords in the frequency order ( t-1) or less, and if it is determined that the minimum frequency of the keywords selected in each client is different from the frequency of the t-th keyword of the client, among all the keywords acquired from each computer, It is made to function as means for selecting the top k keywords in order of frequency from the top (t−1) keywords in order of frequency of each computer.

  According to the present invention, in order to obtain the top k keywords in the order of frequency, it is not necessary to obtain the top k keywords in the order of frequency in all the computers, and only t keywords less than k need to be obtained. The calculation processing load on each computer can be reduced, and the number of top keywords in order of frequency in each computer can be narrowed down in order to obtain the top k keywords in order of frequency. Time can be greatly reduced.

It is a block diagram which shows typically the structure of the frequency order keyword extraction system which concerns on embodiment of this invention. It is a functional block diagram of an information processor concerning an embodiment of the invention. It is an illustration figure of the keyword allocation of the frequency order keyword extraction system which concerns on embodiment of this invention. It is a schematic diagram for demonstrating the calculation method of the probability P (n, k, t) of the frequency order keyword extraction system which concerns on embodiment of this invention. It is a flowchart which shows the process sequence of CPU of the information processing apparatus which concerns on embodiment of this invention. It is a functional block diagram in the case of not calculating the probability of the information processing apparatus according to the embodiment of the present invention. It is a flowchart which shows the process sequence when not calculating the probability of CPU of the information processing apparatus which concerns on embodiment of this invention. It is an illustration figure of the screen which receives designation | designated of the selection number k and threshold value probability p. It is a table | surface which shows the example of the calculated value of t for every probability calculation method. It is a table | surface which shows the execution time (second) of the process which selects k keyword by the frequency order top when n = 32 and p = 90%.

  Hereinafter, an apparatus for obtaining keywords in order of frequency according to an embodiment of the present invention will be specifically described with reference to the drawings. The following embodiments do not limit the invention described in the claims, and all combinations of characteristic items described in the embodiments are essential to the solution. It goes without saying that it is not limited.

  The present invention can be implemented in many different modes and should not be construed as being limited to the description of the embodiment. The same symbols are attached to the same elements throughout the embodiments.

  In the following embodiments, an apparatus in which a computer program is introduced into a computer system will be described. As will be apparent to those skilled in the art, the present invention is implemented as a computer program that can be partially executed by a computer. be able to. Therefore, the present invention can take the form of hardware as an apparatus for calculating the frequency order in which keywords appear, the form of software, or the combination of software and hardware. The computer program can be recorded on any computer-readable recording medium such as a hard disk, DVD, CD, optical storage device, magnetic storage device or the like.

  According to the embodiment of the present invention, in order to obtain the top k keywords in the frequency order, it is not necessary to obtain the top k keywords in the frequency order in all the plurality of computers, and only t keywords less than k are obtained. Since the calculation processing load on each computer can be reduced, the number of top keywords in order of frequency in each computer can be narrowed down to obtain the top k keywords in order of frequency as a whole. The response time of the search result can be greatly shortened.

  FIG. 1 is a block diagram schematically showing the configuration of a frequency-order keyword extraction system according to an embodiment of the present invention. A frequency-order keyword extraction system according to an embodiment of the present invention includes an information processing apparatus 1 connected to a plurality of clients (computers) 2 so as to be able to perform data communication via a network 3. . The information processing apparatus 1 includes at least a CPU (central processing unit) 11, a memory 12, a storage device 13, an I / O interface 14, a video interface 15, a portable disk drive 16, a communication interface 17, and an internal to which the hardware described above is connected. It consists of a bus 18.

  The CPU 11 is connected to the above-described hardware units of the information processing apparatus 1 via the internal bus 18, controls the operation of the above-described hardware units, and follows the computer program 100 stored in the storage device 13. Perform various software functions. The memory 12 is composed of a volatile memory such as SRAM or SDRAM, and a load module is expanded when the computer program 100 is executed, and stores temporary data generated when the computer program 100 is executed.

  The storage device 13 includes a built-in fixed storage device (hard disk), a ROM, and the like. The computer program 100 stored in the storage device 13 is downloaded by the portable disk drive 16 from a portable recording medium 90 such as a DVD or CD-ROM in which information such as programs and data is recorded, and from the storage device 13 at the time of execution. The program is expanded into the memory 12 and executed. Of course, a computer program downloaded from an external computer connected via the communication interface 17 may be used.

  The communication interface 17 is connected to an internal bus 18 and can transmit / receive data to / from an external computer or the like by connecting to an external network such as the Internet, LAN, or WAN.

  The I / O interface 14 is connected to input devices such as a keyboard 21 and a mouse 22 and receives data input. The video interface 15 is connected to a display device 23 such as a CRT display or a liquid crystal display, and displays a predetermined image.

  FIG. 2 is a functional block diagram of the information processing apparatus 1 according to the embodiment of the present invention. In FIG. 2, the keyword assignment unit 201 of the information processing apparatus 1 assigns keywords that do not overlap each other to n (n is a natural number) clients 2.

  FIG. 3 is an exemplary diagram of keyword assignment in the frequency-order keyword extraction system according to the embodiment of the present invention. FIG. 3 shows a state in which keywords are assigned to n (n = 3) clients 2 in order to calculate the top k (k = 4) keywords in order of frequency. Specifically, the keywords A, B, C,... Are assigned to the client S1, the keywords E, F, G,. The keywords P, Q, R,... That do not overlap with the keywords of the clients S1 and S2 are respectively assigned.

  Each client 2 obtains the appearance frequency for the assigned keyword, and lists the keywords in order of frequency. Then, the information processing apparatus 1 acquires the keywords and appearance frequencies listed in all the clients 2, and as a whole, the top four keywords in the order of frequency (k = 4), the top four keywords in the order of frequency in the example of FIG. E, A, P, and F are obtained in order of frequency.

  Returning to FIG. 2, the probability calculation unit 202 acquires keywords in the highest order of frequency from the keywords assigned to each client 2, and k keywords (k is a natural number) in the order of frequency as a whole from all the acquired keywords. Is selected, the probability P (n, k, t) that the top keyword in the frequency order of each client 2 is k is calculated. The calculation method of the probability P (n, k, t) is not particularly limited.

  For example, when selecting the top k keywords in order of frequency from n × t keywords with probability P (n, k, t), “select top k keywords in order of frequency from n × t keywords In this case, in the combination of the number of keywords for each computer to be selected, it may be obtained as a probability of “selecting at most (t−1) keywords from any computer” (probability calculation method (1)). When a combination of selecting k keywords from n × t keywords obtained by aggregating t keywords acquired by each client 2 is f (n, k, t), the probability P (n, k, t) is expressed by It is defined as shown in 1).

  In (Expression 1), the combination f (n, k, t) is given by (Expression 2). Accordingly, the probability P (n, k, t) can be directly calculated by substituting values for n, k, and t in (Expression 2).

  The calculation amount of the arithmetic processing based on (Equation 2) is calculated as 0.5 × n × (k / t) 2. On the other hand, f (n, k, t) can also be calculated using dynamic programming. (Equation 3) is a recurrence formula when f (n, k, t) is calculated using dynamic programming.

  Further, the calculation amount of the arithmetic processing based on (Equation 3) is calculated as n × k. Therefore, for example, when the value of t is 5% of k, (k / t) 2 is 400. Therefore, if k> 200, the amount of calculation processing is smaller in (Equation 2).

  Further, when the probability P (n, k, t) is selected in order of frequency from the n × t keywords in the order of frequency, “k keywords in the order of frequency from the n × t keywords” are selected. May be determined as a probability of selecting (t−1) keywords from any computer in a combination of the number of keywords for each computer to be selected (probability calculation). Method (2)). In this case, since the frequency order of the top k keywords in the frequency order is taken into consideration, the probability P (n, k, t) can be obtained using a combination of assignment methods that distinguish and assign k keywords. That is, among the combinations that sequentially assign the top k keywords in the correct frequency order to the n clients 2, the combination of assignment methods in which the number of keywords sequentially assigned to each client 2 is t or less is g (n, k, t), the probability P (n, k, t) is defined as shown in (Equation 4).

  In (Expression 4), g (n, k, t) can be calculated using dynamic programming. (Formula 5) is a recurrence formula when g (n, k, t) is calculated using dynamic programming.

  The probability P (n, k, t) is set to “when a combination that selects k keywords in order of frequency from n computers is selected, at most t from any computer in the selected combination. As a probability of “selecting a keyword” (probability calculation method (3)). When the top k keywords in the correct frequency order are assigned to n clients 2, the probability P () is obtained using a combination g (n, k, t) in which the number of keywords assigned to each client 2 is t or less. (n, k, t) are defined as shown in (Equation 6). Note that g (n, k, ∞) indicates the number of combinations when there is no limit of t, and is therefore equal to n raised to the kth power.

  In (Expression 6), the recurrence formula when g (n, k, t) is calculated using dynamic programming is the same as (Expression 5).

  FIG. 4 is a schematic diagram for explaining a method of calculating the probability P (n, k, t) of the frequency-order keyword extraction system according to the embodiment of the present invention. FIG. 4A shows n × t keywords acquired from the n clients S1, S2,...

  FIG. 4B shows a state where the top k items in the order of frequency are selected from the n × t keywords shown in FIG. The hatched portion in FIG. 4B indicates the selected keyword. In the example of FIG. 4B, t keywords having the largest number of keywords acquired from the client S2 are selected. In this case, if the top t keywords in frequency order are not acquired from all clients, the top k in order of frequency as a whole. Keywords cannot be selected correctly.

  FIG. 4C also shows a state in which the top k items in the order of frequency are selected from the n × t keywords shown in FIG. The hatched portion in FIG. 4C indicates the selected keyword. In the example of FIG. 4C, (t−1) keywords having the largest number of keywords acquired from the client Sn are selected, and there are no clients for which t keywords have been selected. In this case, the top k keywords in order of frequency can be correctly selected as a whole.

  In the probability calculation method (1), the total number of hatched portions is selected without distinguishing the keyword frequency order. That is, as long as it is selected from one client Si, it does not distinguish what the frequency order is. On the other hand, in the probability calculation methods (2) and (3), the keyword frequency order is distinguished and selected.

  When selecting without distinguishing the frequency order of keywords, the case where keywords are selected equally from all clients and the case where the number of keywords selected for each client is biased are considered with the same weight. . When k keywords are randomly assigned to clients, there is a higher possibility that keywords will be selected equally. Therefore, the probability calculation method (1) in which the keyword frequency order is selected without distinction will estimate the value of t larger.

  Returning to FIG. 2, the keyword acquisition number estimating unit (estimating means) 203 acquires the number t () of keywords to be acquired from each client 2 as the top keyword in frequency order based on the calculated probability P (n, k, t). t is a natural number, and t <k) is estimated. Specifically, the probability P (n, k, t) calculated by the probability calculation unit 202 is estimated to be t that is equal to or higher than a predetermined threshold probability p.

  The keyword acquisition unit 204 acquires the top t keywords in the order of frequency from each client 2. The keyword selection unit 205 selects k keywords (k is a natural number) in order of frequency from all the keywords acquired from the n clients 2.

  The result notification unit 206 notifies the user of information regarding whether or not the selected keyword is accurate. The notification method to the user is not particularly limited, and may be displayed and output on the display device 23, or a fixed message may be sent by e-mail to a predetermined address.

  Specifically, the selected top k keywords in the frequency order are selected from the top (t−1) keywords in the order of frequency of each client 2, and the minimum frequency among the keywords selected in each client 2 is selected. When the frequency of the t-th keyword of the client 2 is different from that of the client 2, the result notification unit 206 notifies that the selection result is accurate. In other cases, that is, when any one of the clients 2 selects a keyword from the top t keywords in the frequency order, the frequency higher than the frequency of the keyword having the highest appearance frequency among the top t keywords in the frequency order is set. For the keywords that have them, the fact that the selection result is accurate is notified.

  FIG. 5 is a flowchart showing the processing procedure of the CPU 11 of the information processing apparatus 1 according to the embodiment of the present invention. In FIG. 5, the CPU 11 of the information processing apparatus 1 assigns keywords that do not overlap each other to n (n is a natural number) clients 2 that are connected so as to be able to perform data communication (step S501).

  When the CPU 11 acquires a keyword in the highest order of frequency from the keywords assigned to each client 2 and selects k keywords (k is a natural number) in order of frequency from all the acquired keywords as a whole, each client 2 The probability P (n, k, t) that the top keywords in the frequency order are k in the frequency order is calculated (step S502). Any of the above-described probability calculation methods (1) to (3) is used as a calculation method of the probability P (n, k, t).

  Based on the calculated probability P (n, k, t), the CPU 11 estimates the number t of keywords (t is a natural number, t <k) acquired from each client 2 as the top keyword in the order of frequency (step S503). . Specifically, the probability P (n, k, t) calculated by the probability calculation unit 202 is estimated to be t that is equal to or higher than a predetermined threshold probability p.

  The CPU 11 acquires the top t keywords in the order of frequency from each client 2 (step S504). The CPU 11 selects k keywords (k is a natural number) in order of frequency from the n × t keywords acquired from the n clients 2 as a whole (step S505).

  The CPU 11 selects the top k keywords in the selected frequency order from the top (t−1) keywords in the frequency order of each client 2, and sets the minimum frequency among the keywords selected in each client 2 and the client 2. It is determined whether or not the frequency of the t-th keyword is different (step S506). The CPU 11 is selected from the top (t−1) in order of frequency of each client 2, and the minimum frequency among the keywords selected in each client 2 is different from the frequency of the t-th keyword of the client 2. If the CPU 11 determines that the selection result is correct (step S506: YES), the CPU 11 displays and outputs a message indicating that the selection result is accurate (step S507). The CPU 11 is selected from the top t in the order of frequency in any of the clients 2, or the minimum frequency among the keywords selected in each client 2 is the same as the frequency of the t-th keyword of the client 2. If it is determined that it is (step S506: NO), the CPU 11 displays and outputs a message indicating that the selection result is accurate for a keyword having a frequency greater than the frequency of the t-th keyword (step S508). .

  In other words, the top k keywords in the frequency order are selected from the top (t−1) keywords in the frequency order of each client 2 and the minimum frequency among the keywords selected in each client 2 and its keywords When the frequency of the t-th keyword of the client 2 is different, the fact that the top k keywords in the selected frequency order are accurate is used. Therefore, the top t frequency keywords (t is a natural number) are obtained from each client 2, and the k top keywords in order of frequency selected as a whole become the top frequency order (t−1) of each client 2. ) Whether the keyword has been selected correctly by determining whether or not the minimum frequency of the keywords selected in each client 2 is different from the frequency of the t-th keyword of the client 2 It can be determined whether or not. FIG. 6 is a functional block diagram of the information processing apparatus 1 according to the embodiment of the present invention when the probability is not calculated. In FIG. 6, the keyword assignment unit 601 of the information processing device 1 assigns keywords that do not overlap each other to n (n is a natural number) clients 2.

  The keyword acquisition unit 602 acquires the top t keywords in order of frequency from each client 2. The determination unit 603 acquires keywords in the highest order of frequency from the keywords assigned to each client 2 and selects k keywords in the order of frequency (k is a natural number, t <k) as a whole from all the acquired keywords. In this case, it is determined whether or not the number of keywords acquired as the highest-order keyword in each client 2 is (t−1) or less.

  When the keyword selection unit 604 determines that the number of keywords acquired by each client 2 as the highest-order keyword in each client 2 is (t−1) or less by the determination unit 603, all keywords acquired from each client 2 are obtained. Among them, the top k keywords in order of frequency are selected from the top (t−1) keywords in order of frequency of each client 2.

  The result notification unit 605 displays and outputs the selected keywords on the display device 23 in order from the highest frequency.

  FIG. 7 is a flowchart showing a processing procedure when the CPU 11 of the information processing apparatus 1 according to the embodiment of the present invention does not calculate the probability. In FIG. 7, the CPU 11 of the information processing apparatus 1 assigns keywords that do not overlap each other to n (n is a natural number) clients 2 connected so as to be able to perform data communication (step S <b> 701).

  The CPU 11 acquires the top t keywords in the order of frequency from each client 2 (step S702), acquires the keywords in the top in the order of frequency from the keywords assigned to each client 2, and orders all the acquired keywords in the order of frequency. When the top k (k is a natural number, t <k) keywords are selected, the number of keywords acquired as the top keywords in the frequency order in each client 2 is (t−1) or less, and each client It is determined whether or not the minimum frequency among the keywords selected in 2 is different from the frequency of the t-th keyword of the client 2 (step S703).

  The number of keywords acquired by the CPU 11 as the top keywords in the frequency order in each client 2 is (t−1) or less, and the minimum frequency among the keywords selected in each client 2 and the t of the client 2 If it is determined that the frequency of the second keyword is different (step S703: YES), the CPU 11 selects all the keywords acquired from each client 2 in the order of frequency (t−1) keywords of each client 2. As a whole, the top k keywords in the order of frequency are selected (step S704). The CPU 11 displays and outputs the selected keywords on the display device 23 in order from the highest frequency order (step S705).

  The number of keywords acquired by the CPU 11 as a higher frequency order in any one of the clients 2 is t, or the minimum frequency among the keywords selected in each client 2 and the frequency of the t-th keyword of the client 2 Are determined to be the same (step S703: NO), the CPU 11 displays and outputs a message indicating that the selection result is accurate for a keyword having a frequency greater than that of the t-th keyword (step S703). S706).

  The frequency order keyword extraction system described above is applied to the extraction of frequency order keywords from document data in so-called text mining. For example, in the frequency-order keyword extraction system having the same configuration as in FIG. 1, n clients 2 are distinguished from clients S1, S2,. Then, a hash value is calculated for each keyword by a known method, and a value obtained by adding 1 to a remainder obtained by dividing the calculated hash value by the number of clients n (n is a natural number) is used as an argument i. ,..., Sn are assigned keywords.

  The user designates the selected number k and a predetermined threshold probability p in order to extract the top k in order of frequency. FIG. 8 is a view showing an example of a screen for accepting designation of the selected number k and the threshold probability p. The example of FIG. 8 includes a selection number receiving area 81 that receives the designation of the selection number k, and a probability designation area 82 that receives the designation of the threshold probability p. By accepting the selection of the “search” button 83 in the state of accepting the designation, the top k keywords in the order of frequency can be extracted.

  Of course, the value of t satisfying P (n, k, t)> p is set to any one of the above probability calculation methods (1) to (3) according to the selected number k and the threshold probability p received. Needless to say, use and calculate in advance.

  FIG. 9 is a table showing an example of a calculated value of t for each probability calculation method. For example, when n = 32 and k = 100, the value of t that can select the top k keywords in the correct frequency order with a probability of 90% or more is t = 16 in the probability calculation method (1), and the probability In the calculation methods (2) and (3), t = 9. In addition, the value of t that can select the top k keywords in the correct frequency order with a probability of 95% or more is t = 18 in the probability calculation method (1), and in the probability calculation methods (2) and (3). t = 10. In either case, the value of t is smaller than k = 100.

  FIG. 10 is a table showing the execution time (seconds) of the process of selecting the top k keywords in order of frequency when n = 32 and p = 90%. FIG. 10 shows execution times when k = 100, 1000, and 10000, respectively. “Conventional” indicates the execution time in the conventional method, and “1” indicates the execution time in the probability calculation method (1). As can be seen from FIG. 10, by using the frequency order keyword extraction system according to the present embodiment, the calculation processing speed is increased from 1.5 times to 2.5 times.

  As described above, according to the present embodiment, in order to obtain the top k keywords in the frequency order, it is not necessary to obtain the top k keywords in the frequency order in all of the plurality of clients 2, and t less than k keywords are obtained. Since only the keywords need to be obtained, the calculation processing load on each client 2 can be reduced, and in order to obtain the top k keywords in order of frequency as a whole, the number of keywords in the top order of frequency obtained in each client 2 is determined. Since it can narrow down, it becomes possible to greatly shorten the response time of the search result.

  The present invention is not limited to the above-described embodiments, and various changes and improvements can be made within the scope of the present invention. For example, instead of providing the information processing apparatus 1 separately, one of the clients 2 may be an apparatus that obtains a keyword in order of frequency.

1 Information processing device 2 Client 11 CPU
12 Memory 13 Storage Device 14 I / O Interface 15 Video Interface 16 Portable Disk Drive 17 Communication Interface 18 Internal Bus 90 Portable Recording Medium 100 Computer Program

Claims (15)

It is a method that can be executed by a device that obtains keywords in order of frequency using a plurality of computers,
The device is
assigning non-overlapping keywords to n (n is a natural number) computers;
When a keyword with the highest frequency order is acquired from the keywords assigned to each computer and k keywords with the highest frequency order (k is a natural number) are selected as a whole from all the acquired keywords, Calculating a probability P (n, k, t) that the keyword is the top k in frequency order;
Based on the calculated probability P (n, k, t), estimating the number t of keywords (t is a natural number, t <k) to be acquired from each computer as the top keyword in the order of frequency, and
A method of acquiring the top t keywords in frequency order from each computer and selecting the top k keywords in order of frequency from all the acquired keywords.   The probability P (n, k, t) can be determined from any computer in the combination of the number of keywords for each computer to be selected when the top k keywords in order of frequency are selected from the n × t keywords ( The method according to claim 1, wherein t−1) is a probability of selecting keywords.   The probability P (n, k, t) is any computer in the combination of the number of keywords for each computer to be selected when the top k keywords in order of frequency are selected from the n × t keywords including the frequency order. The method according to claim 1, wherein the probability is that at most (t−1) keywords are selected.   The probability P (n, k, t) is determined by selecting at most t keywords from any computer in the selected combination when selecting a combination that selects k keywords in order of frequency from n computers. The method of claim 1, wherein the method is a probability of selection. It is a method that can be executed by a device that obtains keywords in order of frequency using a plurality of computers,
The device is
assigning non-overlapping keywords to n (n is a natural number) computers;
Obtaining t keywords (t is a natural number) in order of frequency from each computer;
When acquiring the top keywords in the order of frequency from the keywords assigned to each computer, and selecting all k keywords (k is a natural number, t <k) in order of frequency from all the acquired keywords, Whether or not the number of keywords acquired as the top keywords in the frequency order is (t-1) or less, and the minimum frequency among the keywords selected in each client is different from the frequency of the t-th keyword of the client A step of determining whether or not
The number of keywords acquired as high-order keywords in each computer is (t−1) or less, and the minimum frequency among the keywords selected in each client and the frequency of the t-th keyword of that client are A method of selecting the top k keywords in order of frequency from the top (t−1) keywords in order of frequency among all the keywords acquired from each computer when it is determined that they are different. A device that uses a plurality of computers to obtain keywords in order of frequency,
a keyword assigning unit for assigning non-overlapping keywords to n (n is a natural number) computers;
When a keyword with the highest frequency order is acquired from the keywords assigned to each computer and k keywords with the highest frequency order (k is a natural number) are selected as a whole from all the acquired keywords, A probability calculating unit that calculates the probability P (n, k, t) that the keyword is the top k in frequency order;
An estimation unit that estimates the number of keywords t (t is a natural number, t <k) to be acquired from each computer as a top keyword in order of frequency based on the calculated probability P (n, k, t);
An apparatus that acquires the top t keywords in order of frequency from each computer, and selects the top k keywords in order of frequency from all the acquired keywords.   The probability P (n, k, t) can be determined from any computer in the combination of the number of keywords for each computer to be selected when the top k keywords in order of frequency are selected from the n × t keywords ( The apparatus according to claim 6, wherein t−1) is a probability of selecting keywords.   The probability P (n, k, t) is any computer in the combination of the number of keywords for each computer to be selected when the top k keywords in order of frequency are selected from the n × t keywords including the frequency order. The apparatus according to claim 6, wherein the probability is that at most (t−1) keywords are selected.   The probability P (n, k, t) is determined by selecting at most t keywords from any computer in the selected combination when selecting a combination that selects k keywords in order of frequency from n computers. The apparatus of claim 6, wherein the apparatus is a probability of selection. A device that uses a plurality of computers to obtain keywords in order of frequency,
a keyword assigning unit for assigning non-overlapping keywords to n (n is a natural number) computers;
A keyword acquisition unit that acquires the top t (t is a natural number) keywords in order of frequency from each computer;
When acquiring the top keywords in the order of frequency from the keywords assigned to each computer, and selecting all k keywords (k is a natural number, t <k) in order of frequency from all the acquired keywords, Whether or not the number of keywords acquired as the top keywords in the frequency order is (t-1) or less, and the minimum frequency among the keywords selected in each client is different from the frequency of the t-th keyword of the client And a determination unit for determining whether
The number of keywords acquired as high-order keywords in each computer is (t−1) or less, and the minimum frequency among the keywords selected in each client and the frequency of the t-th keyword of that client are An apparatus that selects k keywords as a whole in order of frequency from the top (t−1) keywords in order of frequency among all keywords acquired from each computer when it is determined that they are different. A computer program that can be executed by a device that determines a keyword in order of frequency using a plurality of computers,
Said device,
keyword assigning means for assigning non-overlapping keywords to n (n is a natural number) computers;
When a keyword with the highest frequency order is acquired from the keywords assigned to each computer and k keywords with the highest frequency order (k is a natural number) are selected as a whole from all the acquired keywords, A probability calculating means for calculating a probability P (n, k, t) that the keyword has the top k in the frequency order;
Based on the calculated probability P (n, k, t), an estimation means for estimating the number t of keywords to be acquired from each computer as the top keyword in the order of frequency (t is a natural number, t <k), and the frequency from each computer A computer program that obtains the top t keywords in order and functions as means for selecting the top k keywords in order of frequency from all the obtained keywords.   The probability P (n, k, t) can be determined from any computer in the combination of the number of keywords for each computer to be selected when the top k keywords in order of frequency are selected from the n × t keywords ( The computer program according to claim 11, which is a probability of selecting t−1) keywords.   The probability P (n, k, t) is any computer in the combination of the number of keywords for each computer to be selected when the top k keywords in order of frequency are selected from the n × t keywords including the frequency order. The computer program according to claim 11, wherein the computer program has a probability of selecting at most (t−1) keywords.   The probability P (n, k, t) is determined by selecting at most t keywords from any computer in the selected combination when selecting a combination that selects k keywords in order of frequency from n computers. The computer program according to claim 11, wherein the computer program is a probability of selection. A computer program that can be executed by a device that determines a keyword in order of frequency using a plurality of computers,
Said device,
keyword assigning means for assigning non-overlapping keywords to n (n is a natural number) computers;
Keyword acquisition means for acquiring t keywords (t is a natural number) in order of frequency from each computer;
When acquiring the top keywords in the order of frequency from the keywords assigned to each computer, and selecting all k keywords (k is a natural number, t <k) in order of frequency from all the acquired keywords, Whether or not the number of keywords acquired as the top keywords in the frequency order is (t-1) or less, and the minimum frequency among the keywords selected in each client is different from the frequency of the t-th keyword of the client A determination means for determining whether or not the number of keywords acquired as top-ranked keywords in each computer is (t-1) or less, and the minimum frequency among the keywords selected in each client and the client If it is determined that the frequency of the t th keyword is different from each computer, A computer program that functions as means for selecting, from all acquired keywords, the top k keywords in order of frequency from the top (t-1) keywords in order of frequency of each computer.

Images