JP2005275440A - Data analyzing method and information processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To increase overall processing efficiency by reducing the standby time between processors in data analysis to be performed by using a plurality of processors. <P>SOLUTION: Data analysis processing is carried out by using one master 0101 and a plurality of slaves 0109 connected through a communication path 0110. In this data analysis processing, the master 0101 divides the data analysis problem into the larger number of partial problems(jobs) than the number of the slaves 0109, and assigns those jobs to the slaves. The slaves 0109 process the jobs assigned from the master 0101, and store the processing results in the slaves, and repeats the processing to request the jobs to the master until the assignment of the jobs from the master stops. Finally, the slaves 0109 transmit the stored processing results to the master 0101. The master 0101 receives the analytic results from all the slaves 0109, and outputs the overall analytic result. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a data analysis method, and more particularly to a data analysis method for performing analysis in parallel using a plurality of data processing devices.

  A technique for discovering knowledge from a large amount of data is called data mining. As a specific example of discovered knowledge, a characteristic rule is known. The feature rule and its discovery method are described in Patent Document 1.

  The feature rule can be written in the form of “IF A THEN B”. A (called a condition part) is a combination of one or more conditions, and B (called a conclusion part) is a single condition. Here, “condition” is a set of data attributes and their values, and is described as “age = 20s”, for example. A feature rule is a trade-off between generality (an index that indicates how much data a feature rule can be applied to) and accuracy (an index that indicates how accurately the conditional part of a feature rule represents a conclusion). An evaluation scale calculated considering off is defined. This evaluation scale is calculated as a function of P (A) and P (B | A) in the feature rule “IF A THEN B”. Here, P (A) and P (B | A) are ratios of data satisfying the condition A among all data, and P (B | A) is a further condition among the data satisfying the condition A. It is the ratio of data that also satisfies B. According to the feature rule generation algorithm, rules are evaluated using this evaluation scale, and the higher rankings sorted in descending order of the values are output as results.

  Also, data mining algorithms may require enormous amounts of computation. For this reason, a method for executing the algorithm in a distributed environment has been devised.

  A technique for discovering feature rules using a plurality of analyzers is described in Patent Document 2. In this technique, for a given problem, a counter that counts a value necessary for calculating an evaluation scale of all possible feature rules is divided and held by a plurality of data analysis apparatuses. Then, one data storage device transmits data for each record to all the analyzers at the same time, and all the analyzers receive them in order to calculate the values of P (A) and P (B | A). The counter that satisfies the condition for the data of one record is counted up. When all the analyzers have completed this process, the data storage device transmits the next data to the analyzer. By repeating such processing, data is processed for all records, and thereafter, the evaluation values of all the feature rules are calculated, and the feature rule having the higher evaluation scale is output as a result.

  Further, Non-Patent Document 1 describes a technique for solving a data analysis problem using a plurality of analyzers. According to SETI @ home described in this document, the signal data obtained from the radio telescope is divided into fixed-length work units (WUs) and sent to client programs executed on many computers via the Internet. Distributed. The client program returns the calculation result to the server and obtains the next WU. Clients have no relationship with each other.

JP-A-8-77010

Japanese Patent Laid-Open No. 2001-167098 "SETI @ home: An Experiment in Public-Resource Computing" Communications of the ACM Magazine, Vol. 45, No. 11, pp.56-61, November 2002

  However, the above conventional techniques have the following problems.

  The algorithm described in Patent Document 1 is based on the premise that a rule is discovered using a single processing device. For this reason, when the number of items increases and the number of rules to be searched increases, a single device cannot obtain sufficient calculation capability, and it takes a very long time to stop the algorithm. In particular, when searching for feature rule discovery by increasing the number of conditions allowed in the condition part, the number of rules to be searched increases explosively. For this reason, the time required for the search may become enormous.

  Further, in the technique described in Patent Document 2, all processing devices do not proceed to processing of the next record until the processing of the analysis target data 1 record is completed. For this reason, the processing speed of the entire system that performs analysis does not exceed the processing speed of the slowest analyzing apparatus.

  In the technique described in Non-Patent Document 1, no means is provided for dividing the entire problem to be analyzed into more than the number of computers serving as clients. For this reason, in the case where the job is divided into a smaller number of jobs than the number of clients participating in the analysis, at least one problem cannot be assigned to one machine, and the use of computer resources may be wasted. is there.

  Accordingly, an object of the present invention is to improve the overall processing efficiency in data analysis processing executed by a plurality of data processing devices.

The present invention
In a data analysis method for performing data analysis of an analysis problem using a first data processing device and a plurality of second data processing devices connected via a communication path,
The first data processing device divides the analysis problem into a plurality of partial problems larger than the number of the second data processing devices, and assigns the partial problem to each of the plurality of second data processing devices one by one. Processing,
Until all of the plurality of partial problems have been assigned to any of the second data processing devices, the first data processing device is one of the plurality of second data processing devices. A process of allocating an unassigned partial problem among the plurality of partial problems to the second data processing device each time processing of the partial problem assigned to
A process in which the first data processing device receives a processing result of a partial problem assigned to the second data processing device from each of the second data processing devices;
A data analysis method characterized by comprising:

The present invention also provides:
In a data analysis method for performing data analysis of an analysis problem using a first data processing device and a plurality of second data processing devices connected via a communication path,
Each time the second data processing device is assigned the partial problem of the analysis problem from the first data processing device until the assignment of the partial problem of the analysis problem from the first data processing device is stopped, Processing the subproblem and storing the result of the processing;
A process in which the second data processing apparatus in which assignment of the partial problem of the analysis problem is stopped transmits the stored processing result to the first data processing apparatus;
A data analysis method characterized by comprising:

Furthermore, the present invention provides
In a data analysis method for analyzing an analysis problem using a first data processing device and a plurality of second data processing devices connected via a communication path,
Each of the second data processing devices has input means for receiving an interruption instruction,
The data analysis method is
Each of the second data processing devices processes the partial problem each time the partial problem of the analysis problem is assigned from the first data processing device until the input means receives an interruption instruction, and Processing to save the results;
A process in which the second data processing apparatus that has received the interruption instruction by the input means transmits the stored processing result to the first data processing apparatus;
A data analysis method characterized by comprising:

  According to the present invention, the analysis problem is divided into more jobs than the number of data processing devices participating in the analysis, the job processed by each data processing device is transmitted, and the data processing device that has finished processing one job. By giving the next job without waiting for the end of the job of another data processing device, the waiting time due to the difference in processing capacity of each data processing device is suppressed, and the computer resources of the information processing system are used efficiently Can do.

  The first embodiment according to the present invention will be described below.

  FIG. 1 shows the configuration of a data analysis system according to the first embodiment of the present invention.

  In this data analysis system, a first data processing device (master) 0101, a data storage device 0108, and one or more second data processing devices (slave) 0109 are connected via a bus-type communication path 0110. Here, the master 0101 is a management device that manages analysis, and the slave is an analysis device 0109 that actually performs analysis.

  The master 0101 includes a processing device 0102, an input / output device 0105, a storage device 0107, a communication path 0111 for connecting these devices to each other, and the like. The processing device 0102 includes a memory 0103 for temporarily storing data necessary for processing executed by the processing device 0102. In the memory 0103, a program 0104 in which processing executed by the processing device 0102 is described is stored in the storage device 0107. Loaded. The input / output device 0105 includes an input device such as a keyboard and a display, input / output devices including an output device, and a communication device 0106 that manages communication with the outside of the device. The master 0101 is connected to the communication path 0110 via the communication device 0106. The storage device 0107 is a device such as a hard disk device having a capacity for storing a large amount of data. In the present embodiment, the analysis execution device 0109 and the data storage device 0108 also include similar processing devices, storage devices, input / output devices, and the like. The program stored in the storage device 0107 may be installed from a storage medium such as a CD-ROM, or may be installed via the communication line 0110.

  In this embodiment, a feature rule generation algorithm is used for knowledge discovery. Details of the feature rule generation will be described below.

  First, analysis target data and feature rules output as analysis results will be described.

  An example of analysis target data used in this embodiment is shown in FIG.

  The analysis target data is a set of records composed of a plurality of attribute items (hereinafter sometimes simply referred to as items). All records have the same attribute items. Each attribute item stores a value representing a certain attribute of each record. What the records and attribute items specifically represent depends on the data to be analyzed. For example, when the purchase history data of a certain store is to be analyzed, one record corresponds to one customer, and each attribute item includes the customer ID, sex, age, purchase frequency, total purchase amount, etc. of the customer Corresponds to the attribute of.

  In feature rule generation, as pre-processing, attribute values of each attribute item may be converted into a small number of categories. This process is called categorization. For example, the attribute item “age” can normally have an attribute value of about 10 to 100, but this is converted into a plurality of categories such as “25 years old or younger”, “26 to 35 years old”, “36 years old or older”, and the like. . In this case, the attribute item of age is converted into three categories. As in the case of gender, an attribute value that originally takes only binary attribute values of “male” and “female” may be used as a category as it is. Specifically, the sex is an attribute item having two categories “male” and “female”. The attribute items of the analysis target data shown in FIG. 2 have been subjected to such categorization processing.

  The feature rule can be written as follows, for example.

  “IF age = 26 to 35 AND purchase frequency = high THEN purchase total = high”.

  That is, the feature rule is expressed as an IF to THEN rule having a clause combining the attribute item and its category in a condition part below IF and a conclusion part below THEN. An attribute appearing in the condition part is called a condition item, an attribute appearing in the conclusion part is called a conclusion item, and a combination of the condition item appearing in the condition part of the feature rule and its category is called a conditional clause.

  The feature rule has an evaluation value that serves as an index for quantitatively evaluating how well the rule represents the feature of the target data. When the characteristic rule is generally expressed as “IF A THEN B”, the evaluation value is defined by the following expression.

P (A) ^ a * log (P (B | A) / P (B))
Here, P (A) and P (B) are ratios of records satisfying the condition A and the condition B, respectively, and P (B | A) is a record of the records satisfying the condition A. Of these, the ratio is the ratio of records satisfying both condition A and condition B. The index a is generality (an index indicating how much data a rule can be applied to) and accuracy (an index indicating how accurately the condition part of a rule represents a conclusion part) in the rule evaluation. It is a positive constant that determines which of these should be emphasized. This evaluation value may be specified by an analyst or may be determined by the following equation.

P (A) ^ a * P (B | A) * log (P (B | A) / P (B))
Regardless of which value is used for the evaluation value, the number of records that satisfy the condition part that appears in a certain characteristic rule, the number of records that satisfy the conclusion part, the number of records that satisfy the condition part and the conclusion part, and the entire analysis target data The evaluation value can be calculated by knowing the number of records. In the feature rule generation, the feature rules are evaluated based on the evaluation value, sorted in descending order from the feature rule having the highest evaluation value, and the higher order is output as a result. An example of the sorted feature rules is shown in FIG.

  The feature rule generation is an algorithm for finding a feature rule having a large evaluation value based on the evaluation value defined above. Before performing this process, the analyst should limit the number of feature rules to be discovered, the attribute item that is the conclusion item, its category, and the multiple attributes that are candidates for the attribute item included in the condition part. It is assumed that an upper limit (hereinafter referred to as the maximum number of conditional clauses) of items (hereinafter referred to as condition items) and the number of condition items included in one feature rule is given.

  Now, condition items A (1) to A (M) (M is the number of condition items) are given, and each condition item is B (A (m), 1) to B (A (m), N ( (N (A (m)) is the number of categories of the condition item A (m), and m is an integer of 1 ≦ m ≦ M). If the maximum number of conditional clauses is C_max, the feature rule “IF X THEN Y” can be described as follows.

  The condition part X can be expressed as a logical product “C (1) and... And C (k)” of the conditional clause C (i). Here, i is an integer of 1 ≦ i ≦ k, and k is an integer of 1 ≦ k ≦ C_max. Conditional clause C (i) is a combination of condition item A (m) and category B (A (m), n) of A (m). In the rule, “A (m) = B (A (m ), n) ".

  Here, a feature rule including a plurality of conditional clauses has the following properties. In a conditional part C (1) and... and C (k) (2 ≦ k ≦ C_max) composed of k conditional clauses, arbitrary two conditional clauses are represented by C (i), C (j) (C (i ) Is A (m) = B (A (m), n), and C (j) is A (P) = B (A (P), q)). P. That is, in one rule, the same condition item does not appear, and the condition clauses are arranged in ascending order by the subscript of the condition item.

  There are many feature rules that can be considered from the above description, but their relationship can be represented by a tree structure as shown in FIGS. In feature rule generation, a rule is found by scanning (searching) this tree (hereinafter referred to as a search tree).

  The order relation between rules in the search tree is defined as follows.

  It is assumed that the order relationship of the condition items and the order relationship between the categories of the condition items are determined in advance. For example, when an appropriate index number is given to a condition item and an index number is given to an included category in each condition item, the order relationship can be determined by the size of the index number. As for the order relationship between the conditional clauses and the categories, the order with the smaller index number is first.

  Based on the order relation of the condition items and the order relation of the categories, the order relation between the conditional clause and the condition part is determined. The order relation of the conditional clauses is first determined by the order relation of the condition items. When the condition items are the same, the order relation of the categories is determined. The order relation of the condition parts is determined by arranging the condition clauses included in the condition part in the order of the order relation, and comparing these order relations from the first condition clause.

  A specific example of the search tree is shown in FIG. In this search tree, all nodes except the root node 0401 correspond to one feature rule. If the number of branches from the root node 0401 to a certain node is called depth, and a set of nodes at the same depth is called a layer, a set 0404 of depth 1 nodes surrounded by a broken line is one. One layer is formed.

  The layer 0404 includes all possible combinations of condition items and their categories as nodes, and each node corresponds to a rule having one conditional clause. Assuming that the conclusion rule is “total purchase amount = high” and a feature rule is found, for example, the node 0402 corresponds to the rule “IF sex = male THEN total purchase amount = high”. A layer 0405 having a depth of 2 positioned below the layer 0404 includes nodes corresponding to feature rules having two conditional clauses. For example, the node 0403 corresponds to a rule of “IF sex = male AND purchase frequency = high THEN purchase total = high”. That is, the depth of the node corresponds to the number of conditional clauses in the rule.

  Here, of the two nodes connected by branches, the upper node is called a parent node, the lower node is called a child node, and the rules corresponding to each node are called a parent rule and a child rule. . In the search tree, when the child rule has a condition part “C (1) and... And C (k−1) and C (k)”, the condition part of the parent rule is the last from the condition part of the child rule. The conditional part “C (1) and... And C (k−1)” is removed from the conditional clause C (k) appearing in FIG.

  As another example of the search tree, FIG. 5 shows an outline of the entire search tree. FIG. 5 is a conceptual diagram of a search tree in the case where the search is performed with the maximum number of conditional clauses being 3, in which the five condition items A to E are each divided into five categories.

  Considering the characteristics of the feature rule according to the present embodiment, the search tree for generating the feature rule is selected when N or less items (M is the maximum number of conditional clauses) are selected from N items (N is the number of conditional items). Considering that the combination is written and each condition item is categorized, it can be said that the possible combinations of the condition item and the category value are written without duplication.

  As procedures for evaluating all nodes in the search tree, algorithms such as a breadth-first search and a depth-first search can be considered. The breadth-first search is a search method in which a node in the same layer is evaluated first from a layer having a depth of 1 in the search tree, and when all nodes in the layer have been evaluated, the process moves to the next layer. The depth-first search is a search method that pays attention to the parent-child relationship of nodes and circulates the nodes by giving priority to the child nodes over the nodes in the same layer.

Next, search tree division processing according to the present embodiment will be described. An example of a method for dividing the search tree into partial trees will be described with reference to FIG. The search tree shown in FIG. 6 is the same as that in FIG. 5. In FIG. 6, the node group 0601 surrounded by a broken line represents all the nodes included in the layer of depth 1. When nodes having the same condition items are grouped into one group and nodes included in the node group 0601 are divided into smaller node groups, node groups 0602 to 0604 are obtained. Similarly, the node group included in the layer of depth 2 is also divided into node groups so that the child nodes of the nodes that are in the same group in the layer of depth 1 become one group. Hereinafter, a node group included in a layer having a depth of 2 or more is similarly divided. Each node group divided in this way is called a job for searching a search tree.

  According to such a dividing method, since jobs are separated independently based on each layer, feature rule generation can be performed by applying a breadth-first search algorithm.

  Another example of the division method will be described with reference to FIG. The tree shown in FIG. 7 is a search tree in feature rule generation, and is the same as FIG.

  Focus on layer 0701, which is a set of feature rules having one condition. In this layer 0701, a node corresponding to a rule having the same condition item in the condition part and a set of subtrees having these nodes as root nodes are regarded as one job, and the entire search tree is divided. That is, in FIG. 7, each of the partial trees 0702 to 0704 represents one job. According to such a dividing method, it is possible to generate a feature rule by applying a depth-first search algorithm to each subtree.

  In the present embodiment, it is considered that the search tree dividing method shown in FIG. 6 is adopted, and a feature rule is generated by processing a job divided by this dividing method with a plurality of slaves.

  A breadth-first search is used as a search algorithm for each job. When feature rule generation is performed by breadth-first search, nodes are evaluated in order from a layer having a depth of 1. When searching for a feature rule in each layer, it is necessary to refer to the rule generation result up to the previous layer (layer having a depth of 1 less) and to search for a feature rule while eliminating redundant rules. . Here, the redundancy rule is the following rule. That is, in the rule set R including the rules R_a and R_b, when the condition part R_a is composed of some condition clauses included in the condition part R_b and the evaluation value of R_a is equal to or larger than R_b, the rule set R Among these, R_b is a redundancy rule.

  That is, in the breadth-first search, the search for the layer n + 1 cannot proceed until the rule search for the layer n (where 1 ≦ n ≦ C_max−1) is completed. For this reason, in the rule generation procedure according to this embodiment, processing for waiting is performed, and processing of jobs included in layer n + 1 is not started until the master receives all analysis results of jobs in layer n from the slave. It is like that.

  Details of feature rule generation according to the present embodiment will be described with reference to FIGS.

  FIG. 8 is a detailed flowchart of a process for the master to manage the analysis execution device (slave). The process of managing the slave by the master differs between when the rule is being generated and when the rule is not being generated. FIG. 8 is a flowchart of processing when the analysis of feature rule generation is not performed.

  An example of the slave management table is shown in FIG.

  This slave management table is used to manage information on slaves that perform analysis by communicating with the master, and is held in the memory 0103. In this slave management table, at least ID 0901 for uniquely identifying the slave and information 0902 necessary for communicating with the slave using the communication path are stored for each slave. This information 0902 is, for example, the IP address and port number of the slave when communication is performed using TCP / IP.

  In FIG. 8, the master 0101 executes a parameter input reception process in parallel with executing the processes S0801 to S0804 for managing the slave 0109. Processes S0801 to S0804 and the parameter input reception process are processed in parallel using a mechanism that simultaneously executes a plurality of tasks in the processing apparatus, such as multithreading.

  In the parameter input acceptance process, the analyst designates the file name of the analysis target data stored in the data storage device 0108 by the input / output device 0105 and reads it into the memory 0103. Here, the analysis target data is data for which rule generation is performed using feature rule generation, and is stored in the data storage device 0108 in the form of a table format file, data in a database, or the like, for example. . Here, the analysis target data is stored in the data storage device 0108. However, the analysis target data is stored in the storage device 0107 in the master so that the analysis target data is read from the storage device 0107 into the memory 0103. Also good.

  In storing the analysis target data in the memory 0103, each attribute item is assigned an index number for uniquely identifying the item. For example, serial numbers are assigned to the attribute items in the order in which the attribute items appear in the analysis target data, and this value can be used as the index number.

  The analyst further inputs information related to the categorization of the analysis target data, condition items and conclusion part used for rule generation, rule generation parameters, and assumed maximum value of the number of slaves participating in the analysis, using the input / output device 0105. Is stored in the memory 0103.

  The information regarding the categorization of the analysis target data is information for categorizing each attribute item included in the analysis target data. This information includes the number of categories of each attribute item, the boundary value for converting the value of the attribute item into a category, and the number of records of data included in each category. In each attribute item, an index number is assigned to each category of the item.

  The information regarding the condition item and the conclusion part includes information on the condition item and the conclusion item used in rule generation, and the category of the conclusion part used as the conclusion part. These are managed in the master 0101 with index numbers assigned to attribute items and categories.

  A condition item and a conclusion part used for rule generation are a combination of an item that is a candidate for an attribute item that appears in the condition part, and a combination of an attribute item that is a conclusion part and a category.

  The rule generation parameters include the number R_max of feature rules to be discovered and the maximum number of conditional clauses C_max.

  As the assumed maximum value of the number of slaves, a value larger than the number of slaves used for analysis is designated.

  The parameter input acceptance process ends after the input of these pieces of information.

  In S0801, the master 0101 checks whether a signal from the slave 0109 is not received. As a result, if it is determined that a slave addition signal has been received from the slave, the master 0101 executes a slave addition process S0802. If it is determined that a slave deletion signal has been received from the slave, the master 0101 executes slave deletion processing S0803. On the other hand, if it is determined that no signal has been received, the master 0101 executes a branch process S0804.

  In the slave addition process S0802, the master 0101 assigns an ID number for identifying the slave to the source slave of the signal received in S0801, acquires the IP address of the slave and the port number used by the slave for communication, Is stored in the slave management table. The ID number is assigned to the slave by, for example, assigning a serial number to the signal source slave in the order of signal reception in the master. Thereafter, the master 0101 returns the process to S0801.

  In the slave deletion process 0802, the master 0101 deletes the source slave of the interruption request signal received in S801 from the management table.

  On the other hand, in S0804, the master 0101 checks the input content from the input / output device 0105, and checks whether there is an input of a rule generation execution start signal. As a result, if there is an input of a rule generation execution start signal, the master 0101 executes the analysis execution start process S0805; otherwise, the master 0101 returns the process to S0801.

In the analysis execution start process S0805, the master 0101 prepares for rule generation ((1) preparation for transmitting analysis target data to the slave, (2) preparation for storing analysis result rules, and (3) search tree. Dividing into jobs, (4) Setting initial conditions for analysis).
(1) Preparation for transmitting analysis target data to slave The master 0101 categorizes the data of each attribute item corresponding to the condition item and the conclusion part item of the analysis target data stored in the memory 0103.
(2) Preparation for storing analysis result rules The master 0101 reserves a sufficient area in the memory 0103 to store analysis result feature rules. In this area, the upper R_max rules received from the slave at a certain point during the analysis are stored as intermediate results of the analysis.
(3) Division of Search Tree into Jobs Master 0101 divides the search tree into jobs as follows according to the method shown in FIG.

  The master 0101 assigns the index numbers of the condition item A (1), the condition item A (2), etc. to the condition items of the analysis target data in the order stored in the condition item memory 0103, thereby defining the order relationship. For each category of condition items, an index number is assigned in the same manner to define an order relationship. Thereby, the order relationship is defined for the condition item and the category, and the order relationship of the condition part can be defined.

  Information regarding each divided job is managed by a job management table in the master 0101. An example of the job management table is shown in FIG. As shown in FIG. 10, the job management table includes at least a job ID 1001 for uniquely identifying a job, a layer number 1002, a job analysis start node 1003, an analysis end node 1004, and an analysis of the job for each job. A slave 1005 that stores the ID of the slave that handles the job, and a job status flag 1006 that holds information relating to the analysis status (for example, analysis in progress, analysis completed, etc.) are included. This job management table is stored in the memory 0103.

  The master 0101 can uniquely determine the node included in each job by specifying the analysis start node and the analysis end node of the job based on the above-described order relationship. For example, by specifying the condition part “A = a (1)” as the analysis start node of job 0602 and “A = a (5)” as the analysis end node of job 0602, the node to be analyzed in job 0602 is uniquely identified. Can be determined. In the master 0101 and the slave 0109, the node is expressed as a set of a condition item and a category index number. For example, if the index number of the condition item A is 1, the index number of the category a (1) is 1, and the index number of the category a (5) is 5, the start node and the end node of the job 0601 are (1 , 1), (1, 5). When there are a plurality of conditional clauses, one node is represented by a set of a plurality of condition items and category index numbers. In the following, when a node is operated in a computer, it is assumed that such an index number is actually operated in the computer.

  Based on this, the search tree is divided into jobs as follows.

  The master 0101 repeatedly executes the following processing with the initial value of the variable I representing the depth of the layer = 1 and the initial value of the variable m representing the number of the condition item = 1.

  The conditional clause X_s corresponding to the analysis start node of a job is written as C_s (1) and C_s (I), and the conditional clause X_e corresponding to the analysis end node is written as C_e (1) and… and C_e (I). To do. Here, C_s (1) is (m, 1), C_s (i) (2 ≦ i ≦ I) is (m + i−1,1), and C_e (1) is (m, N (A (m)) )), C_e (i) (2 ≦ i ≦ I) is (M−I + i, N (A (M−I + i))), and M is the number of condition items. By this method, the master 0101 determines a set of condition items and categories of the analysis start node and the analysis end node, and includes the analysis start node 1003, the analysis end node 1004, a job ID 1001 for uniquely identifying the job, and the job. The layer 1002 to be stored is stored in the job management table. Here, the job ID is given to the job, for example, by giving a serial number in the registration order of the job. Further, the slave 1005 in charge of this job is set to “unassigned”.

  The master 0101 repeats this process while increasing the variable m by 1 until m becomes equal to M-I + 1. When m becomes equal to M−I + 1, the master 0101 returns m to the initial value “1”, increases I by 1, and repeats the same processing until I becomes equal to the maximum condition clause number C_max.

  When the number of jobs included in each layer is smaller than the assumed maximum value of the number of slaves participating in the analysis, the master 0101 repeats the process of dividing the job including the most rules into two layers. The number of jobs included in the is made larger than the assumed maximum value of the number of slaves. This process is performed in the same manner for all layers. Here, the process of dividing the job into two is realized as follows.

  When dividing the job J into the job J (1) and the job J (2), the master 0101 calculates the number of rules included in the job from the category division number of each condition item, and divides this by two. Note that the category division number information is stored in the memory 0103. After that, the master 0101 rounds down the quotient obtained by dividing the number of rules by 2, and sets the obtained integer as N. The master 0101 sets the start node of J as the start node of J (1), counts the nodes from the start node of J (1) according to the order of the nodes, and sets the Nth node as the end node of J (1). To do.

  In addition, the master 0101 sets the next node after the end node of J (1) as the start node of J (2), sets the end node of J as the end node of J (2), and sets J (1) and J (2). Store information in the job management table. As a result, the information on job J in the job management table is overwritten with the information on J (1), and the information on J (2) is added to the management table.

As described above, as long as the number of nodes included in one layer does not fall below the assumed maximum value of the slave, the rule search of one layer can be divided into more jobs than the slave.
(4) Process for Setting Initial Conditions for Analysis The master 0101 prepares a parameter for storing the assignable layer number, and sets the layer number 1 as the assignable layer number. In addition, the master 0101 prepares a parameter for storing the minimum evaluation value of the adopted rule, and initializes the parameter. Although the value used for parameter initialization varies depending on the rule evaluation method, when the evaluation value according to the present embodiment is used, initialization is performed by setting “0” to the parameter. This parameter is used when evaluating rules by the slave. Specifically, when it is found that the evaluation value of the rule is lower than the parameter value, the subsequent evaluation processing for the rule is not performed. Thereby, processing can be reduced. Note that an area for storing these parameters is secured in the memory 0103.

  After these preparation processes (1) to (4) are executed, in analysis execution start process S0805, master 0101 starts the slave management process shown in FIG. 11 and the analysis process shown in FIG. These processes are executed in parallel with the process in FIG. This process is realized by using a mechanism that simultaneously executes a plurality of tasks in the processing apparatus, such as a multi-thread. Note that the slave management process shown in FIG. 11 is a process for managing the slave during the execution of the analysis.

  When the analysis execution start process S0805 ends, the process proceeds to the analysis execution notification process S0806.

  In the analysis execution notification process S0806, the master 0101 transmits an analysis execution notification signal via the communication path 0110 to all the slaves registered in the slave management table. After the analysis execution notification process is completed, the process in this procedure is terminated.

  FIG. 11 shows a detailed flowchart of the slave management process executed during the analysis execution start process S0805.

  In branch processing S1101, the master 0101 checks a signal related to slave management from the slave. As a result, the master 0101 executes slave addition processing 1102 when the signal is a slave addition signal, and executes slave deletion processing 1103 when the signal is a slave deletion signal. On the other hand, when those signals are not received from the slave 0109, the master 0101 executes a branch process S1104.

  In the slave addition process S1102, the master 0101 executes a process similar to the slave addition process 0802 of FIG. 8 and then executes an analysis execution notification process S1105. In the analysis execution notification process S1105, the master 0101 transmits an analysis execution notification signal to the slave added in the slave addition process S1102, and then returns the process to S1101.

  In the slave deletion process 1103, the master 0101 performs the same process as the slave deletion process 0803 of FIG. 8, and returns the process to S1101.

  In the branch process S1104, the master 0101 checks whether or not the analysis process of FIG. 12 has ended. If it has ended, the process in the procedure of FIG. 11 ends, and if not, the process returns to S1101. .

  FIG. 12 shows a detailed flowchart of the analysis process executed during the analysis execution start process S0805.

  In branch processing S1201, the master 0101 checks the content of the signal received from the slave under analysis, and branches the processing according to the type. In this embodiment, the type of signal received from the slave is one of a data parameter request signal, a midway progress request signal, a job request signal, a job analysis completion notification signal, and a result transmission signal.

  When the communication content from the slave is a data parameter request signal, the master 0101 transmits the analysis target data and the analysis parameter to the transmission source slave of the signal in the data parameter transmission processing 1202, and then performs the processing in S1201. return. Here, the analysis target data is converted into an index column of category values of data obtained by categorization of the analysis target data and transmitted. In addition, the analysis parameter transmitted in this process includes the upper limit R_max of the number of feature rules to be found, the index number of the attribute item as the conclusion item and the index number of the category, the index number of the condition item, the maximum number of conditional clauses C_max It is included.

  When the communication content from the slave is a halfway progress request signal, the master 0101 executes a halfway progress transmission process S1203. In step S1203, the master 0101 transmits the analysis result rule group stored in the memory 0103 to the slave 0109, and then returns the process to S1201.

  When the signal from the slave 0109 is a job request signal, the master 0101 executes a branch process S1204. In branch processing S1204, the master 0101 checks whether there is an assignable job. Specifically, the master 0101 refers to the job management table and checks whether there is a job whose item of the slave 1005 is “unassigned” among jobs having the layer number 1002 equal to the assignable layer number. To do. As a result, if such a job exists, the master 0101 determines that there is an assignable job, and executes job assignment processing S1205. On the other hand, if such a job does not exist, the master 0101 determines that there is no job that can be assigned, and executes job assignment end processing S1206 in the current layer.

  In the job assignment process S1205, the master 0101 assigns one of the jobs whose assigned slave item is “unassigned” to the slave that issued the job request signal, and sets the job status flag 1006 of the corresponding job in the job management table. The state of “under analysis” is set, and the ID of this slave is stored in the item of the slave 1005 in charge. Thereafter, the process returns to S1201. In this assignment processing, the values of the analysis start node 1003 and the analysis end node 1004 stored in the job management table and the information of the minimum evaluation value stored in the master 0101 are transmitted from the master 0101 to the slave 0109.

  In job assignment end processing S1206 in the current layer, the master 0101 transmits a signal requesting transmission of the analysis result of the current layer to the slave 0109, and then returns the processing to S1201.

  When the signal from the slave 0109 is a job analysis completion notification, the master 0101 performs parameter update processing S1207. This signal includes the minimum evaluation value of the feature rule held by the slave. In the parameter update processing S1207, the master 0101 updates the job status flag 1006 of the job that the slave is analyzing in the job management table to “analysis completed”. Also, the master 0101 compares the parameter value relating to the minimum evaluation value received from the slave with the value of the minimum evaluation value being stored in the master, and the larger value of the two values is the minimum evaluation value being stored in the master. Update the value. Thereafter, the process returns to S1201.

  When the signal from the slave is a result transmission, the master 0101 executes a midway result merge process S1208. In the intermediate result merge process S1208, the master 0101 receives the result of the analysis performed by the slave from the slave 0109, and merges the analysis result with the analysis result stored in the memory 0103. Specifically, the master 0101 refers to the feature rule stored in the memory and the received feature rule from the slave 0109, and selects R_max rules having higher evaluation values from these rules. Update stored result rules.

  Thereafter, the master 0101 refers to the item of the responsible slave 1005 in the job management table, and the job status of the job management table for the job whose slave ID stored in this item is the same as the ID of the slave that transmitted the result. The flag 1005 is set to “accept result”.

  After that, in branch processing S1209, the master 0101 refers to the job management table, and for the job having the layer number 1002 that is currently being analyzed, whether or not the job status flags 1006 of all jobs are values indicating acceptance of the results. Check. As a result, if the result is accepted, the master 0101 executes the branch process S1210. Otherwise, the master 0101 returns the process to S1201.

  In S1210, the master 0101 refers to the value of the currently analyzed layer number and the maximum number of conditional clauses. If the currently analyzed layer number is smaller than the maximum number of conditional clauses, the master 0101 executes the analysis continuation notification processing S1211. If so, the analysis end notification process S1212 is executed.

  In the analysis continuation notification process S1211, the master 0101 increments the assignable layer number by 1 and transmits an analysis continuation notification to all the slaves registered in the slave management table, and then returns the process to S1201.

  On the other hand, in the analysis end notification process 1212, the master 0101 transmits an analysis end signal to all slaves registered in the slave management table, and then executes S1213. In result output processing S1213, the master 0101 outputs a set of feature rules currently held to the input / output device as a final result. Thus, the rule generation process is terminated.

  Next, details of the slave processing according to the present embodiment will be described with reference to FIG.

  The slave 0109 transmits a slave addition signal to the master 0101 in the analysis execution registration process S1301, and waits for an analysis execution notification signal from the master 0101 in the analysis execution initialization process S1302. Upon receiving the analysis execution notification signal, the slave 0109 transmits a data / parameter transmission request to the master 0101 and receives analysis target data and parameters used for rule generation from the master 0101. Further, the slave 0109 secures an area for storing the rule group of the analysis result and an area for storing the rule group of the intermediate result in the memory, and initializes these areas to be empty. As these areas, a sufficient memory area for storing R_max rules is secured.

  Thereafter, in the intermediate result request process S1303, the slave 0109 transmits the intermediate result request signal to the master 0101, thereby obtaining the intermediate result including the set of feature rules as the rule generation result held by the master 0101. Receive from 0101. The slave 0109 stores the intermediate result received from the master 0101 in a memory for storing the rule group of the intermediate result secured in S1302.

  Thereafter, the slave 0109 transmits a job request signal to the master 0101 in the job request process S1304, and in response to the response signal to the job request transmitted in S1304 in the branch process S1305, the response signal is assigned to a new job. It is checked whether it is a signal or an analysis result transmission request signal. As a result, if it is a signal for assigning a new job, the slave 0109 executes analysis execution processing S1306, and if it is a signal for requesting transmission of the analysis result, the slave 0109 executes intermediate result transmission processing S1307.

  In the analysis execution process 1306, the slave 0109 holds the minimum evaluation value received from the master 0101 in the memory, refers to the analysis start node and analysis end node of the assigned job, and relates to the present embodiment as follows. Evaluate nodes according to node ordering.

  The slave 0109 sets the analysis start node as the current node, and calculates an evaluation value of a rule corresponding to the current node (hereinafter referred to as a current rule). Thereafter, the slave 0109 determines whether to adopt this rule. Specifically, the slave 0109 has (1) that the evaluation value of the current rule is larger than the minimum evaluation value of the rule in this slave, and (2) the current rule is redundant in the rule group of the intermediate result received in S1303. If neither of the two conditions is true, this rule is adopted.

  When the current rule is adopted, the slave 0109 compares the current rule with the rule stored in the analysis result storage area secured in S1302 in order from the rule with the smallest evaluation value, and evaluates the comparison target rule. When the value falls below the evaluation value of the current rule, the current rule is inserted above the comparison target rule. As a result, when the number of rules stored in the analysis result storage area exceeds R_max, the slave 0109 deletes the rule with the lowest evaluation value and then stores it again in the analysis result storage area. The evaluation value of the lowest rule is referred to, and the parameter of the minimum evaluation value is updated with this value. After that, the slave 0109 checks whether or not the current node is equal to the analysis end node. If they are equal, the node evaluation ends, and if not, the next node of the current current node is set as the current node. The node evaluation and the adoption determination are executed again.

  Thereafter, in job analysis result transmission processing S1308, the slave 0109 transmits the evaluation value of the rule with the lowest evaluation value among the stored characteristic rules of the analysis result to the master 0101, and then returns the processing to S1304. However, if the number of rules stored in the area holding the rule group of the analysis result is less than R_max, the slave 0109 transmits “0” as the evaluation value to the master 0101 and performs the process in S1304. return.

  On the other hand, in the intermediate result transmission process S1307, the slave 0109 transmits information on the feature rule of the retained analysis result to the master 0101. Here, the feature rule information transmitted to the master 0101 includes the combination of the condition item and category included in the condition part of the analysis result feature rule, and the evaluation value of each feature rule. Thereafter, in branch processing S1309, the slave 0100 waits until a signal related to the continuation of analysis is received from the master 0101. Here, if an analysis continuation notice is received from the master 0101, the slave 0109 returns the process to S1303, and if an analysis end notice is received from the master 0101, the slave 0109 executes an analysis end process S1310.

  In the analysis end process S1310, the slave 0109 executes a process necessary for the end of the analysis, such as releasing the memory used in the analysis, and then ends the process. Note that the slave 0109 may return the process to S1302 without terminating the process and wait for the start of the next rule generation problem.

  The slave processing according to the present embodiment can be changed so that the slave 0109 receives an analysis interruption instruction from the input / output device 0105. The presence / absence of the analysis interruption instruction may be checked before the slave 0109 requests the master 0101 for a job in S1304, for example. When there is an instruction to interrupt the analysis, the slave 0109 performs the intermediate result transmission process S1307 and the analysis end process S1309, and transmits the slave deletion signal to the master 0101, and then the process ends. .

  As described above, according to the present embodiment, the analysis problem is divided into more than the number of jobs of the analysis device, and the analysis device that has finished processing one job does not wait for the end of the job of the other analysis device. Is given the following job. For this reason, the waiting time due to the difference in the processing capabilities of the processing devices participating in the analysis is suppressed, and the computer resources of the processing devices can be used efficiently.

  Hereinafter, a second embodiment according to the present invention will be described.

  In the present embodiment, slave management is performed by the same processing as in the first embodiment described above. The details are the same as those described in FIG. 8 and FIG. 11 except for a part. In this embodiment, instead of performing the processing described in the first embodiment in the analysis execution start processing S0805. The following processing is executed.

  In the analysis execution start process S0805, as preparations for generating a rule, preparations for transmitting analysis target data to a slave, preparations for storing analysis result rules, division of search trees into subtrees, and setting of initial conditions for analysis I do. Processing related to preparation for transmission to the slave and preparation for storing the rule of the analysis result is the same as the processing described in the first embodiment.

  Unlike the first embodiment, the search tree is divided into subtrees as shown in FIG. 7, unlike the first embodiment. Prior to explaining the details of the search tree dividing method, FIG. 14 shows the data structure of a table for managing jobs in the master 0101 in this embodiment. In this job management table, for each partial problem, at least a job ID (partial problem ID) 1401 for uniquely identifying a job, a slave 1403 for storing an ID of a slave in charge of analyzing the job, and each job are stored. A condition item 1402 for storing a condition item in charge of analysis, and a job status flag 1404 for holding information on the state of analysis (for example, “analyzing” or “analysis completed”) are included.

  Details of the search tree division method are as follows.

  In the division of the search tree into subtrees, the condition item index number is stored in order from 1 to the condition item number in the assigned condition item 1402 of the job management table, and the assigned slave 1403 is set to “unassigned”. In the present embodiment, with a condition item in a layer having a depth of 1 as a reference, a subtree having these nodes as a root is handled as one job. For this reason, if information specifying the condition item is stored in the job management table, the subtree corresponding to the job can be uniquely represented.

  In the analysis execution start process S0805, the master 0101 also starts the slave management process shown in FIG. 11 and the analysis process shown in FIG. After completing the analysis execution start process S0805, the master 0101 executes the analysis execution notification process S0806.

  Details of processing of the master 0101 according to the present embodiment will be described with reference to FIG.

  In the branch process S1501, the master 0101 checks the communication content from the slave being analyzed, and branches to a process corresponding to the content. Here, the communication content from the slave 0109 is any one of a data parameter request signal, a job request signal, a job analysis completion notification, and a result transmission.

  When the communication content from the slave 0109 is a data parameter request signal, the master 0101 transmits the analysis target data and the analysis parameter to the transmission source slave of the signal as the data parameter transmission processing, and the branch processing S1501 Return. Here, the data is converted into an index column of category values of data obtained by categorization of analysis target data and transmitted. The parameters sent in this process include the upper limit R_max of the number of feature rules to be found, the index number of the attribute item that is the conclusion item and its category index number, the index number of the condition item, and the maximum conditional clause number C_max. Yes.

  If the signal from the slave 0109 is a job request signal, the master 0101 refers to the job management table in the branching process 1503 to determine whether there is a job in which the assigned slave item 1403 is “unassigned”. Check if. As a result, if there is such a job, the master 0101 executes a job assignment process S1504, and if there is no such job, the master 0101 executes a job assignment end process S1505.

  In the job assignment processing S1504, the master 0101 assigns one of the jobs whose assigned slave item is “unassigned” to the job request signal transmission source slave. Specifically, the master 0101 transmits data including the assigned condition item number and the minimum evaluation value of the rule stored in the master 0101 to the slave 0109. Thereafter, the master 0101 sets the job status flag 1404 of the assigned job in the job management table to “analyzing”, stores the assigned slave ID in the item of the assigned slave 1403, and then returns to the branch processing S 1501.

  In the job assignment end process S1505, the master 0101 transmits a signal requesting transmission of the analysis result to the slave 0109. Thereafter, the master 0101 returns to the branch process S1501.

  When the signal from the slave is a job analysis completion notification, the master 0101 receives a parameter including the minimum evaluation value of the generation rule in the slave from the slave 0109 in the parameter update processing S1506, and stores the minimum evaluation value in the master. Compare with the value of the minimum evaluation value. As a result of this comparison, the smallest evaluation value of the master is updated with the evaluation value having the larger value as the new value. Then, the master 0101 refers to the items of the responsible slave 1403 and the job status flag 1404 in the job management table, the ID of the responsible slave is the same as the slave that sent the signal, and the job status flag 1404 is “under analysis”. The job status flag 1404 of the job is “analysis completed”. Thereafter, the process returns to the branch process S1501.

  When the signal from the slave 0109 is a result transmission, the master 0101 receives the analysis result of the slave from the slave 0109 in the result merge process S1507. The master 0101 merges this result with the analysis result stored in the memory 0103 by the following processing. That is, the master refers to the characteristic rule stored in the memory 0103 and the evaluation value of the characteristic rule received from the slave, selects the R_max rule having the higher evaluation value from those rules, and the master stores it. Update the result rule.

  Thereafter, the master 0101 refers to the item of the responsible slave 1403 in the job management table and sets the item of the job status flag 1404 to “result acceptance” for a job storing the same ID as the ID of the slave that has transmitted the result. And

  In branch processing S1508, the master 0101 checks whether or not the job status flags 1404 of all jobs in the job management table are “accepted”. As a result, if the job status flag 1404 of all jobs in the job management table is “accepted”, the master 0101 executes the result output process S1509; otherwise, the master 0101 performs the branch process S1501. Return to.

  In result output processing S1509, the master 0101 outputs the set of feature rules currently held by the master to the input / output device 0105 as the final result. Thereby, the analysis process is completed.

  Next, details of the slave processing according to the present embodiment will be described with reference to FIG.

  The slave 0109 sends a slave addition signal to the master 0101 in the analysis execution registration process S1601, and waits for an analysis execution notification signal from the master 0101 in the analysis execution initialization process S1602. Upon receiving this signal, the slave 0109 transmits a data / parameter transmission request signal to the master 0101, and receives analysis target data and parameters used for rule generation from the master 0101. Further, in the analysis execution initialization process S1602, the slave 0109 secures areas for storing the rule group of the analysis result in the memory, and initializes these areas to be empty. Here, it is assumed that a sufficient memory area is secured to store R_max rules.

  After that, the slave 0109 transmits a job request signal to the master 0101 in the job request process S1603. In the branch process S1604, the response signal for the job request requests transmission of a new job assignment signal and analysis result. Check which signal is to be used. As a result, if it is a signal for assigning a new job, the slave 0109 executes analysis execution processing S1605, and if it is a signal for requesting transmission of the analysis result, the slave 0109 executes intermediate result transmission processing S1606.

  In the analysis execution process S1605, the slave 0109 searches for a subtree corresponding to the assigned job, and extracts R_max rules having a high evaluation value. If the assigned item number transmitted in the job assignment process is m, this process is specifically realized by the following procedure.

  The slave 0109 sets the condition part “A (m) = B (A (m), 1)” as the current node, and calculates the evaluation value of the rule (current rule) corresponding to the current node. Then, the slave 0109 determines whether to adopt this slave. Specifically, the slave 0109 determines whether or not the condition that the evaluation value of the current rule is larger than the minimum evaluation value of the rule in this slave is satisfied, and if this is true (the evaluation value of the current rule) If this is the case, use this rule.

  When the slave 0109 adopts the current rule, the slave 0109 compares the current rule with the rule stored in the analysis result table secured in S1602 in order from the rule with the smallest evaluation value, and the evaluation value of the comparison target rule. When the current value falls below the evaluation value of the current rule, the current rule is inserted above the comparison target rule. As a result, when the number of rules stored in the analysis result table exceeds R_max, the slave 0109 deletes the rule with the lowest evaluation value, and then re-stores the maximum number of rules stored in the analysis result table. The evaluation value of the lower rule is referred to, and the parameter of the minimum evaluation value is updated with this value. Thereafter, the slave 0109 tries to generate a node that performs next evaluation and adoption determination. As a result, if the node generation is successful, the slave 0109 sets the generated rule as the current node, performs node evaluation and adoption again, and if node generation fails, the slave 0109 determines the node evaluation. Exit.

  Generation of a node next to the current node is performed as follows.

  Here, the condition part X of the current rule is expressed as C (1) and C (2) and.

  First, the slave 0109 tries to generate a child rule of the current node. When the conditional clause C (k) at the right end of the conditional clause of the current rule has the conditional item A (n), the slave 0109 has A (n + 1) = B (A (n (n)) in the conditional part of the current rule. A rule having a conditional part added with +1) and 1) is generated. However, if the number of conditional clauses in the current rule has already reached the maximum number of conditional clauses C_max, or if C (k) has conditional clause A (M) (M is the number of conditional items) as a conditional item , Child rule generation fails. When the generation of the child rule is successful, the slave 0109 sets the generated rule as a rule for performing the next evaluation. On the other hand, when the generation of the child rule fails, the slave 0109 proceeds to the next process.

  Next, the slave 0109 tries to generate the next rule under the constraint that “the next rule also has the same parent rule as the current rule”. Here, the next rule is determined based on the order relation of the condition parts described in the first embodiment. If the slave 0109 can generate the next rule according to the above-described constraint conditions, the slave 0109 sets this rule as a rule to be evaluated next. On the other hand, if the next rule that satisfies the above constraint conditions does not exist (if generation fails), the slave 0109 returns from the current rule to the parent rule (specifically, the rightmost from the condition part of the current rule). ) And try to generate the next rule in that layer with the same constraints as above. However, when trying to generate the next rule in the depth 1 layer, the generation of the next rule with the conditional clause A (m) = B (A (m), N (A (m))) fails. And

  The slave 0109 repeats the process of returning to the parent rule every time generation of the next rule in the same layer fails. When the slave 0109 returns to the root node, the slave 0109 determines that generation of the next rule to be evaluated has failed. .

  After executing such processing, the slave 0109 transmits the evaluation value of the rule with the lowest table value among the characteristic rules held by the slave 0109 to the master 0101 in the job analysis result transmission processing S1607. Thereafter, the slave 0109 returns to the job request process S1603. However, if the number of rules stored in the area holding the rule group of the intermediate result is less than R_max, the slave 0109 transmits “0” as the evaluation value to the master 0101 in S1607. The process returns to S1603.

  In the result transmission process S1606, the slave 0109 transmits information on the feature rule that is held to the master 0101. The feature rule information transmitted to the master at this time includes a set of condition items and categories included in the condition part of each feature rule, and an evaluation value of each feature rule.

  Thereafter, in the analysis end process S1608, the slave 0109 executes the processes necessary at the end of the analysis, such as releasing the memory used in the analysis, and then ends the process. Note that the slave 0109 may return the process to S1602 without terminating the process and wait for the start of the next rule generation problem.

  In the slave processing according to the present embodiment, the slave 0109 can be changed to accept an analysis interruption instruction from the input / output device 0105. The presence / absence of an analysis interruption instruction may be checked before the slave 0109 requests a job from the master 0101 in S1603, for example. When there is an instruction to stop the analysis, the slave 0109 performs the result transmission process S1606 and the analysis end process S1608, and transmits the analysis stop signal to the master 0101, and then the process ends. .

  As mentioned above, although two embodiment which concerns on this invention was described, these embodiment can be changed as follows, for example.

  First, the assignment of jobs to slaves in the first and second embodiments may be changed so that assignment is performed in descending order of the number of nodes included in the job.

  Secondly, in the first and second embodiments, the job assignment to the slave is performed by dynamically changing the job size according to the progress of the rule generation, and re-dividing the job. You may change so that it may be assigned to. For example, if the number of jobs that are not assigned to slaves in rule generation falls below a certain number determined by the analyst, the analyst will analyze the remaining jobs only once through one rule generation. Depending on the division method specified, the job can be subdivided into smaller jobs and assigned to slaves. As such a dividing method, for example, a method of dividing all jobs that have not been analyzed into two equal parts can be considered.

  According to the first and second modified examples, the size of the job is reduced as the end of rule generation is approached, and the time required for the slave to process the job is shortened. For this reason, it is possible to perform rule generation with the processing end times in each slave being aligned with higher accuracy, and as a result, it is possible to improve the overall processing speed.

Third, the division of the search tree into jobs in the analysis execution start process S0805 of the first embodiment may be realized by the following processes (1) to (10).
(1) The values of the division parameter a and the minimum job size parameter b are determined in advance. Alternatively, an analyst inputs in advance from the input / output device 0105. Here, a and b are integers of 1 or more. The master 0101 reads the number of slaves (S) registered in the slave management table.
(2) The master 0101 sets the initial value of the layer number I to 1.
(3) The master 0101 counts the number of nodes included in layer I (N).
(4) The master 0101 calculates N / (a * S) and truncates the decimal part. This is Z. Here, Z is an integer and is the number of nodes included in one job.
(5) If Z <b, the master 0101 substitutes b for Z.
(6) The master 0101 sets the first node of layer I as X_s.
(7) In layer I, the master 0101 sets the Zth node counted from X_s as X_e. However, when the last node of layer I is reached before counting up to the Zth, the master 0101 sets the last node of layer I as X_e.
(8) The master 0101 adds a job to the job management table using X_s as a start node and X_e as an end node.
(9) If X_e is the last node of layer I, master 0101 returns to (10), otherwise master 0101 sets X_s as the next node of X_e in layer I and returns to (7). .
(10) If I is equal to the maximum number of conditional clauses, the master 0101 ends the process. Otherwise, the master 0101 adds 1 to I and returns to (3).

  According to the third modified example, each layer can be divided into jobs more flexibly, and the load can be flexibly allocated to the slaves.

  The data analysis method according to the present invention can be used for data analysis that needs to search a huge search space such as analysis of purchase data and gene expression analysis using a DNA chip.

It is a schematic block diagram of the information processing system which performs the data analysis which concerns on embodiment of this invention. It is the figure which showed notionally the data structure of the analysis object data which concern on 1st embodiment of this invention. It is the figure which showed notionally the data structure of the characteristic rule which concerns on 1st embodiment of this invention. It is the figure which showed an example of the tree structure showing the search space for discovering a characteristic rule. It is the figure which showed an example of the tree structure showing the search space for discovering a characteristic rule. FIG. 6 is a diagram for explaining a method of dividing the tree structure of FIG. 5 into jobs. FIG. 6 is a diagram for explaining a method of dividing the tree structure of FIG. 5 into jobs. It is a flowchart of the process for managing the slave which a master performs based on the 1st Embodiment of this invention. It is the figure which showed notionally the data structure of the slave management table which concerns on embodiment of this invention. It is the figure which showed notionally the data structure of the job management table which concerns on the 1st Embodiment of this invention. It is a flowchart of the slave management processing which a master performs by S0805 based on embodiment of this invention. It is a flowchart of the analysis management process which the master based on the 1st Embodiment of this invention performs by S0805. It is a flowchart of the slave process which concerns on the 1st Embodiment of this invention. It is the figure which showed notionally the data structure of the job management table which concerns on the 2nd Embodiment of this invention. It is a flowchart of the analysis process which the master based on the 2nd Embodiment of this invention performs. It is a flowchart of the slave process which concerns on the 2nd Embodiment of this invention.

Explanation of symbols

0102 ... Processing device, 0103 ... Memory, 0104 ... Program, 0105 ... Input / output device, 0106 ... Communication device, 0107 ... Storage device, 0110 ... Communication path, 0111 ... Communication path

Claims (11)

In a data analysis method for performing data analysis of an analysis problem using a first data processing device and a plurality of second data processing devices connected via a communication path,
The first data processing device divides the analysis problem into a plurality of partial problems larger than the number of the second data processing devices, and assigns the partial problem one by one to each of the plurality of second data processing devices. Processing,
Until all of the plurality of partial problems have been assigned to any of the second data processing devices, the first data processing device is the second data processing device of any of the plurality of second data processing devices, A process of allocating an unassigned partial problem among the plurality of partial problems to the second data processing apparatus each time processing of the partial problem assigned to the second data processing apparatus is terminated;
A process in which the first data processing device receives a processing result of a partial problem assigned to the second data processing device from each of the second data processing devices;
A data analysis method characterized by comprising: In a data analysis method for performing data analysis of an analysis problem using a first data processing device and a plurality of second data processing devices connected via a communication path,
Each time the second data processing device is assigned the partial problem of the analysis problem from the first data processing device until the assignment of the partial problem of the analysis problem from the first data processing device is stopped, Processing the subproblem and storing the result of the processing;
A process in which the second data processing apparatus in which assignment of the partial problem of the analysis problem is stopped transmits the stored processing result to the first data processing apparatus;
A data analysis method characterized by comprising: In a data analysis method for analyzing an analysis problem using a first data processing device and a plurality of second data processing devices connected via a communication path,
Each of the second data processing devices has input means for receiving an interruption instruction,
The data analysis method is
Each of the second data processing devices processes the partial problem each time the partial problem of the analysis problem is assigned from the first data processing device until the input means receives an interruption instruction, and Processing to save the results;
A process in which the second data processing apparatus that has received the interruption instruction by the input means transmits the stored processing result to the first data processing apparatus;
A data analysis method characterized by comprising: A data analysis method according to any one of claims 1, 2 and 3,
The analysis problem is to obtain a feature rule of the form IF to THEN ... describing the characteristics of the data from the analysis target data,
The data analysis method is
The first data processing device receives a plurality of feature rules and evaluation values of the feature rules as processing results of partial problems assigned to the second data processing device, respectively, from the plurality of second data processing devices, From the received all feature rules, select a higher-order feature rule, and output the selected feature rule as an analysis result of the analysis problem.
A data analysis method characterized by that. The data analysis method according to any one of claims 1, 2, 3, and 4,
The data analysis method, wherein the first data processing device divides the analysis problem into the plurality of partial problems so that a calculation amount necessary for processing the plurality of partial problems becomes uneven. A data analysis method according to claim 5, comprising:
The first data processing device assigns the plurality of partial problems to the plurality of second data processing devices in descending order of calculation amount necessary for processing. A data analysis method according to claim 1, comprising:
After starting the assignment of the partial problem to the second data processing apparatus, the first data processing apparatus assigns one or more unassigned partial problems to a larger or smaller number than the unassigned partial problem. The data analysis method characterized by including the process of subdividing into the subproblems of. In the data analysis method of any one of Claims 1-7,
The first data processing device has an input / output device;
The data analysis method is
A process in which the input means receives an input of an estimated maximum value of the number of the plurality of second data processing devices;
The first data processing device counts the number of the plurality of subproblems, and if the number is smaller than the estimated maximum value accepted by the input means, a division process for further dividing the plurality of subproblems;
A process in which the first data processing device repeats the division process until the number of the plurality of partial problems becomes larger than the estimated maximum value;
A data analysis method characterized by comprising:   The program for making a computer perform the process according to the data analysis method of any one of Claims 1-8.   A computer-readable storage medium on which the program according to claim 9 is recorded. In an information processing apparatus that is connected to a plurality of data processing devices via a communication path and causes the plurality of data processing devices to execute a partial problem of an analysis problem,
Control means for dividing the analysis problem into a plurality of partial problems larger than the number of the plurality of data processing devices, and assigning the partial problem to each of the plurality of data processing devices one by one;
A communication means for receiving the processing result of the partial problem assigned to the data processing device from each of the data processing devices via the communication path;
Have
The control means includes
Each time the data processing device of any of the plurality of data processing devices ends the processing of the partial problem assigned to the data processing device until all of the plurality of partial issues have been assigned, An information processing apparatus characterized in that, among the partial problems, an unassigned partial problem is assigned to the data processing apparatus, and the data processing apparatus outputs the assigned partial problem from the communication unit so as to be transmitted.

Images