JP5637071B2 - Processing program, processing method, and processing apparatus - Google Patents

Description

  This case relates to a processing program, a processing method, and a processing apparatus.

  An analysis process for a large amount of data requires a very long processing time. On the other hand, recently, an approach has been taken to reduce processing time by performing distributed / parallel processing using a plurality of machines. As the distributed / parallel processing, for example, there is a method using the MapReduce algorithm (see, for example, Non-Patent Document 1). In addition, Apache Hadoop exists as an open source implementation of the MapReduce algorithm.

  MapReduce mainly includes a “Map process” that divides the original data into a large number of key and value sets, and a “Reduce process” that aggregates these key and value sets according to a certain rule. Since each of the Map processing and Reduce processing can be executed in parallel, the processing performance of the plurality of machines can be utilized by assigning them to a plurality of processing machines (such as servers).

  However, in order to enhance the effect of distributed / parallel processing by MapReduce, it is necessary to increase the independence of each Map processing and Reduce processing so that the processing can be performed without depending on other portions.

  One type of analysis processing is to group related data out of a large amount of data. For example, as shown in FIG. 22A, business logs performed at a certain time are grouped into a series of business flow units as shown in FIG. 22B. In the grouping process, it is not necessary to consider the data of another group when handling a data group of a certain group. Therefore, the process can be efficiently performed by distributing the processes of each group to a plurality of servers.

  In addition, when grouping data in which a series of business flows are represented by one key type (flow ID in FIG. 22A) as shown in FIG. 22A, the grouping is easily achieved by using MapReduce. Is done. This is because the processing machine that performs MapReduce has a standard function to aggregate a group of data having a certain key value in one place.

Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified Data Processing on Large Clusters OSDI 2004

  However, as shown in FIG. 23A, there may be a plurality of keys (three types in FIG. 23A) indicating a series of business flows. FIG. 23B shows an example in which the data of FIG. In such a case, grouping cannot be performed simply. In the process of aggregating related data groups using multiple key types (hereinafter referred to as “multi-key aggregation process”), it is necessary to look at the entire data to determine which key combination represents a series of data groups. This is because it is not completely fixed. For example, in the case of FIG. 23 (a), if it is attempted to aggregate with the slip number = 001, the data of the slip detail table cannot be consolidated. On the other hand, when data is consolidated with the slip detail number = 001-001-001, the data in the slip table cannot be consolidated.

  In this case, it is necessary to devise a way to proceed with the aggregation process while updating the information of the key value combination according to the progress of the process. May occur.

  On the other hand, it is also conceivable to create a table for managing the relationship between key types in an RDB (Relational Database). However, if there is a place where each processing machine that performs distributed / parallel processing refers / updates in common, the performance / scalability of the distributed processing may deteriorate.

  Therefore, this case has been made in view of the above problems, and it is possible to prevent omission of data aggregation and improve performance and scalability when a plurality of data classified by a plurality of key types is aggregated. It is an object to provide a processing program, a processing method, and a processing apparatus.

  The processing program described in this specification includes a key list that is a list in which a plurality of data classified by a plurality of key types are aggregated, a key that is a list of keys associated with each data, and one of the keys. Map data associated with an aggregate key that is one key is generated and stored in a storage unit, and a key list of a group of map data associated with the same aggregate key is obtained by referring to the storage unit, and the acquisition is performed When the maximum number of keys in the number of keys included in the selected key list is compared with the total number of keys obtained by merging all the acquired key lists, and as a result of the comparison, the total number of keys is greater In addition, each of the keys included in the merged key list is set as an aggregate key, and the new map data associated with the merged key list as the key list is merged. Stored in the storage unit generates only a few minutes of the keys contained in Risuto a processing program for executing processing.

  In the processing method described in this specification, a computer that aggregates a plurality of data classified by a plurality of key types includes a key list that is a list in which keys associated with each data are collected, and one of the keys. A step of generating map data associated with an aggregate key that is one key and storing the map data in a storage unit; referring to the storage unit; obtaining a key list of a group of map data associated with the same aggregate key; The step of comparing the maximum number of keys in the number of keys included in the acquired key list with the total number of keys obtained by merging all the acquired key lists, and the result of comparison in the comparing step, the total When the number of keys is larger, each key included in the merged key list is used as an aggregate key, and a new list in which the merged key list is associated as a key list. The Updater is a processing method for executing the steps of storing in the storage unit generates only a few minutes of the keys contained in the key list that the merged.

  The processing device described in this specification is a processing device that aggregates a plurality of data classified by a plurality of key types, and stores the plurality of data and map data generated from the plurality of data A key list that is a list of related keys for each data, and map data that associates an aggregate key that is one of the keys, and stores the map data in the storage unit. With reference to the storage unit, the storage unit, to obtain a key list of a group of map data associated with the same aggregate key, the maximum number of keys among the number of keys included in the acquired key list, A comparison unit that compares the total number of keys obtained by merging all the acquired key lists, and a result of comparison in the comparison step, if the total number of keys is greater, the merged key number Each key included in the list is used as an aggregate key, and new map data in which the merged key list is associated as a key list is generated by the number of keys included in the merged key list and stored in the storage unit. And a second generation / storage unit.

  The processing program, the processing method, and the processing device described in this specification can prevent omission of data aggregation and improve performance and scalability when a plurality of data classified by a plurality of key types are aggregated. There is an effect.

1 is a diagram schematically illustrating a configuration of a distributed processing system according to an embodiment. FIG. It is a figure which shows the hardware constitutions of a processing server. FIG. 3A is a functional block diagram of the processing server, and FIG. 3B is a functional block diagram of the management server. It is a figure for demonstrating the basic processing content of MapReduce processing. It is a flowchart shown about a series of flows of MapReduce processing. FIG. 6A is a diagram illustrating an example of the processing target data, and FIG. 6B and FIG. 6C are diagrams illustrating Map data generated from the data in FIG. It is a figure which shows an example of the data of aggregation object. FIGS. 8A and 8B are diagrams showing two Map data generated from the data in FIG. 7 where the original key is Y = 0101, the related keys are X = 01, and Y = 0101. is there. It is a figure (the 1) which shows 1st aggregation. It is a flowchart which shows the specific process of the Reduce process performed in parallel by each process server in step S14 of FIG. It is a figure (the 2) which shows 1st aggregation. It is FIG. (The 3) which shows 1st aggregation. It is FIG. (The 4) which shows 1st aggregation. It is a figure (the 1) which shows 2nd aggregation. It is a figure (the 2) which shows 2nd aggregation. It is a figure (the 3-1) which shows the 2nd aggregation. It is a figure (the 3-2) which shows the 2nd aggregation. It is a figure which shows the 3rd aggregation. It is a figure which shows 4th aggregation. It is a figure which shows the 5th aggregation. FIG. 21A is a diagram showing a data aggregation state when the determination in step S16 in FIG. 5 is affirmed, and FIG. 21B is a data aggregation after steps S18 and S220 are performed. It is a figure which shows a state. It is FIG. (1) for demonstrating a prior art example. It is FIG. (2) for demonstrating a prior art example.

  Hereinafter, an embodiment will be described in detail with reference to FIGS. FIG. 1 schematically shows the configuration of the distributed processing system 100. In the distributed processing system 100 according to the present embodiment, “multiple key aggregation processing” for aggregating data classified by a plurality of key types is performed, and the MapReduce algorithm is applied in this processing. Here, aggregation means obtaining and collecting data of the same key.

  As shown in FIG. 1, the distributed processing system 100 includes a processing server 10 as n processing devices (computers) that execute processing, and a management server 20 that manages processing of each processing server 10. Each processing server 10 and management server 20 are connected to a network 30 such as a LAN (Local Area Network) or the Internet.

  FIG. 2 shows the hardware configuration of the processing server 10. As shown in FIG. 2, the processing server 10 includes a CPU 90, a ROM 92, a RAM 94, a storage unit (here, HDD (Hard Disk Drive)) 96, a network interface 97, a portable storage medium drive 99, and the like. Each component of the processing server 10 is connected to the bus 98. In the processing server 10, the CPU 90 executes a program (processing program) stored in the ROM 92 or the HDD 96 or a program (processing program) read from the portable storage medium 91 by the portable storage medium drive 99. The function of each part 3 (a) is realized. The management server 20 has the same configuration as that of the processing server 10, but in the management server 20, a program (management program) stored in the ROM or HDD or a drive for a portable storage medium is portable storage. When the CPU executes a program (management program) read from the medium, the functions of the respective units in FIG. 3B are realized. The details of the functions shown in FIGS. 3A and 3B will be described later.

  Returning to FIG. 1, the disk (HDD 96) of each processing server 10 is incorporated in a distributed file system 40 as a storage unit that appears virtually as one disk. For convenience of illustration, in FIG. 1, the HDD 96 is shown outside each processing server 10.

  FIG. 3A shows a functional block diagram of the processing server 10. The processing server 10 causes the CPU 90 to execute a processing program, thereby causing the Map processing unit 12 as the first generation / storage unit, the Reduce processing unit as the second generation / storage unit, and the comparison unit illustrated in FIG. 14 is realized.

  The Map processing unit 12 executes Map processing, which will be described later, using data stored in the distributed file system 40. The Reduce processing unit 14 performs Reduce processing, which will be described later, using the data (Map data) that has been subjected to the Map processing in the Map processing unit 12.

  FIG. 3B shows a functional block diagram of the management server 20. In the management server 20, the CPU executes the management program, thereby realizing the functions as the process management unit 22, the key number increase flag management unit 24, and the communication unit 26 illustrated in FIG. In the present embodiment, the processing program of each processing server 10 is started in conjunction with the execution of the management program in the management server 20.

  The process management unit 22 grasps the state of each processing server 10 via the communication unit 26 and refers to a key number increase flag (described later) managed by the key number increase flag management unit 24 to each processing server 10. To execute the process.

  The key number increase flag management unit 24 manages the value of the key number increase flag based on information acquired from each processing server 10 via the communication unit 26. The key number increase flag can take a value of “0” or “1”. Note that only the key number increase flag management unit 24 can refer to the value of the key number increase flag. Therefore, in each processing server 10, the key number increase flag management unit 24 is only remotely and asynchronously instructed to change the flag value without referring to the counter value. For this reason, the processing of the processing program is not waited under the influence of the counter.

  The communication unit 26 communicates with the processing server 10 and transmits a communication result to the processing management unit 22 and the key number increase flag management unit 24. In addition, the communication unit 26 transmits an instruction from the process management unit 22 to the processing server 10.

  The above configuration is a standard configuration when Apache Hadoop is used as the distributed processing middleware. Therefore, the functions of the distributed file system 40, processing program start / stop / control, and counter are implemented as basic functions of Hadoop.

  Next, basic processing contents of the MapReduce processing will be described with reference to FIG. As a processing flow, initially, it is assumed that all data to be processed is stored in the storage on the management server 20. When the management server 20 uploads this to the distributed file system 40, the data to be processed is automatically divided and stored in the storage of each processing server 10 so as to have a uniform amount. After this, MapReduce processing is executed. FIG. 4 is a diagram showing the basic concept of the MapReduce process.

  In MapReduce processing, processing data on the distributed file system 40 is divided into map data consisting of a primary key (which is also called an “aggregation key” because it is a key used for aggregation) and a value (Map processing), The processing (Reduce processing) for collecting the Map data according to the key value is performed in a distributed and parallel manner in each processing server 10. The format of the primary key and the value in the Map process is arbitrary, and there may be a Map process that does not generate any Map data. For example, it is assumed that Map data as shown in the upper part of FIG. 4 is generated by Map processing performed in a distributed and parallel manner in each processing server 10.

  On the other hand, the Reduce process means that the Map data generated by the Map process is used as the original data, the processing server is distributed according to the primary key value, and some process is performed on the Map data group grouped according to a certain standard. . In general, a group of Map data having the same primary key value is grouped, and processing is performed on the group of Map data.

  FIG. 4 shows the most basic processing server distribution method. Specifically, each processing server 10 or management server 20 calculates a unique hash value by a known calculation method for the value of the primary key of each Map data, and calculates the hash value by the number of processing servers (in FIG. 4). Find the remainder (0-2) divided by 3). In this case, regarding each processing server 10, the processing server 10 that processes each Map data can be determined by determining the corresponding remainder value with the numbers (0 to 2) shown in the lower part of FIG. 4. it can. Since the same hash value is always obtained for the primary key having the same value, the Map data group having the same primary key value is collected in one processing server. Also, if the hash value is premised on that there is no bias, the processing of each Map data can be distributed to each server without bias. Note that the method for uniquely determining a processing server from the hash value described above is the simplest example. Therefore, for example, in addition to the hash value, the processing server at that time may be taken into consideration and the processing server may be determined at a higher level. In general, since the types of primary key values are larger than the number of processing servers 10, each processing server 10 processes a plurality of Map data groups. Each processing server 10 simultaneously processes a plurality of Map data groups by multi-process or multi-thread.

  Each processing server 10 (Reduce processing unit 14) refers to the value of the primary key of the collected Map data. Each processing server 10 (Reduce processing unit 14) sets Map data (Map data group) having the same primary key value as one group, and performs Reduce processing on the group (lower part of FIG. 4). Primary key = see CCC group). Since the management server 20 (process management unit 22) knows the state of each process server 10, it also knows whether the map process and the reduce process in each process server 10 have been completed. For this reason, the management server 20 can repeatedly perform the MapReduce process by receiving the result of the Reduce process and causing each process server 10 to execute the Map process again.

  Next, details of the multiple key aggregation processing in the distributed processing system 100 of this embodiment will be described. The multiple key aggregation process in this case means an aggregation process when there are a plurality of key types (key types) indicating a series of business flows as shown in FIG.

  FIG. 5 is a flowchart showing a specific processing flow in the multiple key aggregation processing. The process of FIG. 5 is started after the process management unit 22 of the management server 20 uploads the original data to be processed to the distributed file system 40. In the multiple key aggregation process of FIG. 5, the Map process and the Reduce process are repeated as many times as necessary. By repeating this MapReduce process, the grouping of data progresses, and the result of correctly grouping the data linked with multiple keys at the end can be obtained. In the following, one MapReduce process is referred to as “aggregation”.

  In the process of FIG. 5, first, in step S10, the key number increase flag management unit 24 of the management server 20 initializes the key number increase flag to zero.

  Next, in step S12, the Map processing unit 12 of the processing server 10 executes Map processing for generating Map data from the original data. The Map process of each processing server 10 is activated by the Hadoop function. In the map processing of each server, among the processing target data uploaded to the distributed file system 40, the original data (for example, FIG. 6A) stored in the server is compared with FIG. 6B and FIG. A process of generating Map data having a structure as shown in c) is performed. In general, in Hadoop, a file uploaded to the distributed file system 40 is copied and held in a plurality of servers with the same data for redundancy. However, here, for simplification of explanation, it is assumed that data having the same content exists only in one place. Here, since the data in FIG. 6A is data associated with a plurality of keys, each record always has a primary key and also has 0 to a plurality of related keys. It is obvious that which item is a primary key and which item is a related key in one record and is defined in advance. For example, it is assumed that a key that is the first key in a lexicographic manner among a plurality of keys (a key that is the head in the order of X → Y → Z and in ascending order of numerical values) is the main key. This definition exists on the management server 20 and is notified to each processing server 10 as a parameter when starting the Map processing. Map data has a “primary key” and a “value”, and a key list and an event list are stored in the “value”. Here, the event refers to the input data itself.

  In the Map process, the Map processing unit 12 reads one record of data to be processed, lists up item names and values of the primary key and related keys, and creates a key list. FIG. 6A shows an example of processing one record of the slip detail table, where “slip detail number (Y)” is the primary key and its value is “0101”. The related key is “slip detail number (Z)” and the value is “010101”. Therefore, the key list is composed of two keys.

  The Map processing unit 12 creates Map data for the number of elements in the key list (for the number of keys). In the example of FIG. 6A, since the number of keys included in the key list is two, the generated map data is two as shown in FIGS. 6B and 6C. Each element of the key list is used as the primary key of each generated Map data. Therefore, the primary key of the first Map data is “0101” of “slip detail number (Y)”, and the primary key of the second Map data is “010101” of “slip detail number (Z)”. become.

  Also, the Map processing unit 12 evaluates the key value in the generated Map data lexicographically, and stores the content (data content) of the input record as one element only in the event list of the top Map data. . At the time of the first aggregation, the event list stores only a maximum of one. For Map data whose key value is other than the head, the event list is emptied (see FIG. 6C). If the original input data includes useless data that is not used in the subsequent processing, it is possible to filter when storing the event list by defining which items are unnecessary. is there.

  Here, FIG. 7 shows an example of an initial state of a plurality of processing target data in the present embodiment. In the example of FIG. 7, the bold rectangle frame indicates event data, and the broken rectangle frame indicates a key list. One event corresponds to one record of input data, and the key list below each event indicates the main key and related keys included in the record. In the present embodiment, it is not necessary to distinguish between the main key and the related key in the key list, but the value of the main key of the input data is described as “original key” for event determination. In FIG. 7, the broken line connecting the data indicates the relationship between the related key and the primary key. The input data consists of three layers, and the relationship is set from the bottom layer to the second layer and from the second layer to the top layer. In FIG. 7, the correct answer is finally grouped into two groups (groups divided by a one-dot chain line). FIG. 7 includes a case where one type of lower layer key is associated with a plurality of upper layer keys in the left group, such as Z = 0101101. For this reason, it is not possible to apply a method of aggregation by simply repeating aggregation from the lowest layer to the upper layer.

  For the data to be processed shown in FIG. 7, the map processing unit 12 performs the map processing as described above. For example, when the original key is Y = 0101 and the related key is X = 01, Y = 0101, two pieces of Map data as shown in FIGS. 8A and 8B are generated. Become. These two Map data are Map data indicated by arrows (→) in FIG. Note that the Map process is similarly performed on the other data in FIG. 7, and a large number of Map data as shown in FIG. 9 is generated. In FIG. 9, the key described at the top of each Map data means the main key.

  After the completion of the Map process, the generated Map data is redistributed to each processing server as described with reference to FIG. The detection of the completion of the Map process, the distribution of the Map data, and the start of the Reduce process are executed by the Hadoop function.

  Returning to FIG. 5, in step S <b> 14, the Reduce processing unit 14 aggregates the data group for each primary key and executes the Reduce process. In this step S14, specifically, the processing according to the flowchart of FIG. It should be noted that the Reduce process is executed for the Map data group having the same primary key value. Here, the processing unit is referred to as “aggregation group” or “group”.

  In the process of FIG. 10, in step S30, the Reduce processing unit 14 initializes an event list, a key list, and the maximum number of input keys of each group (primary keys are X = 01 to Z = 010106). In this case, the initial value of the maximum number of input keys is 0. FIG. 11 is a diagram showing the Map data of FIG. 9 collectively for each group. Thereafter, in steps S32 to S40, a loop process for each Map data is performed.

  In step S32, the Reduce processing unit 14 acquires map data to be processed. In this case, the Reduce processing unit 14 of each processing server 10 acquires one Map data from the aggregation group (data group having the same primary key value) shown in FIG.

  Next, in step S34, if the number of elements (number of keys) in the key list of the Map data is larger than the maximum number of input keys in the same group acquired so far, the Reduce processing unit 14 sets the number of keys as the maximum input key. Set to a number. That is, the Reduce processing unit 14 counts the number of elements (number of keys) of the key list of the Map data for the acquired Map data (group), and if it is larger than the maximum number of input keys of the group, the maximum input key of the group Update the number with that value. Therefore, when step S34 is performed for all the map data of the aggregate group, the maximum number of input keys of the group is finally the maximum number of keys in the key list of the input map data group. Will match. For example, in the group with the primary key X = 01 in FIG. 11, the maximum number of input keys is “2” which is the maximum number of keys in the two key lists.

  Next, in step S36, the Reduce processing unit 14 copies the contents of the key list of the Map data to the group key list (see the thick broken line frame in FIG. 12). However, if the same key is already included in the group key list, the key (duplicate key) is not copied and discarded. Therefore, when step S36 is performed on all the map data of the aggregation group, finally, a key list (a thick broken line frame in FIG. 12) of the group in which the key lists are merged (aggregated) is generated. It will be.

  Next, in step S38, if the event is included in the Map data, the Reduce processing unit 14 adds (copies) the event to the event list. Regarding events, there are never a plurality of items having the same contents (even if the contents are the same, they are different data caused by different records). Therefore, the Reduce processing unit 14 can copy the event to the event list without particularly checking the contents.

  Next, in step S40, the Reduce processing unit 14 determines whether or not unprocessed Map data exists. When judgment here is affirmed, it returns to step S32. On the other hand, if the determination in step S40 is negative, the process proceeds to step S42. At the stage where the determination in step S40 is negative, the processing of all the Map data has been completed, and the maximum number of input keys for each group and the number of elements (key number) in the merged key list for each group are obtained. ing. In FIG. 12, in the numerical value (a → b) described in the upper right of each group, “a” means the maximum number of input keys of each group, and “b” indicates the merged key list of each group. Means the number of elements (number of keys).

  If the determination in step S40 is negative and the process proceeds to step S42, the Reduce processing unit 14 determines whether the number of keys (b) in the merged key list of the group is greater than the maximum number of input keys (a). In this comparison, either the maximum number of input keys matches the number of elements in the merged key list (number of keys) or the number of elements in the merged key list (number of keys) becomes larger.

  If the determination in step S42 is affirmative (if the number of keys in the merged key list is greater than the maximum number of input keys), the process proceeds to step S44. In FIG. 12, the determination in step S <b> 42 is affirmed in the case of processing in any of the groups in which the primary key is X = 03, 05, Y = 0101, 0102, 0103, 0104, 0105, and Z = 0101101. As described above, when the number of keys in the merged key list is larger, it means that the association between new keys is found by the aggregation, so that the aggregation is again performed using the relationship between the new keys.

  Accordingly, if the number of keys in the merged key list is larger, in step S44, the Reduce processing unit 14 notifies the management server 20 and the key number increase flag management unit 24 sets the key number increase flag to 1. Set to. When the key number increase flag is set to 1, since the process management unit 22 of the management server 20 always executes re-aggregation, the same effect as when the re-aggregation is instructed from the processing server 10 is obtained.

  Next, in step S46, the Reduce processing unit 14 creates Map data for the number of elements in the key list (number of keys) for the group of a <b. Here, the key of each Map data becomes each key included in the merged key list. Further, the key list of each group is stored as it is in the key list of each Map data. In addition, among the Map data, only the Map data in which the key value is lexicographically headed (Map data heading in the order of X → Y → Z and in ascending order of numerical values), the event list of the group in the event list Store as it is. In other words, the Reduce processing unit 12 specifies map data having a primary key as a primary key among the keys included in the merged key list, and associates the data only with the event list of the specified map data.

  In step S48, the Reduce processing unit 14 outputs the Map data created in step S46 as intermediate generation data of the Map data group to an intermediate file on the distributed file system 40. FIG. 13 is a diagram illustrating a state immediately after all the processes in FIG. 10 are completed (immediately after the first aggregation process is completed). In FIG. 13, Map data (any one) surrounded by a one-dot chain line is output to the intermediate file. This process can be said to be a process of copying and distributing the merged key list to a group of Map data whose primary key is a key that is newly determined in the merged key list. Thereafter, all the processes in FIG. 10 are terminated, and the process proceeds to step S16 in FIG.

  The output Map data is restored as Map data from the intermediate file by Map processing when re-aggregation is performed, and is redistributed to the processing server 10 corresponding to the key value in the next Reduce processing. Therefore, the Map process (S12) in the second and subsequent aggregations is different from the Map process (S12) in the first aggregation that generates Map data from the input data file, and only restores the Map data serialized in the intermediate file as an object. It becomes processing.

  On the other hand, if the determination in step S42 of FIG. 10 is negative, the process proceeds to step S50. In step S50, the Reduce processing unit 14 determines whether an event exists in the event list. When judgment here is affirmed, it transfers to step S52. Note that the process proceeds to step S52 only when any of the groups X = 01, 02, 04 in FIG. 12 is processed.

  In step S52, single Map data is generated using the key that is lexicographically first in the key list as a main key. Specifically, only one Map data is generated, and the first value in the group key list is used as a key, the key list stores the group key list as it is, and the event list stores the group. Store the event list as it is. In step S48, the Reduce processing unit 14 outputs the Map data generated in step S52 to the intermediate file as intermediate generation data of the Map data group. Note that the multiple Map data is not generated here because if the key list does not increase, the new relationship between the keys is not known and the information does not increase, so the key list is sent to all aggregation groups. This is because it is useless. Note that the Map data generated in Step S52 and output to the intermediate file in Step S48 is Map data (any one) that is not surrounded by a one-dot chain line in FIG. Thereafter, all the processes in FIG. 10 are terminated, and the process proceeds to step S16 in FIG.

  On the other hand, if the determination in step S50 of FIG. 10 is negative, that is, if there is no event in the event list, no new key association has occurred in the aggregation group. In this case, without outputting anything as the intermediate generation data (that is, by deleting the group), the entire processing in FIG. 10 is terminated, and the process proceeds to step S16 in FIG. If the determination in step S50 is negative, nothing is output as intermediate generation data if the number of keys does not increase, and if there is an event related to that key group, that event This is because the key list having the same contents should be sent to the group in which the key is processed, so that the key list should be held in that group. In other words, in a group in which no event exists, information is not lost even if the key list is discarded.

  Returning to FIG. 5, after the processing by each processing server 10 (steps S <b> 12 and S <b> 14) is completed, the process proceeds to step S <b> 16 in FIG. 5. In step S16, the key number increase flag management unit 24 checks the key number increase flag and determines whether or not the flag is “1”. If the determination here is affirmative, that is, if the key number increase counter is “1”, the key aggregation has not been completed, so the process returns to step S10, and under the instruction of the process management unit 22, each process server 10 executes the same MapReduce process again.

  For example, in the case of the second processing in steps S12 and S14 (in the case of the second aggregation), the Map data in FIG. 13 is aggregated as shown in FIG. FIG. 15 is a diagram in which the maximum input key number a and the merged key list key number b are clearly shown in FIG. As shown in FIG. 15, in the second aggregation process, eight groups of primary keys X = 03, 05, Y = 0101, 0102, 0103, 0104, 0105, and Z = 0101101 are merged key list keys. The number (b) is larger than the maximum input key number (a). Accordingly, with respect to these groups, in step S48, Map data indicated by a one-dot chain line in FIGS. 16 and 17 is generated and output as an intermediate file.

  On the other hand, in the three groups where the primary key is X = 01, 02, 04, the determination in step S42 is denied and the determination in step S50 is affirmed. For this reason, in step S52, three pieces of Map data not surrounded by the one-dot chain line in FIG. 16 are generated and output as an intermediate file. Other primary key groups are not output (deleted) as intermediate files.

  Even in the second aggregation, the determination in step S42 may be affirmed, so the key number increase flag is set to “1”. Therefore, the third aggregation is also performed (step S16 is affirmed).

  In the third aggregation, the Map data in FIGS. 16 and 17 are aggregated as shown in FIG. In this case, as shown in FIG. 18, the number of elements in the key list is larger than the maximum number of input keys in five groups of primary keys X = 03, Y = 0101, 0102, 0103, and Z = 0101101 (step The determination in S42 is affirmed). Further, in the two groups with the primary key X = 01, 05, the determination in step S42 is denied and the determination in step S50 is affirmed.

  Next, in the fourth aggregation, steps S44, S46, S48, and S52 are executed as described above. Thereby, as shown in FIG. 19, Map data is collected.

  Next, in the fifth aggregation, as shown in FIG. 19, there is no group (group of a <b) for which the determination in step S42 is affirmed. On the other hand, the groups for which the determination in step S50 is affirmative are two groups whose primary keys are X = 01,05. Therefore, in the fifth aggregation, as shown in FIG. 20, the two groups are aggregated.

  In the fifth aggregation, since step S42 is not affirmed as described above, the key number increase flag is “0”. For this reason, the determination of step S16 in FIG. 5 is denied. When the key number increase flag is “0”, it means that the aggregation of the related key group is completed, so the key aggregation loop process (steps S10 to S16) is completed, and the process proceeds to step S18 in FIG. It will be.

  In step S18, the Map processing unit 12 executes the Map process of the final aggregation process. Specifically, as the final aggregation process, the Map processing unit 12 receives the Map data (FIG. 21A) output to the intermediate file in the last Reduce process of the loop, and executes the final Map process. In this Map processing, the Map processing unit 12 receives the Map data of the intermediate file as input, creates new Map data with the key list of each Map data as a key, and the event list as a value, and the original Map data. Is discarded. Note that “using the key list of Map data as a key” means creating a new key by enumerating each key in the key list. That is, new Map data is generated from the Map data as shown in FIG. 21A with the key (primary key) as shown in FIG.

  Next, in Step S20, the Reduce processing unit 14 executes the Reduce process of the final aggregation process for the new Map data created in the Map process of Step S18. In the Reduce process, all event data is extracted from the event list of the collected Map data. If event type and time stamp information is extracted from these event data and the events are arranged in time series, one flow instance is completed for each aggregation group.

  Since related events are aggregated in the final aggregation, movement of events is not essential in the aggregation within the loop. However, moving events within a loop and collecting events in stages makes it possible to perform statistical processing of events in stages within Reduce processing, and so on, which has a great merit.

  When all the processes in FIG. 5 are completed by the above processing, data of a plurality of keys can be correctly totaled as shown in FIG.

  Note that the processing in steps S18 and S20 may not be performed when all events (data) can be correctly counted in the processing before step S18 as shown in FIG. However, in practice, since there is a possibility that all data cannot be correctly summed up unless the processes of steps S18 and S20 are performed, in this embodiment, the processes of steps S18 and S20 are always performed.

  As described above in detail, according to the present embodiment, the processing server 10 that aggregates a plurality of data classified by a plurality of key types includes a key list that is a list of related keys for each data, and Map data associated with a primary key, which is one of the keys, is generated and stored in the distributed file system 40. With reference to the distributed file system 40, a map data group associated with the same primary key is generated. Obtain a key list and compare the maximum number of keys (maximum number of input keys) among the number of keys included in the obtained key list with the total number of keys that merged all the obtained key lists. As a result, if the total number of keys is larger, each key included in the merged key list is set as the primary key, and the key list merged as the key list is related. Digit new Map data generated by the number of the keys contained in the merged key list stored in the distributed file system 40. As a result, the merged key list can be copied and distributed to a group of Map data in which the keys included in the merged key list (a plurality of keys determined to be related) are primary keys (step S46). . By doing so, it is possible to aggregate related keys only by referring to the Map data (particularly the key list) without using an RDB (relational database). Therefore, when a plurality of data classified by a plurality of key types are aggregated, it is possible to obtain a performance / scalability improvement effect. In addition, since the newly generated Map data is subjected to the Reduce process in each processing server, omission of data aggregation can be eliminated. Further, in this embodiment, when the number of keys in the key list associated with the same primary key increases, new Map data is generated using each key included in the merged key list as the primary key. The number of aggregation processes to be executed can be increased. Thereby, related keys can be aggregated by repeating the MapReduce process a small number of times.

  In addition, according to the present embodiment, the Map data includes an event list that is a list of data contents, and the maximum number of keys (maximum number of input keys) among the number of keys included in the acquired key list, If the total number of keys is greater as a result of comparing the total number of keys merged for all the acquired key lists, the lexicographically leading key in the merged key list is the primary key. Map data is specified, and the data is associated only with the event list of the specified Map data. Accordingly, aggregation of related keys and aggregation of events (data) can be appropriately performed, so that aggregation can be performed by repeating the MapReduce process a small number of times.

  Further, according to the present embodiment, when the total number of keys is not larger than the maximum number of input keys (when the total number of keys matches the maximum number of input keys), the acquired event list includes data. In this case, a key that is lexicographically first in the merged key list is used as a primary key, and an event list obtained by merging the acquired event list and new map data associated with the merged key list are generated. Thereby, in this embodiment, generation | occurrence | production of the aggregation process more than necessary of the Map data which is not related to data (event) can be suppressed.

  Further, according to the present embodiment, when the total number of keys is not larger than the maximum number of input keys in the group key list associated with all primary keys stored in the distributed file system 40 (total keys). When the number matches the maximum number of input keys), a unique value is created based on the key list stored in the distributed file system 40, and Map data using the value as a primary key is generated. As a result, even if all data cannot be correctly aggregated before the total number of keys in all groups becomes larger than the maximum number of input keys, it is possible to correctly generate Map data using a unique value as a primary key. Data that could not be aggregated can be aggregated correctly.

  Here, the inventor (and the applicant) of the present application has applied for an invention for solving the same problem as in the present application in Japanese Patent Application No. 2011-050745 (hereinafter referred to as “prior application”). In the present embodiment, by increasing the number of aggregation processes that are simultaneously distributed and executed, the number of aggregations can be reduced as much as possible. Therefore, in the present embodiment, in particular, even when a processing system composed of a large number of processing servers 10 can be occupied by a single aggregation process for a certain data, or even when a plurality of aggregation processes are performed. When there is a margin in system performance, processing can be completed in the shortest processing time. On the other hand, in the prior application, although the number of times of aggregation is slightly larger than that of the present embodiment, the amount of processing in one aggregation is small, so that aggregation processing for a plurality of data is simultaneously executed in a certain processing system. In some cases and when there is no margin in processing system performance, it is possible to obtain the highest throughput performance. In other words, the present embodiment and the prior application can be appropriately used according to conditions such as the scale of the processing system and the number of users for the processing system.

  In addition, although the case where the management server 20 was provided was demonstrated in the said embodiment, not only this but the process similar to the management server 20 is good also as any of the processing servers 10 performing.

  The above processing functions can be realized by a computer. In that case, a program describing the processing contents of the functions that the processing apparatus should have is provided. By executing the program on a computer, the above processing functions are realized on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium.

  When the program is distributed, for example, it is sold in the form of a portable recording medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disc Read Only Memory) on which the program is recorded. It is also possible to store the program in a storage device of a server computer and transfer the program from the server computer to another computer via a network.

  The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute processing according to the program. Further, each time the program is transferred from the server computer, the computer can sequentially execute processing according to the received program.

  The above-described embodiment is an example of a preferred embodiment of the present invention. However, the present invention is not limited to this, and various modifications can be made without departing from the scope of the present invention.

In addition, the following additional notes are disclosed regarding the above description.
(Supplementary note 1) To a computer that aggregates multiple data classified by multiple key types,
A key list that is a list of keys associated with each data, and map data that associates an aggregate key that is one of the keys, and stores the map data in a storage unit;
Referring to the storage unit, obtain a key list of a group of map data associated with the same aggregate key, and obtain the maximum number of keys among the number of keys included in the obtained key list and all the obtained keys Compare the total number of keys merged with the key list,
As a result of the comparison, if the total number of keys is larger, each key included in the merged key list is set as an aggregate key, and new map data associating the merged key list as a key list, A processing program for generating a number of keys included in a merged key list and storing it in the storage unit.
(Supplementary Note 2) The map data includes an event list that is a list of contents of the data,
In the process of generating and storing the new map data, if the total number of keys is larger as a result of the comparison, a key that is lexicographically leading among the keys included in the merged key list is selected. The processing program according to appendix 1, wherein map data as an aggregation key is specified, and data is associated only with an event list of the specified map data.
(Supplementary Note 3) If the total number of keys is not larger as a result of the comparison in the comparison process and the acquired event list includes data, the dictionary is included in the merged key list. The first key in the formula is used as an aggregate key, and an event list obtained by merging the acquired event list and new map data associated with the merged key list are generated and stored in the storage unit. The processing program according to appendix 1 or 2.
(Supplementary Note 4) In the key list associated with all the aggregate keys stored in the storage unit as a result of the comparison in the comparison process, the total number of keys is not larger in the storage unit. Any one of appendices 1 to 3, wherein the computer is caused to execute a process of creating a unique value based on a stored key list and generating map data using the value as an aggregate key. Processing program.
(Supplementary Note 5) A computer that aggregates a plurality of data classified by a plurality of key types,
Generating a key list that is a list of related keys for each data and a map data that associates an aggregate key that is one of the keys and storing the map data in a storage unit; and
Referring to the storage unit, obtain a key list of a group of map data associated with the same aggregate key, and obtain the maximum number of keys among the number of keys included in the obtained key list and all the obtained keys A step of comparing the total number of keys merged with the key list;
If the total number of keys is larger as a result of the comparison in the comparing step, each key included in the merged key list is set as an aggregate key, and a new map in which the merged key list is associated as a key list And generating data corresponding to the number of keys included in the merged key list and storing the data in the storage unit.
(Supplementary Note 6) The map data includes an event list that is a list of contents of the data,
In the step of generating and storing the new map data, if the total number of keys is larger as a result of the comparison, a lexicographically leading key is included in the keys included in the merged key list. 6. The processing method according to appendix 5, wherein map data as an aggregation key is specified, and data is associated only with an event list of the specified map data.
(Supplementary Note 7) If the total key number is not larger as a result of the comparison in the comparing step and the acquired event list includes data, the dictionary is included in the merged key list. A step of generating a new map data associated with the event list obtained by merging the acquired event list and the merged key list, using the head key in the formula as an aggregate key, and storing the map in the storage unit; The processing method according to appendix 5 or 6, wherein:
(Supplementary Note 8) In the key list associated with all the aggregate keys stored in the storage unit as a result of the comparison in the comparison process, if the total key number is not larger, the storage unit stores The computer executes a step of creating a unique value based on a stored key list and generating map data using the value as an aggregate key. Processing method.
(Supplementary note 9) A processing device that aggregates a plurality of data classified by a plurality of key types,
A storage unit for storing the plurality of data and map data generated from the plurality of data;
A first generation / storage unit that generates a map list in which a key list that is a list of related keys for each piece of data and an aggregate key that is one of the keys is associated and is stored in the storage unit When,
Referring to the storage unit, obtain a key list of a group of map data associated with the same aggregate key, and obtain the maximum number of keys among the number of keys included in the obtained key list and all the obtained keys A comparison unit for comparing the total number of keys merged with the key list;
If the total number of keys is larger as a result of the comparison in the comparing step, each key included in the merged key list is set as an aggregate key, and a new map in which the merged key list is associated as a key list A processing device comprising: a second generation / storage unit that generates data for the number of keys included in the merged key list and stores the data in the storage unit.
(Supplementary Note 10) The map data includes an event list that is a list of contents of the data,
In the second generation / storage unit, when the total number of keys is larger as a result of the comparison by the comparison unit, the lexically leading key among the keys included in the merged key list is an aggregate key. The processing apparatus according to appendix 9, wherein the map data is specified and the data is associated only with the event list of the specified map data.
(Supplementary Note 11) As a result of comparison by the comparison unit, when the total number of keys is not larger and the acquired event list includes data, the merged key list is lexicographically. A third generation / storage unit that generates a new map data associated with the event list obtained by merging the acquired event list and the merged key list using the leading key as an aggregate key, and stores the generated map data in the storage unit; The processing apparatus according to appendix 9 or 10, provided.
(Supplementary Note 12) When the total number of keys is not larger in the key list associated with all the aggregate keys stored in the storage unit as a result of the comparison by the comparison unit, it is stored in the storage unit. The processing apparatus according to any one of appendices 9 to 11, further comprising a generation unit that generates a unique value based on the key list that is generated and generates map data using the value as an aggregate key.

10 processing server (computer, processing device)
12 Map processing unit (first generation / storage unit)
14 Reduce processing unit (comparison unit, second generation / storage unit)
40 Distributed file system (storage unit)

Claims (6)

To a computer that aggregates multiple data classified by multiple key types,
A key list that is a list of keys associated with each data, and map data that associates an aggregate key that is one of the keys, and stores the map data in a storage unit;
Referring to the storage unit, obtain a key list of a group of map data associated with the same aggregate key, and obtain the maximum number of keys among the number of keys included in the obtained key list and all the obtained keys Compare the total number of keys merged with the key list,
As a result of the comparison, if the total number of keys is larger, each key included in the merged key list is set as an aggregate key, and new map data associating the merged key list as a key list, As many as the number of keys included in the merged key list are generated and stored in the storage unit.
A processing program characterized by causing processing to be executed. The map data includes an event list that is a list of contents of the data;
In the process of generating and storing the new map data, if the total number of keys is larger as a result of the comparison, a key that is lexicographically leading among the keys included in the merged key list is selected. 2. The processing program according to claim 1, wherein map data as an aggregation key is specified, and data is associated only with an event list of the specified map data.   As a result of the comparison in the comparison process, when the total key number is not larger and the acquired event list includes data, the merged key list includes a lexicographic top The key is a consolidated key, and an event list obtained by merging the acquired event list and new map data associated with the merged key list are generated and stored in the storage unit. 2. The processing program according to 2. When the total number of keys is not larger in the key list associated with all the aggregate keys stored in the storage unit as a result of the comparison in the comparison process, the key list is stored in the storage unit. A process for creating a unique value based on the key list and generating map data using the value as an aggregate key,
The processing program according to any one of claims 1 to 3, wherein the computer is executed. A computer that aggregates multiple data classified by multiple key types,
Generating a key list that is a list of related keys for each data and a map data that associates an aggregate key that is one of the keys and storing the map data in a storage unit; and
Referring to the storage unit, obtain a key list of a group of map data associated with the same aggregate key, and obtain the maximum number of keys among the number of keys included in the obtained key list and all the obtained keys A step of comparing the total number of keys merged with the key list;
If the total number of keys is larger as a result of the comparison in the comparing step, each key included in the merged key list is set as an aggregate key, and a new map in which the merged key list is associated as a key list And generating data corresponding to the number of keys included in the merged key list and storing the data in the storage unit. A processing device that aggregates a plurality of data classified by a plurality of key types,
A storage unit for storing the plurality of data and map data generated from the plurality of data;
A first generation / storage unit that generates a map list in which a key list that is a list of related keys for each piece of data and an aggregate key that is one of the keys is associated and is stored in the storage unit When,
Referring to the storage unit, obtain a key list of a group of map data associated with the same aggregate key, and obtain the maximum number of keys among the number of keys included in the obtained key list and all the obtained keys A comparison unit for comparing the total number of keys merged with the key list;
If the total number of keys is larger as a result of the comparison in the comparing step, each key included in the merged key list is set as an aggregate key, and a new map in which the merged key list is associated as a key list A processing device comprising: a second generation / storage unit that generates data for the number of keys included in the merged key list and stores the data in the storage unit.

Images