JPWO2012131927A1 - Computer system and data management method - Google Patents

Abstract

A computer system in which a plurality of computers execute analysis processing on a data set including a plurality of data composed of keys and data values, and each computer has a divided region obtained by dividing the data set for each predetermined key range. The division information for managing the division position key, which is a key indicating the division position, is held for each data set. All the division position keys included in the division information of each data set are the same, and the computer system is newly added to the file system. When a data set is stored, based on the data size of each divided area after the new data set is stored, whether or not there is a target area that is a divided area with a data size larger than a predetermined threshold When it is determined that the target area exists, the target area is divided into a plurality of new divided areas.

Description

  The present invention relates to a technique for combining data in a computer system that processes a large amount of data.

  As a technique related to the process of joining tables (tables, relations, etc.) in a database, a technique is known in which table joins are processed in parallel using a sort / merge join technique (for example, see Patent Document 1).

  The sort / merge join technique is a method of sorting the tables to be joined based on the key values, then reading the rows of each table from the top, and merging the rows having the corresponding key values.

  In Patent Document 1, in order to parallelize the processing, each table is divided at a position corresponding to the same key value to generate a divided region corresponding to each table, and a sort / merge join technique is performed for each divided region. It is described that tables are joined using. Further, Patent Document 1 describes that a divided area is allocated to processors so that the processor load in the system is not biased.

  As a basic database technology, a table (index) that associates key values with data storage locations corresponding to the key values is prepared, and the key value is specified during data search processing. (See, for example, Patent Document 2). Patent Document 2 describes a matrix index that associates data storage positions with combinations of two or more keys.

  In addition, a technique is commonly used in which a plurality of storage areas can be used by changing a storage area for storing data for each key value range (see, for example, Patent Document 3). Patent Document 3 describes a method of leveling the usage amount of each storage area while suppressing the amount of data movement from the existing storage area to the newly added storage area when adding the storage area.

JP 7-11718 JP-A-6-52231 JP 2001-142751 A

  In the data analysis system, periodically acquired data is accumulated, and analysis processing is executed by combining the accumulated data as necessary.

  Here, an example of data processed by the data analysis system will be described with reference to the drawings.

  FIG. 20 is an explanatory diagram showing an example of data processed in a conventional data analysis system. FIG. 21 is an explanatory diagram showing an example of a schema in conventional data. 22A to 22C are explanatory diagrams illustrating an example of data processed in the conventional analysis processing.

  In the example shown in FIG. 20, the movement history of the user is shown. Specifically, the data includes a user ID for identifying the user, a position X and a position Y that are coordinate information specifying the position of the user, and a time stamp that is a time when the user moves to the position.

  In the analysis processing for data as shown in FIG. 20, for example, the data is converted based on the schema as shown in FIG. Further, the converted data is grouped for each user ID as shown in FIG. 22A, and analysis processing such as tabulation is executed.

  However, since it takes time to convert the data as shown in FIG. 20 into the data as shown in FIG. 22A during the analysis process, in this data analysis system, as shown in FIG. Data converted into correct data is stored, and analysis processing is executed using the stored data.

  In this specification, data composed of one or more records is referred to as a data set. Further, a data set as shown in FIG. 20 is described as raw data, and data having a structure as shown in FIG. 21 is described as structured data.

  In the accumulation process, data in the format shown in FIG. 20 is collected periodically (for example, in units of months), converted into data in the format in FIG. 22A, and then accumulated in the data analysis system. For this reason, when a plurality of data are aggregated and analysis processing for data for one year and analysis processing for a specific month in each year are executed, it is necessary to combine a plurality of data in the format shown in FIG. 22A. .

  For example, the data analysis system combines two pieces of data as shown in FIGS. 22A and 22B to form data as shown in FIG. 22C.

  Here, since the row data (records) of the same user ID are merged, it is necessary to perform processing equivalent to the join (join) in the database. Furthermore, in the above-described example, there are cases where not only two data to be joined but many tables are joined.

  In addition, the periodically accumulated data may have a different size distribution for each data. For example, in the data of a user whose service usage count differs from month to month, a difference in the size distribution of the data for each month occurs.

  Patent Document 1 does not describe a method for determining a position (division position) for dividing a table. Generally, in order to divide a table equally, distribution information of keys included in the table is required. When acquiring the key distribution information, the method of scanning the entire table takes time to complete the processing.

  As another method for acquiring key distribution information, there is a method using an index described in Patent Document 2. Since the index includes all key values in the table, key distribution information can be obtained by scanning the index. Since the index has a smaller data size than the table, the processing time can be shortened.

  However, when joining a large number of tables, it is necessary to scan the index by the number of tables, which increases the processing time. In addition, when there is a large amount of target data, there is a problem that it takes time to create an index when creating a table and to update an index when updating a table.

  On the other hand, it is conceivable to use the method described in Patent Document 3 without using an index. That is, it is conceivable to use a method in which tables divided into a plurality of divided areas are managed in advance, the divided areas of each table are associated with each other, and merge join processing is executed in parallel for each divided area.

  However, since the division position of the table generally differs from table to table, the divided regions cannot be associated. Even if the division positions of all the tables are matched, there is another problem that a deviation in data size occurs in each divided area when data is updated.

  That is, since the data size distribution is different for each periodically accumulated data, the data size of each divided region is biased depending on the combination of data at the division positions fixed in advance. Therefore, there is a problem in that when the joint processing is executed in parallel, a variation in processing amount occurs and parallel processing cannot be performed efficiently.

  A typical example of the invention disclosed in the present application is as follows. That is, a computer system in which a plurality of computers execute an analysis process on a data set including a plurality of data composed of keys and data values, wherein each of the computers includes a processor, a memory connected to the processor, A storage device connected to the processor; and a network interface connected to the processor, wherein each of the computers is a key indicating a division position of a divided area obtained by dividing the data set for each predetermined key range. The division information for managing a certain division position key is held for each data set, and all the division position keys included in the division information of each data set are the same, and are stored on the storage area of the plurality of computers. Includes a file system for storing the data set, and the computer system includes the analysis process. A plurality of tasks are generated for each of the divided areas, the generated tasks are allocated to the respective computers, and the analysis processing is performed by combining the data included in the divided areas of the respective data sets. When a new data set is stored in the file system, based on the data size of each divided area after the new data set is stored, the divided area having a data size larger than a predetermined threshold is used. It is determined whether or not a certain target area exists, and when it is determined that the target area exists, the target area is divided into a plurality of new divided areas.

  According to a typical embodiment of the present invention, it is possible to execute a join process between data sets in parallel without creating an index. In addition, when a new data set is added, variation in the data amount for each divided region can be suppressed, so that the processing amount between tasks that execute the combining process can be leveled.

It is a block diagram explaining the system configuration | structure of the data analysis system in the 1st Embodiment of this invention. It is a block diagram explaining the hardware constitutions of the node in the 1st Embodiment of this invention. It is a block diagram explaining the software configuration of the master node in the 1st Embodiment of this invention. It is a block diagram explaining the software configuration of the slave node in the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the data management table in the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the division | segmentation table in the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the division | segmentation table in the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the partition table in the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the key size table in the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the key size table in the 1st Embodiment of this invention. It is a flowchart explaining the combination process and analysis process of the data in the 1st Embodiment of this invention. It is a flowchart explaining the data addition process in the 1st Embodiment of this invention. It is a flowchart explaining the detail of the grouping process in the 1st Embodiment of this invention. It is a flowchart explaining the data output process in the 1st Embodiment of this invention. It is a flowchart explaining the confirmation process of the data size in the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the key size table in the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the division | segmentation table after the division | segmentation in the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the division | segmentation table after the division | segmentation in the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the key size table after a division | segmentation in the 1st Embodiment of this invention. It is explanatory drawing which shows the schema of the record in the 2nd Embodiment of this invention. It is explanatory drawing which shows an example of the record in the 2nd Embodiment of this invention. It is explanatory drawing which shows the file in the 2nd Embodiment of this invention. It is explanatory drawing which shows the file in the 2nd Embodiment of this invention. It is explanatory drawing which shows the file in the 2nd Embodiment of this invention. It is explanatory drawing which shows an example of the division | segmentation table in the 2nd Embodiment of this invention. It is explanatory drawing which shows an example of the data processed in the conventional data analysis system. It is explanatory drawing which shows an example of the schema in the conventional data. It is explanatory drawing which shows an example of the data processed in the conventional analysis process. It is explanatory drawing which shows an example of the data processed in the conventional analysis process. It is explanatory drawing which shows an example of the data processed in the conventional analysis process.

  [First Embodiment]

  Hereinafter, a first embodiment of the present invention will be described.

  FIG. 1 is a block diagram illustrating a system configuration of a data analysis system according to the first embodiment of the present invention.

  The data analysis system includes a client node 10, a master node 20, and a slave node 30, and the nodes are connected to each other via a network 40. The network 40 may be a SAN, LAN, WAN, or the like, but may be any network as long as each node can communicate. Each node may be directly connected.

  Here, the node indicates a computer. Hereinafter, the computer is referred to as a node.

  The client node 10 is a node used by a user of the data analysis system. The user transmits various instructions to the master node 20 and the slave node 30 using the client node 10.

  The master node 20 is a node that manages the entire data analysis system. The slave node 30 is a node that executes each process (task) in accordance with an instruction transmitted from the master node 20. This data analysis system is a kind of parallel distributed processing system, and the processing performance of the system can be improved by increasing the number of slave nodes 30.

  Note that the hardware configurations of the client node 10, the master node 20, and the slave node 30 are the same, and details will be described later with reference to FIG.

  Storage devices 11, 21, and 31 such as HDDs are connected to each node. Each of the storage devices 11, 21, and 31 stores a program for realizing a function included in each node such as an OS. Each program is read from the storage devices 11, 21, and 31 by the CPU (see FIG. 2) and executed by the CPU (see FIG. 2).

  FIG. 2 is a block diagram illustrating a hardware configuration of the node according to the first embodiment of this invention.

  In FIG. 2, the client node 10 is described as an example, but the master node 20 and the slave node 30 also have the same hardware configuration.

  The client node 10 includes a CPU 101, a network I / F 102, an input / output I / F 103, a memory 104, and a disk I / F 105, and the respective components are connected to each other via an internal bus or the like.

  The CPU 101 executes a program stored in the memory 104.

  The memory 104 stores a program executed by the CPU 101 and information necessary for executing the program. Note that the program stored in the memory 104 may be stored in the storage device 11. In this case, the data is read from the storage device 11 onto the memory 104 by the CPU 101.

  The network I / F 102 is an interface for connecting to other nodes via the network 40. The disk I / F 105 is an interface for connecting to the storage device 11.

  The input / output I / F 103 is an interface for connecting input / output devices such as a keyboard 106, a mouse 107, and a display 108. The user transmits an instruction to the data analysis system using the input / output device and confirms the analysis result.

  Note that the master node 20 and the slave node 30 do not have to include the keyboard 106, the mouse 107, and the display 108.

  Next, the software configuration of the master node 20 and the slave node 30 will be described.

  FIG. 3A is a block diagram illustrating a software configuration of the master node 20 according to the first embodiment of this invention.

  The master node 20 includes a data management unit 21, a process management unit 22, and a file server (master) 23.

  The data management unit 21, the process management unit 22, and the file server (master) 23 are programs stored on the memory 104 and are executed by the CPU 101. Hereinafter, when the process is described with the program as the subject, it is assumed that the CPU 101 is executing the program.

  The data management unit 21 manages data processed by the data analysis system. The data management unit 21 includes a data management table T100, a partition table T200, and a key size table T400.

  The data management table T100 stores management information on data sets processed by the data analysis system. Details of the data management table T100 will be described later with reference to FIG. Here, the data set indicates data composed of a plurality of records.

  The division table T200 stores management information of divided areas obtained by dividing the data set. Here, the divided area represents a record group in which the data set is divided for each predetermined key range. Details of the division table T200 will be described later with reference to FIG.

  The key size table T400 stores management information on the data size of each divided area in the data set. One key size table T400 corresponds to one data set. Also included is a key size table T400 that manages the data size of the data set of the entire data analysis system. Details of the key size table T400 will be described later with reference to FIG.

  The process management unit 22 manages parallel processes executed in a distributed manner on each slave node 30. The process management unit 22 includes a program repository 24 that manages a program that generates processes (tasks) to be executed in parallel. That is, the process management unit 22 generates a task to be executed in each slave node 30 from the program repository 24 and instructs the slave node 30 to execute the generated task.

  The file server (master) 23 manages a file for storing actual data.

  Note that the software configuration of the master node 20 may be realized using hardware.

  FIG. 3B is a block diagram illustrating a software configuration of the slave node 30 according to the first embodiment of this invention.

  The slave node 30 includes a process execution unit 31 and a file server (slave) 32.

  The process execution unit 31 and the file server (slave) 32 are programs stored on the memory 104 and are executed by the CPU 101. Hereinafter, when the process is described with the program as the subject, it is assumed that the CPU 101 is executing the program.

  The process execution unit 31 receives a process (task) execution instruction from the process management unit 22 of the master node 20 and executes a predetermined process (task). That is, the process execution unit 31 generates a process for executing the process (task) based on the received execution instruction of the process (task). By executing the generated process, a plurality of tasks are executed on each slave node 30, and parallel distributed processing is realized.

  The process execution unit 31 of the present embodiment includes a data addition unit (Map) 33 and a data addition unit (Reduce) 34 that execute the above-described task.

  The data adding unit (Map) 33 reads data in record units from the input raw data (see FIG. 20), and outputs the read raw data to the data adding unit (Reduce) 34 for each key range. In the data adding unit (Reduce) 34, a key range in charge of processing is set in advance.

  The data adding unit (Map) 33 includes a partition table T300. The data adding unit (Map) 33 specifies the data adding unit (Reduce) 34 that outputs the read data based on the partition table T300. The partition table T300 will be described later with reference to FIGS. 7A and 7B.

  The data adding unit (Reduce) 34 converts the input raw data into a predetermined format, that is, structured data (see FIG. 21), and outputs the structured data to the distributed file system.

  The data adding unit (Reduce) 34 includes a key size table T400. The key size table T400 is the same as the key size table T400 included in the data management unit 21. However, in the key size table T400, only the management information related to the divided area of the key range handled by the data adding unit (Reduce) 34 is stored.

  The file server (slave) 32 manages files that are distributed. The file server (master) 23 has a function of managing file metadata (directory structure, size, update date and time) and providing one file system in cooperation with the file server (slave) 32.

  The data adding unit (Map) 33 and the data adding unit (Reduce) 34 access the file server (master) 23 to use the files on the file system and execute various tasks. That is, the data adding unit (Map) 33 and the data adding unit (Reduce) 34 can access the same file system.

  Note that the software configuration of the slave node 30 may be realized using hardware.

  Next, details of each table included in the data management unit 21 will be described.

  FIG. 4 is an explanatory diagram illustrating an example of the data management table T100 according to the first embodiment of this invention.

  The data management table T100 includes a data ID (T101) and a division table name T102. The data ID (T101) stores the identifier of the data set. The division table name T102 stores the name of the division table T200 corresponding to the data set.

  Each entry of the data management table T100 corresponds to one data set managed by the data analysis system. The data set corresponds to one table (relation) in a normal database.

  5A and 5B are explanatory diagrams illustrating an example of the division table T200 according to the first embodiment of this invention.

  FIG. 5A shows an example of a partition table T200 of a data set whose partition table name T102 is “log01.part”. FIG. 5B shows an example of a partition table T200 whose partition table name T102 is “log02.part”.

  The division table T200 stores management information indicating a division method for each data set processed by the data analysis system. The division table T200 includes a division table name T201, a data file name T202, a key (T203), and an offset T204.

  The division table name T201 stores the name of the division table T200. The division table name T201 is the same as the division table name T102.

  The data file name T202 stores the name of the file that stores the data corresponding to the divided area.

  The key (T203) stores a key value indicating the key range of the divided area, that is, a key value indicating the division position of the data set. The key (T203) stores a key value representing the end point in the divided area.

  The offset T204 stores an offset corresponding to the value of the division position in the data set. The offset of the key corresponding to the key (T203) is stored in the offset T204. When the data file name T202 is different, the file in which the data is stored is different, so the offset of the corresponding entry is counted again from “0”.

  The start position of the divided area corresponds to the key (T203) and offset T204 of the previous entry. Since the key indicating the start position of the first divided area and the key indicating the end position of the last divided area are not defined, they are not described in the division table T200.

  Each entry of each division table T200 corresponds to one division area managed by the data analysis system.

  For example, the first entry of the data management table T100 shown in FIG. 4 has the division table name T101 “log01.part” and corresponds to the division table T200 shown in FIG. 5A.

  The first entry of the division table T200 shown in FIG. 5A corresponds to the first division area. The first entry indicates that the data of the divided area is stored in a file whose data file name T202 is “log01 / 001.dat”.

  Further, since the key (T203) of the first entry is “034a”, it indicates that the key range of the first divided area is less than “034a”. Further, since the offset T204 of the first entry is “280”, it indicates that the data of the first divided area is stored in the range of the offset on the file “0 to 279”.

  In the second entry of the division table T200 shown in FIG. 5A, the key range of the corresponding division area is “034a” or more and less than “172d”, and the data file name T202 is “log01 / 002.dat”. Indicates that there is. Further, since the data file name T202 is different from the first entry, the offset is counted from “0”. Therefore, it indicates that the data of the divided area corresponding to the range where the offset is “0 to 218” is stored.

  In the third entry of the division table T200 shown in FIG. 5A, the key range of the corresponding division area is “172d” or more and less than “328b”, and the data file name T202 is “log01 / 002.dat”. Indicates that there is. Further, since the data file name T202 matches the second entry, it indicates that the data of the divided area corresponding to the range where the offset on the file is “219 to 455” is stored.

  Also, the second entry of the data management table T100 shown in FIG. 4 has the division table name T101 “log02.part” and corresponds to the division table T200 shown in FIG. 5B.

  The data file name T202 and offset T204 of each entry stored in the division table T200 shown in FIG. 5B are different from the entries in the division table T200 shown in FIG. 5A. However, the keys (T203) representing the division positions of both division tables T200 coincide with each other.

  In the present embodiment, division positions of divided areas in a data set that can be combined, that is, keys (T203) are managed so as to always match. As a result, two or more data sets can be combined in parallel. That is, it is possible to associate entries with the same key (T203) in the partition table T200 of the data set to be combined, and it is possible to execute the combining process in parallel for each divided region.

  The file includes a plurality of records composed of one key and one or more values as shown in FIG. 22A. Each file is stored in the distributed file system in a format sorted based on the key. As a result, when combining processing is performed for each divided region, it becomes possible to merge and combine the same keys.

  Also, the files that store the data of different divided areas may be the same. For example, in FIG. 5A, the second entry and the third entry are the same file. However, the key range of each entry is different.

  As described above, in FIG. 5A, the number of files is three, but the number of divided areas is four, which are different from each other. As will be described later, the number of files matches the degree of parallelism of data addition processing in the data analysis system. On the other hand, the number of divided areas depends on the parallelism of data analysis processing. Accordingly, since the number of files and the number of divided areas depend on different processes, there is no dependency between them, and the number may be determined in any way.

  FIG. 6 is an explanatory diagram illustrating an example of the partition table T300 according to the first embodiment of this invention.

  The partition table T300 divides a newly added data set (raw data), and stores information used for distributing the data in a data adding unit (Reduce) 34 that executes a task. The partition table T300 includes a key (T301) and a destination T302.

  The key (T301) stores a key value representing the division position of the input data set. The destination T302 stores destination information indicating the position of the data adding unit (Reduce) 34 in charge of processing the divided data set. In the example illustrated in FIG. 6, the node and the data addition unit (Reduce) 34 are specified by destination information including an IP address and a port.

  7A and 7B are explanatory diagrams illustrating an example of the key size table T400 according to the first embodiment of this invention.

  The key size table T400 stores the data size of the divided area. The key size table T400 includes a key (T401) and a size T402.

  The key (T401) is the same as the key (T203). The size T402 stores the data size of the divided area with the key (T401) as the division position.

  The size T402 stores the total value of the data sizes of the divided areas to be combined.

  The key size table T400 is dynamically generated when a combination process and an analysis process, which will be described later, and a data addition process are executed.

  Next, data combination processing and analysis processing will be described.

  FIG. 8 is a flowchart for explaining data combination processing and analysis processing according to the first embodiment of the present invention.

  The combination process is always executed together with the analysis process. That is, after the data for one record is combined by the combining process, the analysis process is executed on the data.

  The combination process and the analysis process are executed by the data management unit 21 that has received an instruction from the user. The instruction from the user includes the data ID of the data set to be combined.

  First, the master node 20 creates a key size table T400 corresponding to a data set to be processed (step S101).

  Specifically, the following processing is executed.

  The data management unit 21 searches the data management table T100 based on the data ID included in the instruction transmitted from the user, and acquires the division table name T102 from the corresponding entry.

  Next, the data management unit 21 acquires a partition table T200 corresponding to the acquired partition table name T102.

  Based on the obtained division table T200, the data management unit 21 specifies a key value indicating the division position for each division area, and calculates the data size of the data set to be combined.

  Further, the data management unit 21 creates the key size table T400 based on the above processing result.

  For example, when data sets whose data IDs (T101) are “log01” and “log02” are combined, the corresponding division tables T200 are as shown in FIGS. 5A and 5B, respectively. At this time, the data management section 21 adds the data sizes of the two data sets for each divided region by executing the above-described processing, and creates a key size table T400 as shown in FIG. 7A.

  Next, the master node 20 generates a plurality of tasks including a combination process and an analysis process, and activates each task by assigning each generated task to each slave node 30 (step S102).

  Specifically, the process management unit 22 reads a program necessary for the process from the program repository 24 and generates tasks for the number of parallels designated by the user. Further, the process management unit 22 causes the generated task to be executed on each slave node 30.

  When the parallel number is smaller than the number of entries in the key size table T400 created in step S101, the number of entries is set as the parallel number, and tasks corresponding to the number of entries are executed on the slave node 30.

  Next, the master node 20 assigns a divided area to each task (step S103).

  Specifically, the data management unit 21 assigns a divided area corresponding to each entry of the key size table T400 created in step S101 to each task generated in step S102.

  Note that the data management unit 21 allocates a divided area to each task so that the data sizes are equal based on the size T402 of the key size table T400.

  For example, the data management unit 21 sorts the entries in the key size table T400 based on the size T402, and the task having the smaller allocated data size is performed in order from the larger data size. An allocation method is conceivable.

  After the allocation of the divided areas is completed, the data management unit 21 transmits the data file name and the offset position of the files to be combined to the slave node 30 to which the task is allocated.

  For example, in the case of a task to which a divided area corresponding to the first entry in the key size table T400 in FIG. 7A is assigned, the corresponding entry in the divided table T200 is the first entry in each of FIGS. 5A and 5B. is there. Therefore, the data management unit 21 assigns (data file name, start position, end position) = (log 01 / 001.dat, 0, 280), (log 02 / 001.dat, 0, 200) to the task. To the slave node 30.

  Next, the master node 20 transmits a task execution instruction to each slave node 30 to which the task is assigned, and ends the process (step S104).

  Specifically, the data management unit 21 transmits a task execution instruction to each slave node 30 to which the task is assigned.

  The slave node 30 that has received the instruction from the master node 20 accesses the file server (master) 23 and, based on the data file name and offset position received from the data management unit 21, designates the designated file with the designated offset. Read from position.

  Each slave node 30 adds the keys of the read files and executes a combination process. Further, the slave node 30 outputs the result of the combination processing for each record to the analysis processing task being executed in the same slave node 30.

  For example, in the analysis process for the data set corresponding to FIGS. 5A and 5B, a task is generated for each of the four divided regions, and the above-described combining process is executed by each task.

  At this time, if the division position is different for each data set, parallel processing cannot be realized because the processing is executed for the overlapping key ranges. However, in this embodiment, since the division positions of the respective data sets are the same, it is possible to execute the combination processing in the divided areas of the respective data sets in parallel.

  The above is the description of the data combination processing and analysis processing.

  Next, data addition processing will be described.

  The data addition process is a process for adding a new data set when an existing data set is stored in the data set in which the data management table T100 and the partition table T200 are created, that is, the distributed file system. .

  Usually, the data size of each divided area differs for each data set. For this reason, if the divided areas of the respective data sets are combined without correcting the dividing position, the data size varies between the divided areas. As a result, the processing amount of the task that executes the analysis processing varies, and the efficiency of parallel processing decreases.

  In the present invention, in order to solve the above-described problem, a process described later is executed during the data addition process, so that the divided areas are subdivided and the data size of each divided area is leveled.

  Specifically, after a new data set is added, the division position is controlled so that the data size of each divided area is equal to or less than a predetermined reference value when all data sets that can be combined are combined. As a result, it is possible to equalize the difference in processing amount between tasks of analysis processing executed in parallel when all data sets are used.

  When a part of the data sets is combined, the data size of each divided area is equal to or smaller than the reference value, and the difference in processing amount between tasks of the analysis processing is leveled.

  If the divided area is subdivided and task control overhead of the join processing and analysis processing occurs, and if the allocated divided area becomes smaller, multiple tasks are assigned to the task to which the divided area is assigned. Areas are allocated and the amount of processing executed by one task can be increased.

  Note that the above-described predetermined reference value affects the difference in task throughput, so it is desirable to determine the predetermined reference value based on the allowable task throughput difference.

  If the reference value is made too small, the number of divided areas increases, which increases the overhead of data addition processing. On the other hand, if the reference value is increased too much, the difference in processing amount between tasks increases, and the efficiency of parallel processing decreases.

  Accordingly, the predetermined reference value may be a data amount such that an execution time when one task processes a predetermined data amount is equal to or less than a time allowed as a difference in processing time between tasks.

  Data added in the data addition process is input in a format as shown in FIG. In the data addition process, data in the format as shown in FIG. 22A is converted into a format grouped by user ID and stored in the distributed file system. Hereinafter, the data set in the format shown in FIG. 20 is referred to as raw data, and the data set in FIG. 21 is referred to as structured data.

  Hereinafter, the processing will be specifically described with reference to FIG.

  FIG. 9 is a flowchart for explaining data addition processing in the first embodiment of the present invention.

  The user adds raw data to the distributed file system realized by the file server (master) 23 and the file server (slave) 32, thereby executing data addition processing.

  First, the data management unit 21 samples the input raw data and analyzes the appearance frequency of the key (step S201).

  Specifically, the data management unit 21 samples a record included in the raw data at random. The data management unit 21 creates a key list in which the first field of the read record is the key.

  Since the raw data is composed of data in a format in which one record is one line, the data management unit 21 can read data for one record by detecting a line feed code.

  When increasing the number of samplings in order to improve accuracy, the data management unit 21 may execute the sampling processes in parallel. In this case, the data management unit 21 divides the raw data into a plurality of pieces so that the data sizes are equal, and a sampling process is executed for each divided raw data.

  Specifically, the data management unit 21 assigns the execution task of the sampling process to each slave node 30, and further assigns the raw data divided into the execution task. The data management unit 21 receives the result of the sampling process from the process execution unit 31 of each slave node 30 and totals the results of the sampling process received from all the slave nodes 30 to create a key list.

  Next, the data management unit 21 determines the value of the key that becomes the division position of the raw data based on the created list of keys (step S202).

  The division process is a division process for outputting input raw data in step S204 described later, and is a process different from the division process in the division table T200.

  However, the existing division position is not changed in the process of step S204. Therefore, the division position of the raw data needs to match the division position of the division table T200 of the existing data set.

  Specifically, the following processing is executed.

  The data management unit 21 refers to the division table T200 and creates a key size table T400 including division positions of all existing data sets. For example, a key size table T400 as shown in FIG. 7A is created. However, at this time, no value is stored in the size T402.

  The data management unit 21 specifies a divided area corresponding to each sampled key, and increments the data size of the data corresponding to the key to the size T402 of the corresponding entry in the key size table T400.

  Through the processing as described above, the data management unit 21 can obtain the distribution of the sampled keys.

  For example, when the sampled key is “125d”, since the key is “034a” or more and less than “172d”, the key is “125d” in the size T402 of the entry whose key (T401) is “172d”. Is incremented.

  After obtaining the key distribution, the data management unit 21 merges adjacent divided areas of the key size table T400 so that the parallel number specified by the user matches the number of divided areas. At this time, it is desirable that the data size of each divided area after merging is equal.

  For example, when the number of parallels specified by the user is “2”, the key size table T400 whose key distribution is shown in FIG. 7B has four divided areas, so it is necessary to merge them into two divided areas. There is. Therefore, the data management unit 21 merges the entry whose key (T401) is “034a” and the entry “172d” into one divided area, and combines the entry whose key (T401) is “328b” and the blank entry. Merge as one divided area.

  After the merge process ends, the data management unit 21 stores the merge result in the key (T301) of the partition table T300.

  In the merge process described above, when the number of entries in the key size table T400 is equal to or larger than the parallel number specified by the user, the merge process is not executed and the number of entries is the parallel number.

  The above is the process in step S202.

  Next, the data management unit 21 calculates the data size of all data sets that may be combined in the analysis process (step S203). Further, the data management unit 21 creates a key size table T400 based on the calculation result.

  Specifically, the following processing is executed.

  The data management unit 21 refers to the data management table T100 and acquires the division table name T102 of each data set. Furthermore, the data management unit 21 acquires a list of the corresponding partition table T200 based on the acquired partition table name T102.

  Note that the division positions in the division table T200 of the data sets that can be combined are the same. Therefore, it is possible to execute the combination of the divided areas in parallel in the analysis process.

  The data management unit 21 creates a key size table T400 including the key (T203) of the acquired division table T200. Further, the data management unit 21 calculates the data size of each divided area for each division table T200, and adds the calculated data size to the size (T402) of the created key size table T400.

  By executing the same processing for all the obtained divided tables T200, the key size table T400 regarding all existing data sets existing on the distributed file system can be created.

  For example, the key size table T400 as shown in FIG. 7A is created by executing the above-described processing on the division table T200 shown in FIGS. 5A and 5B.

  The above is the process in step S203.

  Next, the data management unit 21 performs a grouping process on the raw data based on the partition table T300 representing the merge result in step S202 (step S204).

  Here, the grouping process is a process of collecting records included in the raw data for each key (user ID in the example shown in FIG. 20).

  In the grouping process, the data management unit 21, the data addition unit (Map) 33, and the data addition unit (Reduce) 34 cooperate to execute the process.

  The data adding unit (Map) 33 and the data adding unit (Reduce) 34 each execute parallel processing in accordance with an instruction from the data management unit 21.

  Note that the number of entries in the partition table T300 is the degree of parallelism of the data adding unit (Reduce) 34 to which tasks are assigned. On the other hand, the degree of parallelism of the data adding unit (Map) 33 to which tasks are assigned is irrelevant to the number of entries in the partition table T300 and is specified by the user.

  Hereinafter, the data adding unit (Map) 33 is referred to as a Map task, and a task assigned to the data adding unit (Reduce) 34 is also referred to as a Reduce task.

  Specifically, the following processing is executed.

  The data management unit 21 divides the raw data according to the parallel number specified by the user so that the data size is constant. Further, the data management unit 21 calculates an offset position, which is each division position of the division area generated by dividing the raw data, and a data size of the division area. The offset position is adjusted by scanning a part of the raw data so as to coincide with the record boundary.

  The data management unit 21 generates Map tasks for the number of parallels designated by the user in cooperation with the process management unit 22 and assigns the generated Map tasks to each data addition unit (Map) 33. At this time, each data adding unit (Map) 33 is transmitted with the offset position of the divided area, the data size of the divided area, and the file name of the raw data.

  Furthermore, the data management unit 21 cooperates with the process management unit 22 to generate Reduce tasks for the number of entries in the partition table T300.

  In addition, the data management unit 21 associates each entry of the partition table T300 with the data addition unit (Reduce) 34. The data management unit 21 assigns a Reduce task for processing a divided region in the key range corresponding to the key (T301) to each associated data addition unit (Reduce) 34.

  Further, the data management unit 21 transmits the entry corresponding to the transmitted key range in the key size table T400 created in step S202, to the data addition unit (Reduce) 34.

  For example, since the key range of the first entry of the partition table T300 shown in FIG. 6 is less than “172d”, the corresponding entry of the key size table T400 includes the first entry and the second entry of FIG. 7A. It is. Therefore, the data management unit 21 transmits the first entry and the second entry to the corresponding data addition unit (Reduce) 34.

  Further, the data management unit 21 acquires the destination information (address: port number) of the data adding unit (Reduce) 34 and stores the acquired destination information in the destination T302 of the corresponding entry of the partition table T300.

  After the partition table T300 is created, the process management unit 22 transmits the completed partition table T300 to all the data addition units (Map) 33.

  The above is the processing in step S204.

  In step S204, the data adding unit (Map) 33 and the data adding unit (Reduce) 34 execute the data output process after the grouping process is executed. Details of the grouping process will be described later with reference to FIG. 10, and details of the data output process will be described later with reference to FIG.

  The data management unit 21 updates the division table T200 and ends the process (step S205).

  Specifically, the data management unit 21 updates the division table T200 managed by itself based on the division table T200 received from each data addition unit (Reduce) 34. The received division table T200 is a table after the data adding unit (Reduce) 34 has executed processing (see FIGS. 10 and 11) described later.

  The data adding unit (Reduce) 34 processes only a data set in a partial key range. The present embodiment is characterized in that all the division tables T200 in the data analysis system are updated based on the division table T200 updated by one data addition unit (Reduce) 34.

  Further, the data management unit 21 merges the input raw data division table T200 received from each data addition unit (Reduce) 34 into one, and the merged table is input to the raw data division table T200. Manage as.

  This is a process of collecting the processing results for each key range because the processing for the raw data is performed in parallel in each data adding unit (Reduce) 34.

  Further, the data management unit 21 adds an entry corresponding to the raw data division table T200 to the data management table T100.

  Next, details of the grouping process in step S204 will be described.

  FIG. 10 is a flowchart illustrating details of grouping processing according to the first embodiment of this invention.

  The slave node 30 performs a sort process on the input raw data (step S301).

  Specifically, the following processing is executed.

  The data adding unit (Map) 33 reads records one by one from the raw data. The data adding unit (Map) 33 acquires the destination information of the data adding unit (Reduce) 34 from the partition table T300 based on the key of the read record. That is, the data adding unit (Reduce) 34 for processing the read record is specified.

  The data adding unit (Map) 33 classifies each record read for each destination. Hereinafter, the record group classified for each destination is referred to as a segment.

  The data adding unit (Map) 33 reads all the records included in the divided raw data handled by itself, and then sorts the records included in each segment based on the key.

  The above is the process in step S301.

  Next, the slave node 30 transmits the sorted segments to the data adding unit (Reduce) 34 (step S302).

  Specifically, the data adding unit (Map) 33 transmits the sorted segments to the data adding unit (Reduce) 34 corresponding to the destination information acquired in Step S301. Each data adding unit (Reduce) 34 receives a segment transmitted from the data adding unit (Map) 33 of each slave node 30.

  The slave node 30 that has received the segment from the data adding unit (Map) 33 merges the received segment based on the key, and ends the process (step S303).

  Specifically, the data adding unit (Reduce) 34 sequentially reads all received segments, and merges and joins the segments having the same key.

  Further, the data adding unit (Reduce) 34 converts the records included in the merged segment into structured data as shown in FIG. By the process described above, a plurality of records are collected into one record having the same key.

  Next, a data output process executed by the data adding unit (Reduce) 34 in step S204 will be described.

  FIG. 11 is a flowchart for explaining data output processing according to the first embodiment of the present invention.

  First, the outline of the data output process will be described.

  The data adding unit (Reduce) 34 outputs structured data having a format shown in FIG. 22A to the distributed file system by executing a data output process. Tasks are executed in the data adding unit (Reduce) by the number of parallelism. At this time, the file names output by the data adding unit (Reduce) 34 are different.

  Furthermore, in the present invention, the data adding unit (Reduce) 34 adds the data size of the raw data to the key size table T400, and calculates the data size of each divided area after the raw data is added.

  The data adding unit (Reduce) 34 executes a division process of the divided area when there is a divided area having a data size equal to or larger than a predetermined threshold.

  The data adding unit (Reduce) 34 also updates the division table T200 of the existing data set managed by itself when the division process of the division area is executed. Further, the data adding unit (Reduce) 34 transmits the updated division table T200 to the data management unit 21. Based on the updated division table T200, the data management unit 21 executes an update process (step S205) of the division table T200.

  In addition, the data adding unit (Reduce) 34 creates a division table T200 of the input raw data, and transmits the division table T200 created after the processing is completed to the data management unit 21.

  Details of each process will be described below.

  First, before starting the data output process, the data adding unit (Reduce) 34 creates a key size table T400 in which only the key included in the key size table T400 received from the data management unit 21 in step S204 is stored. . Here, the created key size table T400 is a table in which the data size of a predetermined divided area of the raw data is stored.

  Hereinafter, the created key size table T400 is also referred to as an additional key size table T400. At the time when the additional key size table T400 is created, the initial value of the size T402 is set to “0”.

  The key size table T400 received from the data management unit 21 is a table for managing the data sizes of all data sets on the distributed file system included in the key range handled by the data addition unit (Reduce) 34. Hereinafter, the key size table T400 is referred to as an all data key size table T400.

  When the data output process is started, the data adding unit (Reduce) 34 outputs the record created in step S303, and determines whether or not the record is included in a divided area different from the record output last time. (Step S401).

  Specifically, the data adding unit (Reduce) 34 refers to the key (T402) of the key size table T400 for addition, and determines whether the output record is included in a different divided area from the record output last time. judge.

  In this embodiment, since the records sorted based on the keys are output in order, it can be determined whether or not the output records are included in a predetermined key range, that is, a predetermined divided area.

  In the case of the first output record, it is determined that the record is included in the same divided area.

  When it is determined that the record is included in a different divided area, the data adding unit (Reduce) 34 executes a process for confirming the data size of the divided area to which the previous record has been added (step S405), and proceeds to step S402. . The data size confirmation process will be described later with reference to FIG.

  When it is determined that the records are included in the same divided area, the data adding unit (Reduce) 34 writes the record created in step S303 to the distributed file system (step S402).

  At this time, the data adding unit (Reduce) 34 creates record statistical information including the key value of the written record, the offset position on the file in which the record is written, and the data size of the record, and the created record Save statistical information. This is record statistical information of raw data.

  Next, the data adding unit (Reduce) 34 updates the key size table T400 (step S403).

  Specifically, the data adding unit (Reduce) 34 specifies a divided area of the key range including the key of the record written in step S402. The data adding unit (Reduce) 34 searches the addition key size table T400 and the total data key size table T400 for an entry corresponding to the specified divided area. Further, the data adding unit (Reduce) 34 adds the data size of the written record to the size T402 of the corresponding entry in each key size table T400.

  The data adding unit (Reduce) 34 determines whether all records have been output (step S404).

  If it is determined that all the records have not been output, the data adding unit (Reduce) 34 returns to Step S401 and executes the same processing.

  If it is determined that all the records have been output, the data adding unit (Reduce) 34 executes a data size confirmation process for the last divided area, and ends the process (step S406). Note that the data size confirmation processing in step S406 is the same processing as step S405.

  FIG. 12 is a flowchart for explaining data size confirmation processing according to the first embodiment of this invention.

  The data adding unit (Reduce) 34 refers to the entire data key size table T400 updated in step S403, and determines whether or not the data size of the target divided area is larger than a predetermined reference value (step S501). . That is, it is determined whether or not the divided area to which the raw data is added is larger than a predetermined reference value.

  Here, the target divided area is a divided area including a previously input record. Hereinafter, the target divided area is also referred to as a target area.

  Specifically, the data adding unit (Reduce) 34 refers to the size T402 of the corresponding entry in the all data key size table T400, and determines whether or not the data size of the target area is larger than a predetermined reference value.

  If it is determined that the data size of the target area is equal to or smaller than the predetermined reference value, the data adding unit (Reduce) 34 proceeds to step S506.

  When it is determined that the data size of the target area is larger than the predetermined reference value, the data adding unit (Reduce) 34 acquires the existing data set division table T200 from the master node 20 (step S502).

  Here, all the division tables T200 acquired by the master node 20 in step S203 are acquired. The data adding unit (Reduce) 34 may store the division table T200 acquired from the master node 20 as a cache.

  Next, the data adding unit (Reduce) 34 specifies the end position of the target region in each acquired division table T200, that is, the offset (step S503).

  Specifically, the following processing is executed.

  The data adding unit (Reduce) 34 refers to each obtained division table T200 based on the key of the target area, and acquires an entry corresponding to the target area. That is, the data file name T202 and the offset T204 of the data corresponding to the target area are acquired. This process is executed for all the divided tables T200 acquired in step S502.

  For example, if the total data key size table T400 as shown in FIG. 13 in step S501 and the data size of the divided area corresponding to the first entry is larger than a predetermined reference value, the data adding unit (Reduce) 34 Information is acquired from the first entry of the division table T200 shown in 5A and 5B.

  In this case, (data file name, offset) = (/ log01 / 001.dat, 280) in FIG. 5A, and (/log02/002.dat, 200) in FIG. 5B. The acquired offset is the end position of the target area in each division table T200.

  Note that since the start position of the target area is the first entry, the offset of the start position is “0”.

  Next, the data adding unit (Reduce) 34 analyzes the records included in the target area of each existing data set (step S504).

  Specifically, the data adding unit (Reduce) 34 reads a record included in the target area of each existing data set. For example, when there is a data set whose data ID (T101) is “log01” and “log02”, a record is read from the target area of the data set of “log01”, and the target area of the data set of “log02” Records are read from.

  The data adding unit (Reduce) 34 acquires record statistical information including the key of the read record, the data size of the record, and the offset position of the record on the file.

  Since there are a plurality of existing data sets, the analysis processing of the record may be executed in parallel for each data set.

  The data adding unit (Reduce) 34 combines the record statistical information of the raw data acquired in step S402 and the record statistical information of the existing data set to obtain record statistical information of all data sets on the distributed file system.

  Next, the data adding unit (Reduce) 34 determines the value of the key to be a division position for re-division based on the record statistical information of all created data sets (step S505).

  Specifically, the following processing is executed.

  The data adding unit (Reduce) 34 calculates the data size in the target area based on the record statistical information of all data sets.

  The data adding unit (Reduce) 34 calculates the number of divisions in the target area based on the calculated data size and a predetermined reference value.

  Next, the data adding unit (Reduce) 34 divides the data size of the target area by the calculated number of divisions, and calculates the data size of the divided areas after re-division.

  The data adding unit (Reduce) 34 calculates the cumulative value distribution of the data size of the records after sorting the records statistical information entries of all data sets by key. That is, the distribution of the data size of each record included in the predetermined key range in the distributed file system is calculated.

  Based on the calculated cumulative value distribution, the data adding unit (Reduce) 34 determines a point where the data size of the record is an integral multiple of the data size of the divided area after division as a division position for re-division. If it is not an integer multiple, the record closest to the data size is determined as the division position.

  As the key of the subdivision position, a key that exists as data may be used, or a key that does not exist as data may be used.

  The data adding unit (Reduce) 34 specifies the offset corresponding to each determined key range with reference to the record statistical information of all data sets.

  The data adding unit (Reduce) 34 adds an entry corresponding to the divided area after re-division to each division table T200. Further, the data adding unit (Reduce) 34 deletes the entry corresponding to the divided area before the re-division from each division table T200.

  For example, a divided region whose key range is less than “034a” is divided into two divided regions: a divided region whose key range is less than “015d” and a divided region whose key range is “015d” or more and less than “034a”. 5A and 5B, the division table T200 shown in FIGS. 5A and 5B is changed as shown in FIGS. 14A and 14B. A portion indicated by a thick line in the figure is a changed portion.

  The data adding unit (Reduce) 34 also changes the adding key size table T400 and the total data key size table T400 based on the record statistical information.

  For example, when the total data key size table T400 before re-division is the table shown in FIG. 13, the table is changed as shown in FIG. A portion indicated by a thick line in the figure is a changed portion.

  The above is the process of step S505.

  Next, the data adding unit (Reduce) 34 updates the division table T200 (step S506).

  Specifically, the data adding unit (Reduce) 34 stores the entry of the divided region corresponding to the raw data division table T200 based on the key size table for addition and the record statistical information of the raw data. That is, the raw data division table T200 is generated.

  When the re-division process is executed, an entry corresponding to the newly divided division area is stored.

  The data adding unit (Reduce) 34 deletes the record statistical information used in the above-described processing, and ends the processing (step S507).

  [Second Embodiment]

  In the first embodiment, since the contents of the file are stored in one file, unnecessary data may be read during the analysis process. On the other hand, in the second embodiment, a method of saving as a different file for each data item (column) is used. By using this method, it is possible to read out only the items necessary for the analysis process.

  The present invention can also support a storage method (column division storage method) in which data items are stored in different files.

  Hereinafter, the second embodiment will be described focusing on differences from the first embodiment.

  In the second embodiment, since the configuration of the data analysis system is the same as that of the first embodiment, the description thereof is omitted. Further, the hardware configuration and software configuration of the master node 20 and the slave node 30 are the same as those in the first embodiment, and thus description thereof is omitted.

  FIG. 16 is an explanatory diagram illustrating a record schema according to the second embodiment of this invention. FIG. 17 is an explanatory diagram illustrating an example of a record according to the second embodiment of this invention.

  In contrast to the record of the first embodiment, the record of the second embodiment newly includes the user's age.

  There are three types of record items: user ID, movement history (location X, location Y, timestamp history), and age. In this embodiment, the user ID is used as the key.

  18A, 18B, and 18C are explanatory diagrams showing files in the second embodiment of the present invention.

  18A, 18B, and 18C show an example in which the above-described data is stored in a file using the column division method.

  As shown in FIGS. 18A, 18B, and 18C, the user ID is log / 001. key. dat (FIG. 18A), the movement history is log / 001. rec. dat (FIG. 18B), age is log / 001. age. Each is stored in a file called dat (FIG. 18C).

  When data is read, the records are read one by one from the top of each file, and the entire record shown in FIG. 17 can be reconstructed by combining them in order.

  In the examples shown in FIGS. 18A, 18B, and 18C, there is only one set of files. However, a data set that includes files corresponding to user ID, movement history, and age by periodically accumulating data. Will increase.

  Since the actual combining process and analysis process are executed in parallel, the process is executed by each slave node 30 after the file is divided.

  FIG. 19 is an explanatory diagram illustrating an example of a division table T200 according to the second embodiment of this invention.

  The division table T200 in the second embodiment is different from the first embodiment in that the data file name T202 and the offset T204 are stored for each item (user ID, movement history, and age). In the item used as the key, a key value representing a division position is stored in the key (T203).

  Next, the combination process and the analysis process in the second embodiment will be described focusing on the differences from the first embodiment.

  In step S101, when the key size table T400 is created, the data management unit 21 calculates the size of each divided region with reference to the offset of the item used for analysis processing in the divided table T200.

  For example, when an analysis using only the user ID and age is performed, the size of the key size table is obtained using only the “uid” offset and the “age” offset. At this time, the offset for “rec” is not used.

  Thereby, even when only some items are used, the data size of each divided region can be accurately calculated.

  In S104, each slave node 30 to which a task is assigned reads files as many as the product of the number of files used for analysis processing and the number of items used for analysis processing.

  The data addition process also has the following differences.

  In step S203, the data management unit 21 creates the key size table T400 of the existing data set from the offset for each item of the division table T200 of all data sets that may be combined.

  In step S402, when each record is output to a file, each record is output to a separate file. Therefore, in step S402, record statistical information including the key value of the written record, the offset on the written file, and the data size is stored for each item.

  In step S403, the sum of the sizes of the divided areas of all items is added to the corresponding entry in the key size table T400.

  In S506, using the record statistical information and the key size table T400 described above, the offset value of the division position is obtained for each item, and the division table T200 is updated.

  In S504, a file corresponding to all items included in the data is read, and for each item, record statistical information including the key value of the record written in the file, the offset position on the written file, and the data size is obtained. Saved.

  In S505, the data adding unit (Reduce) 34 determines the key of the division position using the sum of the data sizes of the divided areas of all items as the data size of the data set.

  In S506, the data adding unit (Reduce) 34 calculates the offset of the division position for each item using the determined key and record statistical information, and updates the division table T200.

  In the second embodiment, the case where three items are processed has been described. However, the number of items managed in the division table T200 can be changed to any number of items.

  According to an aspect of the present invention, the data analysis system can execute the combination processing in the analysis processing in parallel because the division positions of the respective data sets are the same. In addition, when a data set is newly added, the divided area can be subdivided so that the processing amount between tasks becomes uniform. As a result, it is possible to eliminate processing imbalance between tasks and to combine records for each distributed area during the combining process.

  Although the present invention has been described in detail with reference to the accompanying drawings, the present invention is not limited to such specific configurations, and various modifications and equivalents within the spirit of the appended claims Includes configuration.

Claims (16)

A computer system in which a plurality of computers execute analysis processing on a data set including a plurality of data composed of keys and data values,
Each of the computers includes a processor, a memory connected to the processor, a storage device connected to the processor, and a network interface connected to the processor.
Each of the computers holds, for each data set, division information for managing a division position key, which is a key indicating a division position of a division area obtained by dividing the data set for each predetermined key range,
All the division position keys included in the division information of each data set are the same,
A file system for storing the data set is configured on a storage area of the plurality of computers.
The computer system is
When executing the analysis process, generate a plurality of tasks for each of the divided areas,
Assigning the generated task to each computer, combining the data included in the divided areas of each data set, and executing the analysis process;
When a new data set is stored in the file system, based on the data size of each divided area after the new data set is stored, the target area that is the divided area having a data size larger than a predetermined threshold is Determine if it exists,
A computer system that divides the target area into a plurality of new divided areas when it is determined that the target area exists. When storing a new data set in the file system, the key distribution of the new data set is analyzed,
2. The division information of the new data set is generated based on the analysis result so as to be the same as all the division position keys included in the division information of the existing data set. The computer system described in 1.   The computer system according to claim 2, wherein the division position key in the division information of the existing data set is updated after the target area is divided. When determining whether or not the target area exists, the data sizes of the divided areas of all the data sets are summed up to obtain a first data size that is the data size of the divided areas in the computer system. Calculate
It is determined whether or not there is the divided area in which the calculated first data size is larger than the predetermined threshold;
When dividing the target area, by calculating the second data size, which is the data size of the target area in the computer system, by summing the data sizes of the target areas of all the data sets,
Based on the predetermined threshold and the calculated second data size, calculate the number of divisions of the target area,
Based on the calculated number of divisions, determine a new division position key in the target area,
When updating the division position key of the division information of the existing data set, the information corresponding to the target area is deleted from the division information of the existing data set, and the determined division position key and the new division key are updated. Add information that correlates with the various divided areas,
The division information of the new data set is generated so as to be the same as the division key in the updated division information of the existing data set when the division information of the new data set is generated. Item 4. The computer system according to Item 3. When dividing the target area, calculate the third data size by dividing the data size of the target area by the calculated number of divisions,
5. The computer system according to claim 4, wherein the key in the data corresponding to the calculated third data size is determined as the division position key.   5. The computer system according to claim 4, wherein the predetermined threshold is a data size that makes a processing time of a task to which the new divided area is allocated be equal to or less than a preset allowable time. The data includes data values for a plurality of items,
5. The computer system according to claim 4, wherein, when calculating the first data size, the first data size is calculated by summing up data sizes of all items in the divided area. 6. . When analyzing the key distribution of the new data set, the new data set is divided by a division position key that matches one of the division position keys included in the division information of the existing data set, and is used for a plurality of processes. Generate segmented areas,
3. The computer system according to claim 2, wherein a task for analyzing a key distribution of the new data set is generated for each of the generated processing divided areas, and the task is executed in parallel. A data management method in a computer system in which a plurality of computers execute analysis processing on a data set including a plurality of data composed of keys and data values,
Each of the computers includes a processor, a memory connected to the processor, a storage device connected to the processor, and a network interface connected to the processor.
Each of the computers holds, for each data set, division information for managing a division position key, which is a key indicating a division position of a division area obtained by dividing the data set for each predetermined key range,
All the division position keys included in the division information of each data set are the same,
A file system for storing the data set is configured on a storage area of the plurality of computers.
The method
A first step of generating a plurality of tasks for each of the divided regions when at least one of the computers executes the analysis process;
A second step in which the computer that has generated the task assigns the generated task to each of the computers and combines the data included in the divided regions of each of the data sets to execute the analysis process; Including
When at least one of the computers stores a new data set in the file system, based on the data size of each divided area after the new data set is stored, the data size larger than a predetermined threshold is used. A third step of determining whether or not a target area that is a divided area exists;
A fourth step of dividing the target area into a plurality of new divided areas when the computer that has performed the above-described determination process determines that the target area exists;
A data management method comprising: The third step includes
A fifth step of analyzing a key distribution of the new data set;
And a sixth step of generating the division information of the new data set so as to be the same as all the division position keys included in the division information of the existing data set based on the analysis result. The data management method according to claim 9.   The data according to claim 10, wherein the fourth step includes a seventh step of updating the division position key in the division information of the existing data set after the target region is divided. Management method. The third step includes
An eighth step of calculating a first data size that is a data size of the divided area in the computer system by summing data sizes of the divided areas of all the data sets;
A ninth step of determining whether or not the divided region having the calculated first data size larger than the predetermined threshold exists;
The fourth step includes
A tenth step of calculating a second data size that is a data size of the target area in the computer system by summing up the data sizes of the target areas of all the data sets;
An eleventh step of calculating the number of divisions of the target area based on the predetermined threshold and the calculated second data size;
A twelfth step of determining a new division position key in the target area based on the calculated division number;
In the seventh step, information corresponding to the target area is deleted from the division information of the existing data set, and information in which the determined division position key and the new division area are associated is added. Including a thirteenth step,
The sixth step includes a fourteenth step of generating division information of the new data set so as to be the same as the division key in the updated division information of the existing data set. Item 12. The data management method according to Item 11. The twelfth step includes
Dividing the data size of the target area by the calculated number of divisions to calculate a third data size;
The data management method according to claim 12, further comprising: determining the key in the data corresponding to the calculated third data size as the division position key.   13. The data management method according to claim 12, wherein the predetermined threshold is a data size that makes a processing time of a task to which the new divided area is allocated be equal to or less than a preset allowable time. The data includes data values for a plurality of items,
13. The data management method according to claim 12, wherein in the eighth step, the first data size is calculated by summing up the data sizes of all items in the divided area. The fifth step includes
Dividing the new data set with a division position key that matches one of the division position keys included in the division information of the existing data set to generate a plurality of processing division areas;
Generating a task for analyzing the key distribution of the new data set for each of the generated processing divided regions, and executing the task in parallel on each computer, The data management method according to claim 10.

Images