JP2010271781A - System and method for dividing data - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve the efficiency of data processing by logically dividing large amounts of data into capacities suitable for processing on the basis of an appropriate criterion. <P>SOLUTION: Pre-connection data having a plurality of parameters are stored in a pre-connection data temporary storage part 12 as blocks bk for each datum having the value of the same division in the parameters, and the blocks bk whose values of only selection parameters are different are connected by a data connection part 14 as connection candidate blocks, and connection object blocks are extracted from the selected connection candidate blocks by the data connection part 14 so that the total of the capacities of the pre-connection data included in the blocks bk can exceed a prescribed capacity, and the extracted connection candidate blocks are connected, so that a connection file is generated, and written in a connection file storage part 20 by a data writing part 15. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a data division system and a data division method.

  Conventionally, in a system that performs predetermined data processing, when processing is performed by dividing a large amount of accumulated data, for example, the processing is performed on the divided data group by dividing the data into a certain amount of data, for example. Was. Further, when the accumulated data has some attribute, a method of classifying the data for each attribute is known (for example, refer to Patent Document 1).

Special table 2008-524712

  When various processes are performed on a large amount of accumulated data, if a large amount of data is simply divided into data groups each having a fixed capacity, it is necessary to refer to all of the divided data groups. . In addition, even when the accumulated data has attributes (parameters) and a file is generated by simply dividing each attribute, if the division unit is too small, the capacity of one file becomes small, The number of files becomes enormous and the efficiency of data processing to be performed later is poor.

  Therefore, the present invention has been made in view of such a problem, and when dividing a large amount of data into a capacity suitable for processing, the efficiency of data processing is improved by dividing it according to a logically appropriate standard. It is an object of the present invention to provide a data partitioning system and a data partitioning method that can be used.

  In order to solve the above-mentioned problem, the data division system of the present invention divides a set of pre-combination data, which is data having a plurality of parameters, into a plurality of files, and writes the divided files to a predetermined storage means. A pre-combination data temporary storage means for temporarily storing pre-combination data having values belonging to the same range category in a plurality of parameters for each of a plurality of blocks as a group of the pre-combination data; and a plurality of parameters A selection parameter selection means for selecting a selection parameter that is one selected parameter, and a combination in which only the value of the selection parameter selected by the selection parameter selection means is different from a plurality of blocks stored in the pre-combination data temporary storage means A plurality of blocks including the previous data are set as combination candidate blocks, and the combination candidate blocks are grouped. For each combination candidate block group formed by the combination candidate block selection unit that performs block selection processing for forming one or a plurality of combined combination candidate block groups and the combination candidate block selection unit, from the combination candidate block group, A combination target block extraction unit that executes block extraction processing for extracting one or a plurality of combination candidate blocks as a combination target block so as to satisfy a predetermined division criterion, and a combination extracted by the combination target block extraction unit Combined file generation means for executing combined file generation processing for combining the pre-combination data included in the target block and generating one or a plurality of combined files for each combination candidate block group, and the combined file generated by the combined file generation means The file is stored in a predetermined storage means and included in the target block. Characterized in that it comprises a combined file writing means for performing a writing process of deleting all pre-bound data from the combined pre-data temporary storage means to be.

  In order to solve the above problem, the data division method of the present invention divides a set of pre-combination data, which is data having a plurality of parameters, into a plurality of files, and writes the divided files to a predetermined storage means. A data division method, a selection parameter selection step for selecting a selection parameter that is one parameter selected from a plurality of parameters, and pre-combination data having values belonging to the same range category for the plurality of parameters. A plurality of blocks including pre-combination data in which only the value of the selection parameter selected in the selection parameter selection step is different from the plurality of blocks stored in the pre-combination data temporary storage means that temporarily stores the data in a plurality of blocks. Candidates that combine the candidate blocks as a candidate block A combination candidate block selection step for forming one or a plurality of lock groups and a combination candidate block group formed in the combination candidate block selection step so as to satisfy a predetermined predetermined division criterion from the combination candidate block group For each candidate block group, by combining the combination target block extraction step for extracting one or a plurality of combination candidate blocks as a combination target block and the pre-combination data included in the combination target block extracted in the combination target block extraction step A combined file generation step for generating one or a plurality of combined files, and a combined file generated in the combined file generation step is stored in a predetermined storage means and all pre-combination data included in the combination target blocks are combined A combination file to be deleted from the previous data temporary storage means. And having a yl writing step.

  In the data partitioning system and data partitioning method of the present invention, blocks that differ only in the value of the selection parameter are selected as the combination candidate blocks, so the pre-combination data included in the blocks to be combined have the same parameter values other than the selection parameter It is. As a result, a combined file having a logically appropriate meaning can be divided from the set of data before combining. In addition, since the block to be combined is extracted from the selected combination candidate block so as to satisfy a predetermined division criterion, it is possible to divide a combined file having a capacity suitable for data processing from a set of pre-combination data. It becomes.

  Further, in the data division system of the present invention, the combination target block extraction means, as a predetermined division criterion, so that the total amount of data before combination included in the combination candidate blocks exceeds a predetermined predetermined capacity, or One or a plurality of combination candidate blocks are extracted as a combination target block from the combination candidate block group when a predetermined time has passed.

  In this case, as a division criterion when extracting a candidate combination block from the candidate combination block group as a combination target block, the capacity of data before combination included in the candidate combination block, a certain time interval (elapse of a predetermined time), etc. Can be defined by multiple criteria. As a result, the block to be combined can be extracted so that the total capacity of the data before combining exceeds a predetermined capacity, and the block to be combined can be extracted so that the data is divided at regular time intervals. Therefore, it is possible to divide a combined file having a capacity suitable for data processing.

  In the data division system of the present invention, the pre-combination data has a plurality of first to nth parameters (n is an integer equal to or greater than 2), and the selection parameter selection means selects a plurality of parameters as selection parameters. A block selection process by a combination candidate block selection unit and a block by a combination target block extraction unit are selected for each of the first to nth parameters which are sequentially selected as the selection parameters by the selection parameter selection unit. The extraction process, the combined file generating process by the combined file generating means, and the writing process by the combined file writing means are sequentially executed up to n times.

  In the data division method of the present invention, the pre-combination data has a plurality of first to n-th parameters (n is an integer of 2 or more), a selection parameter selection step, a combination candidate block selection step, a combination target block The extracting step, the combined file generating step, and the combined file writing step are sequentially executed at most n times for each of the first to nth parameters.

  In these configurations, when the pre-combination data has a plurality of first to n-th parameters, a process of sequentially combining the blocks for each parameter to divide and generate a combined file from the set of pre-combination data is performed. Therefore, a combined file having a logically appropriate meaning and suitable for data processing can be divided from the set of data before combining.

  In the data division system of the present invention, the combined file generation unit generates the remaining combined file by combining the pre-combined data included in the remaining combined candidate blocks that are not extracted from the combined candidate blocks as the blocks to be combined. The combined file writing unit stores the remaining combined file generated by the combined file generating unit in a predetermined storage unit and erases all the pre-combination data included in the remaining combined target blocks from the pre-combination data temporary storage unit. It is characterized by doing.

  In this case, even if the total capacity of the pre-combination data contained in the block does not reach the predetermined capacity, even if the data does not meet certain division criteria, it is logically combined with the pre-combination data and logical data that were divided and written out as a combined file. Pre-combination data having specific meaning can be written out as a remaining combined file. Thereby, it is possible to prevent the pre-combination data having the same logical meaning as the logical meaning between the pre-combination data included in the connection file from being left in the pre-combination data temporary storage unit.

  In the data division system of the present invention, in the pre-combination data temporary storage unit, when the pre-combination data included in one block satisfies a predetermined division criterion, the combined file generation unit is included in one block. A combined file is generated by combining the pre-combined data.

  A plurality of pre-combination data included in one block have parameter values in the same range section, and thus have logical meanings. In this configuration, since a plurality of pre-combination data having these logical meanings is written out as a joint file, the logical pre-combination data set has a logically appropriate meaning and has a capacity suitable for data processing. Combined files can be split.

  According to the data division system and the data division method of the present invention, blocks that differ only in the value of the selection parameter are selected as the combination candidate blocks. Therefore, the pre-combination data included in the blocks to be combined is a parameter value other than the selection parameter. Are the same range classification. As a result, a combined file having a logically appropriate meaning can be divided from the set of data before combining. Further, since the block to be combined is extracted from the selected combination candidate block so as to satisfy a predetermined division criterion such that the total capacity of the pre-combination data included in the block exceeds a predetermined capacity, A combined file having a capacity suitable for data processing can be divided from the set. In addition, the selection parameter is not single, but the recursive selection of the candidate block and the extraction of the block to be combined are performed using a plurality of parameters. Therefore, the efficiency of data processing using the divided data can be improved.

1 is an overall configuration diagram including a data division system and a data generation node according to an embodiment. It is a block diagram which shows the functional structure of a data division | segmentation apparatus. It is a hardware block diagram of a data division device. It is a figure which shows an example of a structure of the pre-combination data temporary storage part. It is a flowchart which shows the processing content of the data division method implemented in a data division system. It is a flowchart which shows the processing content of the data division method implemented in a data division system. It is a flowchart which shows the processing content of the data division method implemented in a data division system. It is a flowchart which shows the processing content of the data division method implemented in a data division system. It is a flowchart which shows the processing content of the data division method implemented in a data division system. It is the figure which illustrated the state of a part of pre-combination data temporary storage part in the process of the data division | segmentation method. It is the figure which illustrated the state of a part of pre-combination data temporary storage part in the process of the data division | segmentation method.

  A data division system according to an embodiment of the present invention will be described with reference to the drawings. If possible, the same parts are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 is an overall configuration diagram of a system including a data dividing device 1 according to an embodiment of the present invention. As shown in FIG. 1, the system including the data dividing device 1 includes data generation nodes N1 to N3 and an external file system 2.

  The data dividing device 1 is a device that temporarily accumulates data transmitted from the data generation nodes N1 to N3 as a set of data, divides the accumulated data into a plurality of files, and writes the data to an external file system 2 having a predetermined storage means. For example, it is configured by a server device. In the following description, data transmitted from the data generation nodes N1 to N3 is referred to as pre-combination data. The pre-combination data has a plurality of parameters.

  Each of the data generation nodes N1 to N3 is a device that generates pre-combination data and transmits it to the data dividing device 1, and is configured by a server device, for example. In the present embodiment, the data generation nodes N1 to N3 are devices that collect location information of mobile terminals, and the pre-combination data is data of location information of mobile terminals. The location information can have parameters such as region, data type, time, and information source, for example. Further, when the position information is transmitted from any of the data generation nodes N1 to N3 to the data dividing device 1, the identifiers of the data generation nodes N1 to N3 are attached as parameters. Thus, the pre-combination data in this embodiment is position information having the above parameters. In the present embodiment, three data generation nodes N1 to N3 are illustrated, but the number of data generation nodes is not limited to this.

  The external file system 2 includes a combined file storage unit 20 (predetermined storage means), and uses the combined file sent from the data dividing device 1 as a logical meaning of the pre-combined data included in the combined file. The dependent identifier is attached and stored in the combined file storage unit 20.

  Next, the function of the data dividing device 1 will be described with reference to FIG. FIG. 2 is a block diagram showing a functional configuration of the data dividing device 1.

  Functionally, the data dividing apparatus 1 includes a data acquisition unit 11, a pre-combination data temporary storage unit 12 (pre-combination data temporary storage unit), a timer 13, a data combination unit 14 (selection parameter selection unit, combination candidate block selection unit). , A target block extracting unit, a combined file generating unit), and a data writing unit 15 (a combined file writing unit).

  FIG. 3 is a hardware configuration diagram of the data dividing device 1. As shown in FIG. 3, the data dividing apparatus 1 physically includes a CPU 101, a RAM 102 and a ROM 103 which are main storage devices, a communication module 104 which is a data transmission / reception device such as a network card, an auxiliary storage such as a hard disk and a flash memory. The computer system includes a device 105, an input device 106 such as a keyboard and mouse as input devices, an output device 107 such as a display, and the like. Each function shown in FIG. 2 has a communication module 104, an input device 106, and an output device 107 under the control of the CPU 101 by loading predetermined computer software on the hardware such as the CPU 101 and the RAM 102 shown in FIG. This is realized by reading and writing data in the RAM 102 and the auxiliary storage device 105.

  Again, with reference to FIG. 2, each function part of the data division | segmentation apparatus 1 is demonstrated in detail.

  The data acquisition unit 11 is a part that acquires the location information of the mobile terminals sent from the data generation nodes N1 to N3 as pre-combination data, and stores the acquired pre-combination data in the pre-combination data temporary storage unit 12.

  The pre-combination data temporary storage unit 12 is a storage unit that stores pre-combination data. FIG. 4 is a diagram illustrating an example of the configuration of the pre-combination data temporary storage unit 12. In this embodiment, the pre-combination data has parameters A, B, and C, the value of parameter A belongs to one of the range categories of categories a1 to a10, and the value of parameter B is the value of categories b1 to b10. It belongs to any one of the range categories, and the value of the parameter C belongs to any one of the range categories of the categories c1 to c10. In a1 to a10, the range division of a (n) can be arbitrarily set. As an example, a (n) −s ≦ x <a (n + 1) −s (is 0 ≦ s ≦ a (n + 1) −a ( Any number satisfying n) is set to a range of x. For example, a1 has a range of 0 to 1 (including 1), a2 has 1 to 2 (including 2),... A10 has a range of 9 to 10 (including 10), and some pre-combination data is a parameter If A holds a value a such that a = 1.1, it belongs to the range section of a2. As shown in FIG. 4, the pre-combination data temporary storage unit 12 includes a block bk for managing a plurality of pre-combination data having the same division in parameters A to C. In the example illustrated in FIG. 4, the pre-combination data temporary storage unit 12 includes 1000 (10 × 10 × 10) blocks bk. In the following description, for example, a block bk in which the parameter A classification is a1, the parameter B classification is b1, and the parameter C classification is c1 is expressed as “block bk (a1, b1, c1)”. . The block bk is a concept as a group of pre-combination data, and one block bk may be one file or a group of records in the file.

  In the present embodiment, the pre-combination data has three parameters A, B, and C. However, the number of parameters is not limited to three, and the first to nth (n is an integer of 2 or more). It is good also as having a plurality of parameters.

  When the data acquisition unit 11 acquires pre-combination data from the data generation nodes N1 to N3, the block corresponding to the parameter value in the pre-combination data temporary storage unit 12 shown in FIG. Data before combination is stored in bk. For example, the block bk (a1, b1, c1) includes pre-combination data in which the parameter A segment is a1, the parameter B segment is b1, and the parameter C segment is c1. In the present embodiment, for example, the parameter A can be associated with the data generation time, the parameter B with the region, and the parameter C with the data type.

  The timer 13 is a part that outputs to the data combining unit 14 a trigger for performing the processing at regular intervals. For example, the timer 13 can output to the data combining unit 14 a notification of an opportunity to start processing every hour.

  The data combining unit 14 selects a selection parameter that is one parameter selected from a plurality of parameters, and from the plurality of blocks bk stored in the pre-combination data temporary storage unit 12, only the value of the selection parameter differs. A plurality of blocks bk including data are extracted as connection candidate blocks, and the extracted connection candidate blocks are grouped to form one or a plurality of connection candidate block groups. For example, when the parameter A is selected as a selection parameter, ten blocks bk (a1, b1, c1), block bk (a2, b1, c1),..., And block bk (a10, b1, c1) are selected. Block bk is extracted as a combination candidate block, and a combination candidate block group is formed by these blocks bk. Similarly, another candidate block group is formed for other blocks bk. In the example shown in FIG. 4, a maximum of 100 candidate candidate blocks are formed.

  In addition, the data combining unit 14 is configured so that for each combination candidate block group, the combination candidate block group includes pre-combination data included in the combination candidate block so as to satisfy a certain division criterion such as a predetermined capacity or a predetermined time interval. Or it is a part which extracts a some joint candidate block as a candidate block. Here, using the data capacity and the data time interval as the division criterion, the block to be combined is extracted so that the total capacity of the data before combining exceeds a predetermined capacity. Further, the blocks to be combined are extracted so that the data is divided at regular time intervals (elapse of a predetermined time). For example, a combined candidate block group including 10 combined candidate blocks of a block bk (a1, b1, c1), a block bk (a2, b1, c1),..., And a block bk (a10, b1, c1). , And the capacity of data before combination included in the six combination candidate blocks of block bk (a1, b1, c1), block bk (a2, b1, c1),..., And block bk (a6, b1, c1) If the sum of the two exceeds the predetermined capacity, these six blocks bk are extracted as the blocks to be combined.

  Furthermore, the data combining unit 14 is a part that combines the pre-combination data included in the combination target blocks to generate one or a plurality of combined files for each combination candidate block group. For example, when six blocks bk of block bk (a1, b1, c1), block bk (a2, b1, c1),..., And block bk (a6, b1, c1) are extracted as blocks to be combined Are combined with the pre-combination data included in these six blocks bk to generate a combined file. Since the combined file generated here is generated from the six blocks bk, it will be expressed as a combined file (a1 to a6, b1, c1) in the following description.

  Further, the data combining unit 14 is also a part that generates the remaining combined file by combining the pre-combined data included in the remaining combined candidate block that is the block bk that has not been extracted from the combined candidate blocks as the combining target block. As described above, 6 blocks bk of block bk (a1, b1, c1), block bk (a2, b1, c1),..., And block bk (a6, b1, c1) are extracted as blocks to be combined. If the block bk (a7, b1, c1), the block bk (a8, b1, c1),..., And the block bk (a10, b1, c1) are four remaining combination candidate blocks The remaining combined files (a7 to a10, b1, c1) are generated by combining the data before combining included in these four blocks bk.

  The remaining combined file is generated from the remaining combined candidate blocks when the total capacity of the pre-combined data included in the remaining combined candidate blocks is equal to or less than a predetermined capacity. For example, when the sum of the capacities of pre-combination data included in the block bk (a7, b1, c1) and the block bk (a8, b1, c1) exceeds a predetermined capacity, the block bk (a7, b1, c1) and By combining the pre-join data included in the block bk (a8, b1, c1), a join file (a7 to a8, b1, c1) is generated, and a remaining join file (a9 to a10, b1, c1) is generated. The

  On the other hand, the capacity of pre-combination data included in 10 join candidate blocks of block bk (a1, b1, c1), block bk (a2, b1, c1),..., And block bk (a10, b1, c1) If the sum of the data does not exceed the predetermined capacity, the data combining unit 14 generates the combined pending block bka (a0, b1, c1) in which these ten blocks bk are combined into one without generating a combined file. Generate. Here, “a0” is different from any of the divisions a1 to a10, and the value of the parameter A of the pre-combination data included in the combination pending block bka (a0, b1, c1) is set to any of the divisions a1 to a10. Indicates belonging.

  Further, in the pre-combination data temporary storage unit 12, the data combining unit 14 determines that the pre-combination data included in the block bk when the total capacity of the pre-combination data included in one block bk exceeds a predetermined capacity. It is also a part that generates a combined file by combining. For example, when the sum of the capacities of the data before combining included in the block bk (a5, b5, c5) exceeds a predetermined capacity, the data before combining included in the block bk (a5, b5, c5) is combined. , A combined file (a5, b5, c5) is generated.

  The combined file and the remaining combined file generated by the data combining unit 14 are given identifiers that are identified by the parameter classification of the combined file and the pre-combined data included in the remaining combined file. it can.

  The data writing unit 15 is a part that writes the combined file and the remaining combined file generated by the data combining unit 14 to the combined file storage unit 20 of the external file system 2. The data writing unit 15 is a part for erasing the pre-combination data temporarily included in the combined file and the remaining combined file from the pre-combination data temporary storage unit 12.

  As the processing executed by the data combining unit 14 and the data writing unit 15, the above description has exemplified the case where the data combining unit 14 selects the parameter A as the selection parameter. However, the parameter B and the parameter C are selected as the selection parameters. Processing in the case of selection is also performed sequentially. Here, the parameters have priority, and the combination is given priority in the order of A, B, and C. The combined pending block bka generated by the parameter A process is used for the parameter B process.

  In the present embodiment, each of the functional units 11 to 15 is provided in the data dividing device 1, but is configured to be distributed among a plurality of servers that can communicate with each other via a network. The data division system of the present invention may be configured by a plurality of servers.

  Next, the operation of the data dividing device 1 in the data dividing method of this embodiment will be described. FIG. 5 is a flowchart showing the processing contents executed in the data dividing apparatus 1. The trigger for data integration follows a certain division criterion such as a predetermined capacity or a predetermined time interval. As a constant division criterion, for example, a case where the combining process is performed according to the data amount before combining, a case where the combining process is performed after a lapse of a predetermined time by the timer 13, a combination thereof, or the like can be considered. In the present embodiment, as the composite, the data amount before combination and the time by the timer 13 are triggered by data integration. The processing shown in FIG. 5 is performed when the pre-combination data included in one block bk exceeds a predetermined capacity regardless of the time measurement by the timer 13 (before the time trigger is activated by the timer 13). The data before combining included in the data is combined and written out as a combined file.

  First, in step S1, the data acquisition unit 11 tries to acquire pre-combination data from the data generation nodes N1 to N3, and determines whether or not pre-combination data has been acquired. If pre-combination data has been acquired, the processing procedure proceeds to step S2.

  In subsequent step S2, the data acquisition unit 11 stores the acquired pre-combination data in the pre-combination data temporary storage unit 12. At this time, the data acquisition unit 11 refers to the section to which the values of the parameters A to C of the pre-combination data correspond, and before combining the block bk corresponding to the classification of the parameters A to C in the pre-combination data temporary storage unit 12. Classify the data.

  Subsequently, in step S3, the data combining unit 14 calculates the capacity of the data before combining in the block bk in which the data before combining is classified in step S2.

  Next, in step S4, the data combining unit 14 determines whether the capacity of the data before combining calculated in step S3 is equal to or less than a predetermined capacity as a division criterion. If the capacity of the pre-combination data is less than or equal to the predetermined capacity, the processing procedure ends. If the capacity of the data before connection is not less than the predetermined capacity, the processing procedure proceeds to step S5.

  Subsequently, in step S5, the data combining unit 14 combines the data before combining included in the block bk to generate a combined file.

  Further, in step S6, the data writing unit 15 writes the combined file generated in step S5 to the combined file storage unit 20 of the external file system 2 together with the identifiers according to the parameters A to C of the block bk. For example, it is conceivable that the identifiers of the parameters A to C are given to a file name, a meta header in the file, or the like.

  In step S7, the data combining unit 14 erases (clears) the data of the block bk where the pre-combination data has been written, and sets a time flag (not shown) provided for each block bk. set. The fact that the time flag is set means that the pre-combination data included in the block bk is written before the timer 13 triggers the start of processing.

  As a result of the processing in steps S1 to S7 described above, the pre-combination data included in one block bk in which the total capacity of the pre-combination data exceeds a predetermined capacity within the time set in the timer 13 Therefore, it is divided as a combined file with logical meaning. The combined file has a capacity suitable for data processing.

  Next, the process implemented in the data division | segmentation apparatus 1 is demonstrated using FIGS. The processing shown in FIGS. 6 to 9 is a process in which the timer 13 outputs a processing trigger to the data combining unit 14 at regular time intervals, and the data combining unit 14 combines the pre-combined data at regular time intervals and writes it as a combined file. Is to execute.

  First, in step S <b> 10, the data combining unit 14 determines whether or not a notification of processing opportunity has been acquired from the timer 13. If it is determined that a notification of processing opportunity has been acquired, the processing procedure proceeds to step S11.

  Next, in step S11, the data combining unit 14 refers to the pre-combination data temporary storage unit 12 and selects one block bk among the blocks bk containing pre-combination data.

  Next, in step S12, the data combining unit 14 determines whether or not the time flag of the block bk selected in step S11 is set. If the time flag is not set, the process procedure proceeds to step S16. If the time flag is set, the process procedure proceeds to step S13.

  In subsequent step S13, the data combining unit 14 generates the pre-combination data included in the block bk as a combined file as it is. In step S14, the data writing unit 15 writes the combined file generated in step S13 to the combined file storage unit 20 of the external file system 2 together with the identifiers of the parameters b to b of the block bk. In step S15, the data writing unit 15 erases the data of the block bk in the pre-combination data temporary storage unit 12, and the data combination unit 14 sets the time flag of the block bk in the pre-combination data temporary storage unit 12. clear.

  In step S16, the data combining unit 14 refers to the pre-combination data temporary storage unit 12 and determines whether or not all the blocks bk containing the pre-combination data have been selected. If it is determined that all blocks bk have not been selected, the processing procedure returns to step S11. If it is determined that all blocks bk have been selected, the processing procedure proceeds to step S21 in FIG. .

  In the processing of steps S11 to S16 described above, after the writing is performed to the block bk in which the pre-combination data was written in the processing shown in FIG. When pre-combination data is accumulated before it is issued, the accumulated pre-combination data is written out as a combined file. As a result, in the block bk in which writing has been performed due to the total capacity of the data before combination exceeding the predetermined capacity, the data before combining stored in the block bk after the writing does not reach the predetermined capacity. Therefore, it is divided as a combined file from a set of pre-combined data as having logical meaning. As a result, only data having parameters A, B, and C in the same category can be divided and output according to a certain division criterion (time interval and data amount), and another parameter can be assigned to the output file. It can be assured that no data exists. The uncombined data that cannot be divided here is divided according to the division criterion after being combined with the data having the parameter A of different divisions.

  Next, with reference to FIG. 7, the process performed in the data division | segmentation apparatus 1 is demonstrated. The processing in steps S21 to S32 shown in FIG. 7 is performed by selecting the parameter A as a parameter (selection parameter) to be noted when the data combining unit 14 performs the data division processing.

  First, in step S21, the data combining unit 14 selects the parameters B and C in order to pay attention to the block group bk in which the parameters B and C are in the same section and the parameter A is different. For example, initially, parameter B is b1, and parameter C is c1. In the following description, the parameters B and C selected here are referred to as fixed parameters. This step S21 is repeatedly executed until all the fixed parameters are selected. Here, since there are 10 classifications b1 to b10 for parameter B and 10 classifications c1 to c10 for parameter C, this is executed 100 times.

  Next, in step S22, the data combining unit 14 clears the combined block area and clears the combined A flag. The combined block is a block for temporarily storing a combination of pre-combination data included in one or a plurality of blocks bk, and is provided in the pre-combination data temporary storage unit 12. The combined A flag is a flag indicating that the pre-combined data included in the combined block is written out as a combined file.

  Subsequently, in step S23, the data combining unit 14 selects one block bk in which the parameters B and C are the same as the fixed parameter. Here, all the blocks bk selected in the repetition of the processing of steps S23 to S28 are blocks in which the parameters B and C are in the same segment and the parameter A is different, and constitute a candidate block for combining in the present invention. To do. For example, if the fixed parameters are B = b1, C = c1, for example, the block bk (a1, b1, c1) is selected in step S23.

  Next, in step S24, the data combination unit 14 moves the pre-combination data included in the combination candidate block selected in step S23 to the combination block. In step S25, the data combining unit 14 determines whether the total capacity of the pre-combination data included in the combined block is equal to or less than a predetermined capacity. If the total capacity of the pre-combination data included in the combined block is equal to or less than the predetermined capacity, the processing procedure proceeds to step S28, and the total capacity of the pre-combination data included in the combined block is not less than the predetermined capacity. The processing procedure proceeds to step S26.

  In subsequent step S26, the data combining unit 14 combines the pre-combination data included in the combined block to generate a combined file. When the pre-combination data is moved for each block bk with respect to the combined block and the total capacity of the pre-combination data included in the combined block exceeds a predetermined capacity, the pre-combination data included in the combined block is Since the combined file is generated by combining, the capacity of the combined file is suitable for data processing. In addition, since the pre-combination data included in the combined file has the same parameter B and C classification, but only the parameter A classification is different, the combined file is logically meaningful. .

  Next, in step S27, the data writer 15 attaches the combined file generated in step S26 to the combined file of the external file system 2 while attaching an identifier based on the parameter classification to the file name, metadata in the file, or the like. Write to the storage unit 20. Here, the identifier given to the combined file is, for example, that the block bk included in the combined file is a block bk (a1, b1, c1), a block bk (a2, b1, c1),... And a block bk (a6). , B1, c1) can be “joined files (a1 to a6, b1, c1)”. The block bk selected in the processing of steps S23 to S28 and included in the combined file constitutes a combination target block in the present invention.

  In step S27, the data writing unit 15 erases the combined block data. Since the data of the combined block is erased after the data of the combined candidate block is moved to the combined block, the process of deleting the data of the combined block in step S27 is performed before combining all the pre-combined data included in the combination target block. This corresponds to the process of deleting from the temporary data storage means. Further, in step S27, the data combination unit 14 sets a combination A flag.

  In subsequent step S28, the data combination unit 14 determines whether or not all combination candidate blocks having the same parameters B and C are selected. If it is determined that all the combination candidate blocks have been selected, the processing procedure proceeds to step S29. If it is determined that all the combination candidate blocks have not been selected, the processing procedure returns to step S23.

  If it is determined that all the combination candidate blocks have been selected, in step S29, the data combination unit 14 determines whether or not the pre-combination data remains in the combination block and the combination A flag is set at that time. To do. If it is determined that there is pre-join data in the join block and the join A flag is set, the process proceeds to step S30, where there is pre-join data in the join block and the join A flag is set. If it is not determined that it is, the processing procedure proceeds to step S31.

  In the example of the present embodiment, for example, the block bk (a1, b1, c1) and the block bk (a2, b1, c1) are the cases where pre-join data exists in the join block and the join A flag is set. ,... And the combined file (a1 to a6, b1, c1) are generated from the six blocks bk of the block bk (a6, b1, c1) and written, and the block bk (a7, b1, c1) is written. , Block bk (a8, b1, c1),..., And block bk (a10, b1, c1), the sum of the capacities of the pre-combination data does not exceed a predetermined capacity, and the writing process is not performed. In this state, the pre-combination data remains in the combined block. Thus, the block bk that has not been written out as a combined file constitutes a remaining combined candidate block in the present invention. In such a case, in step S30, the data combining unit 14 combines the data before combining included in the combined block to generate a remaining combined file, and the data writing unit 15 converts the remaining combined file into the parameter Along with the identifiers according to the categories A to C, the data is written in the combined file storage unit 20 of the external file system 2. As a result, only data having parameters B and C in the same segment can be segmented and output according to a certain segmentation criterion (time interval and data amount). It can be assured that no data with C exists.

  On the other hand, in the case where the process proceeds from step S29 to step S31, the sum of the capacities of the pre-combination data included in all the combination candidate blocks composed of the blocks bk that are different only in the parameter A section exceeds the predetermined capacity. This is the case. In this case, since the combined block includes the pre-combination data of all the combination candidate blocks, in step S31, the data combining unit 14 combines the pre-combination data of the combined blocks and sets the parameters B and C. Write to the combined pending block bka identified by the partition. For example, pre-combination data included in 10 blocks bk of block bk (a1, b2, c1), block bk (a2, b2, c1),... And block bk (a10, b2, c1) When writing to the combined pending block, the identifier of the combined reserved block can be set to “(a0, b2, c1)”. That is, the pre-join data included in the join pending block bka (a0, b2, c1) belongs to one of the categories of the parameters A1 to a10. Therefore, this combined pending block bka ensures that no division by parameter A is performed, and logical meaning is given by the division of parameter B and parameter C.

  Next, in step S32, the data combining unit 14 determines whether all fixed parameters have been selected. If it is determined that all the fixed parameters have been selected, the processing procedure proceeds to step S41 shown in FIG. 8, and if it is determined that all the fixed parameters have not been selected, the processing procedure returns to step S21. For example, B = b2 and C = c1 are selected as the next fixed parameters.

  As described above, the processing shown in steps S21 to S32 is processing when the data combining unit selects parameter A as the selection parameter. Therefore, when the processing procedure proceeds to step S41, the pre-combination data shown in FIG. All the blocks bk included in the temporary storage unit 12 are combined with blocks bk that are different only in the section of parameter A and written to the combined file storage unit 20 as a combined file or a remaining combined file, or as a combined pending block bka It remains either in the pre-combination data temporary storage unit 12. FIG. 10 illustrates a partial state of the pre-combination data temporary storage unit 12 when the processing procedure proceeds to step S41. FIG. 10 shows the combination holding block bka left in the pre-combination data temporary storage unit 12. The symbol bkv indicates that the block bk is written and the corresponding block bk and pre-combination data do not exist.

  Next, with reference to FIG. 8, the process performed in the data division | segmentation apparatus 1 is demonstrated. The processing in steps S41 to S52 shown in FIG. 8 is performed by selecting the parameter B as a parameter (selection parameter) to be noted when the data combining unit 14 performs the data division processing.

  First, in step S41, the data combining unit 14 selects a certain parameter C as a fixed parameter in order to pay attention to the block group bka in which the parameter C is the same segment and the parameter B is different. For example, the parameter C is initially set to c1. Note that the block group bka to be noted here is one of the combined reserved blocks bka generated in steps S21 to S32 as described above (see FIG. 10). This step S41 is repeatedly executed until all the fixed parameters are selected. Here, since there are 10 classifications c1-c10 for parameter C, it is executed 10 times.

  Next, in step S42, the data combining unit 14 clears the combined block area and clears the combined B flag. The combined B flag is a flag indicating that the pre-combined data included in the combined block is written out as a combined file.

  Subsequently, in step S43, the data combining unit 14 selects one block bka in which the parameter C is the same as the fixed parameter. The block selected here is one of the combined pending blocks bka shown in FIG. Further, all the blocks bka selected in the repetition of the processes of steps S43 to S48 are blocks in which the parameter C is in the same section and the parameter B is different, and constitutes a combination candidate block in the present invention. For example, if the fixed parameter is C = c1, for example, the block bka (a0, b2, c1) is selected in step S43.

  Next, in step S44, the data combination unit 14 moves the pre-combination data included in the combination candidate block selected in step S43 to the combination block. In step S45, the data combining unit 14 determines whether the total capacity of the pre-combination data included in the combined block is equal to or less than a predetermined capacity. If the total capacity of the pre-combination data included in the combined block is equal to or less than the predetermined capacity, the processing procedure proceeds to step S48, and the total capacity of the pre-combination data included in the combined block is not less than the predetermined capacity. The process proceeds to step S46.

  In subsequent step S46, the data combining unit 14 combines the pre-combination data included in the combined block to generate a combined file. Move the pre-combination data for each block to the combined block, and combine the pre-combination data included in the combined block when the total capacity of the pre-combination data included in the combined block exceeds the specified capacity. Thus, the combined file is generated, so that the capacity of the combined file is suitable for data processing. Further, since the pre-combination data included in the combined file is integrated with respect to the parameter A, the parameter C classification is the same, and the parameter B classification is different, the combined file has a logical meaning. It will be a thing.

  Next, in step S47, the data writing unit 15 attaches the combined file generated in step S46 to the combined file of the external file system 2 while attaching an identifier based on the parameter classification to the file name, metadata in the file, or the like. Write to the storage unit 20. Here, the identifier given to the combined file is, for example, that the block bka included in the combined file is a block bka (a0, b2, c1), a block bka (a0, b5, c1) and a block bka (a0, b6, c1). ) 3 blocks, it can be “joined file (a0, b2 to b6, c1)”. The block bka selected in the processing of steps S43 to S48 and included in the combined file constitutes a block to be combined in the present invention.

  In step S47, the data writing unit 15 erases the data of the combined block. Since the data of the combined block is deleted after the data of the combination candidate block is moved to the combined block, the process of deleting the data of the combined block in step S47 is performed before combining all the data before combining included in the combination target block. This corresponds to the process of deleting from the temporary data storage means. Further, in step S47, the data combining unit 14 sets the combined B flag.

  In subsequent step S48, the data combination unit 14 determines whether or not all combination candidate blocks having the same parameter C as the classification are selected. If it is determined that all combination candidate blocks have been selected, the process proceeds to step S49. If it is determined that all combination candidate blocks have not been selected, the process returns to step S43.

  If it is determined that all the combination candidate blocks have been selected, in step S49, the data combination unit 14 determines whether or not the pre-combination data remains in the combination block and the combination B flag is set at that time. To do. If it is determined that there is pre-combination data in the combined block and the combined B flag is set, the processing procedure proceeds to step S50, where there is pre-combined data in the combined block and the combined B flag is set. If it is not determined that it is, the processing procedure proceeds to step S51.

  In the example of this embodiment, for example, the block bka (a0, b2, c1) and the block bka (a0, b5, c1) are the cases where the combined block has pre-joined data and the combined B flag is set. And the combined file (a0, b2 to b6, c1) is generated from the three blocks of the block bka (a0, b6, c1) and written, and the block bka (a0, b9, c1) and the block bka ( The total capacity of the data before combining included in a0, b10, c1) does not exceed the predetermined capacity, and the data before combining remains in the combined block without performing the writing process. Thus, the block bka that has not been written out as a combined file constitutes the remaining combined candidate block in the present invention. In such a case, in step S50, the data combining unit 14 combines the data before combining included in the combined block to generate a remaining combined file, and the data writing unit 15 converts the remaining combined file into the parameter Along with the identifiers of B to C, the data is written to the combined file storage unit 20 of the external file system 2. As a result, only the pre-combination data having the parameter C of the same division can be divided and output according to a certain division criterion (time interval and data amount), and the output file has the parameter C of a different division. It can be guaranteed that no data exists.

  On the other hand, in the case where the process proceeds from step S49 to step S51, the sum of the capacities of the pre-combination data included in all the combination candidate blocks composed of the blocks bka that are different only in the parameter B section exceeds the predetermined capacity. This is the case. In this case, since the combined block includes the pre-combination data of all the combination candidate blocks, in step S51, the data combining unit 14 combines the pre-combination data of the combined block, and classifies the parameter C. Write to the identified combined pending block bkb. For example, a block bka (a0, b1, c2), a block bka (a0, b4, c2), a block bka (a0, b8, c2) and a block bka (a0, b9, When the pre-join data included in the four blocks c2) (join pending block bka) is written to the join pending block bkb, the identifier of this join pending block bkb is “(a0, b0, c2)”. It can be. That is, in the pre-join data included in the join pending block bkb (a0, b0, c2), the value of the parameter A belongs to any one of the sections a1 to a10, and the value of the parameter B belongs to any of the sections of the b1 to b10. It will be. Therefore, this combined pending block bkb ensures that no division by parameters A and B is performed, and logical meaning is given by the division of parameter C.

  Next, in step S52, the data combining unit 14 determines whether all the fixed parameters have been selected. If it is determined that all the fixed parameters have been selected, the processing procedure proceeds to step S61 shown in FIG. 9. If it is determined that all the fixed parameters have not been selected, the processing procedure proceeds to step S41. Returning, for example, C = c2 is selected as the next fixed parameter.

  As described above, the processing shown in steps S41 to S52 is processing when the data combining unit selects parameter B as the selection parameter. Therefore, when the processing procedure proceeds to step S61, the pre-combination data shown in FIG. All the blocks (joining pending block bka) included in the temporary storage unit 12 are joined in blocks having different parameter A and B classifications and written to the joined file storage unit 20 as a joined file or a remaining joined file, Either it remains in the pre-combination data temporary storage unit 12 as a combined pending block. FIG. 11 shows the combined pending block bkb remaining in the pre-combination data temporary storage unit 12. The symbol bkw indicates that the block has been written and the corresponding block and pre-combination data do not exist.

  Next, with reference to FIG. 9, the process performed in the data division | segmentation apparatus 1 is demonstrated. The processes in steps S61 to S71 shown in FIG. 9 are performed by selecting the parameter C as a parameter (selection parameter) to be noted when the data combining unit 14 performs the data division process.

  First, in step S61, the data combining unit 14 releases the fixed parameter in order to focus on the block group bkb having a different parameter C. That is, for the last parameter C, step S41 is not repeated for the number of fixed parameters, but only once. Note that the block group bkb of interest here is one of the combined pending blocks bkb generated in steps S41 to S52 as described above (see FIG. 11).

  Next, in step S62, the data combining unit 14 clears the combined block area and clears the combined C flag. The combined C flag is a flag indicating that the pre-combination data included in the combined block is written as a combined file.

  Subsequently, in step S63, the data combining unit 14 selects one block bkb. The block selected here is one of the combined pending blocks bkb shown in FIG. In addition, all the blocks bkb selected in the repetition of the processing of steps S63 to S68 are blocks with different parameters C, and constitute a candidate block for combining in the present invention. For example, in step S63, the block bkb (a0, b0, c2) is selected.

  Next, in step S64, the data combination unit 14 moves the pre-combination data included in the combination candidate block selected in step S63 to the combination block. In step S65, the data combining unit 14 determines whether the total capacity of the pre-combination data included in the combined block is equal to or less than a predetermined capacity. If the total capacity of the pre-combination data included in the combined block is less than or equal to the predetermined capacity, the processing procedure proceeds to step S68, and the total capacity of the pre-combination data included in the combined block is not less than the predetermined capacity. The process proceeds to step S66.

  In subsequent step S66, the data combining unit 14 combines the pre-combination data included in the combined block to generate a combined file. Move the pre-combination data for each block to the combined block, and combine the pre-combination data included in the combined block when the total capacity of the pre-combination data included in the combined block exceeds the specified capacity. Thus, the combined file is generated, so that the capacity of the combined file is suitable for data processing. Further, since the pre-combination data included in the combined file is integrated with respect to the parameters A and B, and the classification of the parameter C is different, the combined file is logically meaningful.

  Next, in step S67, the data writing unit 15 attaches the combined file generated in step S66 to the file name, the metadata in the file, etc. while attaching the identifier based on the parameter classification to the file name or the metadata in the file. Write to the storage unit 20. Here, the identifier given to the combined file is, for example, that the block bkb included in the combined file is a block bkb (a0, b0, c2), a block bkb (a0, b0, c4) and a block bkb (a0, b0, c8). ), It is possible to use “joined file (a0, b0, c4 to c8)”. The block bkb selected in the processing of steps S63 to S68 and included in the combined file constitutes a combination target block in the present invention.

  In step S67, the data writing unit 15 erases the combined block data. Since the data of the combined block is erased after the data of the combined candidate block is moved to the combined block, the process of deleting the data of the combined block in step S67 is performed before combining all the data before combining included in the block to be combined. This corresponds to the process of deleting from the temporary data storage means. Further, in step S67, the data combining unit 14 sets a combined C flag.

  In subsequent step S68, the data combination unit 14 determines whether all the combination candidate blocks have been selected. If it is determined that all combination candidate blocks have been selected, the processing procedure proceeds to step S69. If it is determined that all combination candidate blocks have not been selected, the processing procedure returns to step S63.

  If it is determined that all the combination candidate blocks have been selected, in step S69, the data combination unit 14 determines whether or not the pre-combination data remains in the combination block and the combination C flag is set at that time. To do. If it is determined that there is pre-join data in the join block and the join C flag is set, the process proceeds to step S70, where there is pre-join data in the join block and the join C flag is set. If it is not determined that it is, the processing procedure proceeds to step S71.

  In the example of this embodiment, for example, the block bkb (a0, b0, c2) and the block bkb (a0, b0, c4) are the cases where there is pre-join data in the join block and the join C flag is set. And the combined file (a0, b0, c4 to c8) is generated from the three blocks of the block bkb (a0, b0, c8) and written out, and is included in the block bkb (a0, b0, c10). In this state, the total data volume before combining does not exceed a predetermined capacity, and the data before combining remains in the combined block without performing the writing process. In this way, blocks that have not been written out as a combined file constitute a remaining combined candidate block in the present invention. In such a case, in step S70, the data combining unit 14 combines the pre-combination data included in the combined block to generate a remaining combined file, and the data writing unit 15 uses the remaining combined file as a parameter. Along with the identifier by the division of C, it is written in the combined file storage unit 20 of the external file system 2. As a result, all the pre-combination data can be divided and output according to a certain division criterion (time interval and data amount).

  On the other hand, in the case where the process proceeds from step S69 to step S71, the sum of the capacities of the pre-combination data included in all the combination candidate blocks composed of the blocks bkb differing only in the parameter C category exceeds the predetermined capacity. This is the case. In this case, since the combined block includes the pre-combination data of all the combination candidate blocks, in step S71, the data combining unit 14 combines the pre-combination data of the combined blocks and sets the parameter C according to the classification. As a combined file to be identified, it is written to the combined file storage unit 20 of the external file system 2. For example, a block bkb (a0, b0, c2), a block bkb (a0, b0, c4), a block bkb (a0, b0, c8) and a block bkb (a0, b0, When the sum of the capacities of the pre-combination data included in the four blocks c10) (joining pending block bkb) does not exceed the predetermined capacity, the process shown in step S71 is performed. This completes the writing of all the pre-combination data stored in the pre-combination data temporary storage unit 12. In this way, the data dividing process performed in the data dividing device 1 is completed.

  In this embodiment, bk, bka, and bkb are distinguished for convenience. However, since arguments inside the block array are not confused, they can be aggregated into a single block array bk. Also, bkv and bkw can be easily distinguished from bka and bkb by confirming the data amount in the block, and therefore can be integrated into the same block arrangement.

  In the present embodiment, the pre-combination data has three parameters A, B, and C, and the process when parameter A is selected as a selection parameter is performed in steps S21 to S32 (FIG. 7), and parameter B is selected. The process when the parameter is selected is shown in steps S41 to S52 (FIG. 8), and the process when the parameter C is selected as the selection parameter is shown in steps S61 to S71 (FIG. 9). On the other hand, when the pre-combination data has a plurality of first to nth parameters (n is an integer of 2 or more), the data combining unit 14 sequentially sets n parameters n times as selection parameters. The processing shown in steps S21 to S32, steps S41 to S52, or steps S61 to S71 is sequentially executed n times for each selected selection parameter. In this case, since the process of dividing and generating the combined file from the set of pre-combination data by sequentially combining the blocks for each parameter is performed, the logically appropriate meaning is obtained from the pre-combination data set. A combined file having a capacity suitable for data processing can be divided and generated.

  Note that it is possible that the pre-combination data has three parameters A, B, and C, and the combination / division processing is performed only for A and B and not for C. In this case, steps S21 to S32 are performed for the parameter A, and steps S41 to S52 are performed for the parameter B, but the combined pending block is not generated in S51 and is written to the file in the same manner as the processing of S71. As a result, it is possible to ensure that data of a single parameter C category exists in the same file without being integrated with respect to the parameters C.

  Next, operational effects of the data division system and the data division method described above will be described. In the data division system and data division method of the present embodiment, since the block bk having a different selection parameter value is selected by the data combining unit 14 as a combination candidate block, the pre-combination data included in the block bk to be combined is already The processed selection parameters are divided into files and written out or integrated as a combined pending block. For the unprocessed selection parameters, the unselected parameter values other than the selection parameters are in the same category. As a result, a combined file having a logically appropriate meaning can be divided from the set of data before combining. Further, from the selected combination candidate blocks, the data combination unit 14 extracts the combination target blocks so that the total capacity of the pre-combination data included in the block bk exceeds a predetermined capacity, and the extracted combination target blocks are Since a combined file is generated by combining and written to the combined file storage unit 20 by the data writing unit 15, it is possible to divide a combined file having a capacity suitable for data processing from the set of data before combining.

  In the data division system according to the present embodiment, even if the total capacity of the pre-combination data included in the block bk does not reach a predetermined capacity, the pre-combination data that has been divided and written out as a combined file by the data combining unit 14 Pre-combination data having logical meaning is extracted as a residual combination candidate block, a residual combined file is generated from the extracted residual combined candidate block, and the generated residual combined file is stored in the combined file by the data writing unit 15 Part 20 can be written out. Thereby, it is possible to prevent the pre-combination data having the same logical meaning as the logical meaning between the pre-combination data included in the connection file from being left in the pre-combination data temporary storage unit. As a result, only data with unselected parameters in the same category can be divided and output according to certain division criteria (time interval and data volume), and unselected parameters in different categories can be output to the output file. It can be assured that no data exists.

  In the data division system of this embodiment, in the pre-combination data temporary storage unit 12, when the total capacity of the pre-combination data included in one block bk exceeds a predetermined capacity, the data combination unit 14 The pre-combination data included in one block bk is combined to generate a combined file. Since the plurality of pre-combination data included in one block bk have parameter values in the same category, they have logical meanings. Accordingly, since the data writing unit 15 writes a plurality of pre-join data having these logical meanings as a join file, it has a logically appropriate meaning from the set of pre-join data, A combined file having a capacity suitable for data processing can be divided.

  DESCRIPTION OF SYMBOLS 1 ... Data division | segmentation apparatus, 11 ... Data acquisition part, 12 ... Pre-combination data temporary storage part, 13 ... Timer, 14 ... Data combination part, 15 ... Data writing part, 2 ... External file system, 20 ... Combined file storage part, N1, N2, N3... Data generation node, bk... Block, bka, bkb.

Claims (7)

A data division system that divides a set of pre-combination data, which is data having a plurality of parameters, into a plurality of files, and writes the divided files to a predetermined storage unit,
Pre-combination data temporary storage means for temporarily storing the pre-combination data having values belonging to the same range category for the plurality of parameters for each of a plurality of blocks as a group of the pre-combination data;
Selection parameter selection means for selecting a selection parameter that is one parameter selected from the plurality of parameters;
Among the plurality of blocks stored in the pre-combination data temporary storage unit, a plurality of blocks including the pre-combination data that differ only in the value of the selection parameter selected by the selection parameter selection unit is used as a combination candidate block. A combination candidate block selection unit that executes block selection processing for forming one or a plurality of combination candidate blocks obtained by grouping combination candidate blocks;
For each of the combination candidate block groups formed by the combination candidate block selection means, one or a plurality of the combination candidate blocks are combined from the combination candidate block group so as to satisfy a predetermined predetermined division criterion. A block extraction means for combining to execute block extraction processing to be extracted as
A combination for generating a combined file generation process for generating one or a plurality of combined files for each combination candidate block group by combining the pre-combination data included in the combination target blocks extracted by the combination target block extraction unit File generation means;
A write process for storing the combined file generated by the combined file generating unit in the predetermined storage unit and erasing all the pre-combination data included in the combination target block from the pre-combination data temporary storage unit; A data dividing system comprising: a combined file writing means for executing. The combination target block extraction means includes:
As the predetermined division criterion, the combination is performed when the total capacity of the pre-combination data included in the combination candidate block exceeds a predetermined predetermined capacity or when a predetermined time has elapsed. The data division system according to claim 1, wherein one or a plurality of the combination candidate blocks are extracted as a combination target block from the candidate block group. The pre-combination data has a plurality of first to nth parameters (n is an integer of 2 or more),
The selection parameter selection means sequentially selects first to n-th parameters from the plurality of parameters as the selection parameters,
For each of the first to nth parameters selected as the selection parameter by the selection parameter selection unit, the block selection process by the combination candidate block selection unit, the block extraction process by the combination target block extraction unit, and the combination 3. The data division system according to claim 1, wherein the combined file generating process by the file generating unit and the writing process by the combined file writing unit are sequentially executed a maximum of n times. The combined file generating unit generates a residual combined file by combining the pre-combined data included in the remaining combined candidate block that is the block that has not been extracted as the combining target block from the combined candidate block,
The combined file writing unit writes the remaining combined file generated by the combined file generating unit to the predetermined storage unit, and all the pre-combined data included in the remaining combined target block is temporarily stored in the pre-combined data The data division system according to any one of claims 1 to 3, wherein the data is deleted from the storage means. In the pre-combination data temporary storage unit, when the pre-combination data included in one block satisfies the predetermined division criterion, the combined file generation unit includes the pre-combination data included in the one block. The data division system according to claim 1, wherein the combined file is generated by combining data. A data division method for dividing a set of pre-combination data, which is data having a plurality of parameters, into a plurality of files and writing the divided files to a predetermined storage means,
A selection parameter selection step of selecting a selection parameter that is one parameter selected from the plurality of parameters;
The plurality of blocks stored in the pre-combination data temporary storage means for temporarily storing the pre-combination data having values belonging to the same range category in the plurality of parameters for each of a plurality of blocks as a group of the pre-combination data. From the plurality of blocks including the pre-combination data that differ only in the value of the selection parameter selected in the selection parameter selection step as a combination candidate block, one or a plurality of combination candidate block groups obtained by grouping the combination candidate blocks A candidate combination block selection step to be formed;
For each combination candidate block group formed in the combination candidate block selection step, one or a plurality of combination candidate blocks are combined from the combination candidate block group so as to satisfy a predetermined predetermined division criterion. A block to be combined extraction step to extract as
A combined file generation step of combining the pre-combination data included in the combination target blocks extracted in the combination target block extraction step to generate one or a plurality of combined files for each combination candidate block group;
The combined file generated in the combined file generation step is stored in the predetermined storage unit, and all the pre-combination data included in the combination target block is deleted from the pre-combination data temporary storage unit A data dividing method comprising: a writing step. The pre-combination data has a plurality of first to nth parameters (n is an integer of 2 or more),
The selection parameter selection step, the combination candidate block selection step, the combination target block extraction step, the combination file generation step, and the combination file writing step are sequentially executed up to n times for each of the first to nth parameters. The data division method according to claim 6.

Images