JP6355514B2 - File division system and method - Google Patents

Description

  The present invention relates to a file dividing system and method, and more particularly, to a technique for dividing large-capacity data into a plurality of groups at break points or the like as preprocessing for realizing parallel processing such as key break processing for large-capacity data. .

  Key break processing refers to sequentially reading data sorted by a certain item (key) and executing processing such as counting at the timing when the attribute (key) changes (see Non-Patent Document 1).

  For example, as shown in FIG. 5, when a process of “aggregating the amount for each supplier” is given to five data sorted based on the supplier code, the supplier code of each data Are checked in order from the top, and when the supplier code changes, the previous data is extracted, and the process of summing the respective amounts is executed.

2-2. Key break processing Internet URL: http://www003.upp.so-net.ne.jp/NAMBOKU/ruby/ruby022.html Search date: September 10, 2014

  When executing this key break process, it is necessary to compare the key value of each data with the previous key value one by one, so both large-capacity data with the number of data to be processed exceeding several millions Then, it takes a lot of time, and in batch processing after business hours, there may be a situation in which aggregation is not completed by the start of business the next day.

  Originally, we would like to reduce the processing time by dividing the data into multiple files and executing parallel processing using multiple computers or multiple CPU cores. As shown, when a series of data is divided at an appropriate position, so-called “crying separation” occurs in which data having the same key value is dispersed in a plurality of files, which adversely affects subsequent processing.

  However, when a file is divided at an accurate break point, it is necessary to check the data one by one from the top in order to find the break point.

  In addition, it is not limited to the above case that the inconvenience occurs in subsequent processing due to the “breaking up” that occurs when a large file is divided on the basis of the amount of data. A specific character string from a large text file This also applies to the case where the text file is divided into a plurality of files prior to parallel processing using a plurality of CPU cores to output a line including the line.

  For example, when the keyword “Kiku” is given to the text file “… Aiueo \ nKashikaseko \ nSashisuseso \ n…” The original file needs to be split immediately after the newline character (\ n). If it is divided into “Aiueo \ n Kaki” and “Kukeko \ n Sashisoseso”, “Kiku” will cry and will not be recognized as a search target.

  The present invention has been devised in view of such a current situation, and an object of the present invention is to provide a technique capable of efficiently executing a process of dividing large-capacity data at a division allowable point such as a break point.

  In order to achieve the above object, the file division system according to claim 1 divides original data having a plurality of division processing units and a plurality of division allowable points into a plurality of groups based on the amount of data. A data allocation processing unit for sequentially associating the data included in each group with each of the above-described division processing units as allocation data, and a plurality of subsequent processing units. Except for the first division processing unit to which a group is assigned, each division processing unit executes a process of searching for division allowable points from the beginning for its own allocation data, and when the first division allowable point is found, The data before the division allowable point is output to the determined data storage area as determined data, and the data after the first division allowable point is determined The data is output to the determination-unnecessary data storage area as the necessary data, and the first division processing unit directly performs the process of searching for the division allowable point with respect to the assigned data as it is to the determination-unnecessary data storage area. When each of the subsequent processing units outputs determined data related to the division processing unit next to the division processing unit that is in charge of itself, the end of the determination-unnecessary data related to the division processing unit that is in charge Virtually concatenating the determined data to the above, execute the necessary subsequent processing, and if the determined data does not exist, it is necessary only for the determination-unnecessary data related to the division processing unit in charge It is characterized by executing subsequent processing.

  The file division system according to claim 2 is the system according to claim 1, wherein the data allocation processing unit converts the original data including a plurality of records sorted based on the value of the key item on the basis of the number of records. The division processing unit is divided into a plurality of groups, and the division processing unit searches for a break point at which the value of the key item of each record included in the original data changes as a division allowable point.

  The file division system according to claim 3 is the system according to claim 1, wherein the data allocation processing unit divides original data composed of text files including a plurality of line feed characters into a plurality of groups based on the number of characters, The division processing unit searches for a division allowable point immediately after each line feed character included in the original data.

  A file division system according to a fourth aspect is the system according to any one of the first to third aspects, wherein the processing by each of the division processing units is executed by a separate CPU core.

The file division method according to claim 5 is a method executed by a computer , wherein the original data having a plurality of division allowable points is divided into a plurality of groups based on the amount of data, and For each group other than the first group of the original data among a plurality of groups, the search is made for a division allowable point from the beginning, and when the first division allowable point is found, the data before the first division allowable point is detected. Are output to the determined data storage area as determined data, the data after the first division allowable point is output to the determination unnecessary data storage area as determination unnecessary data, and the first group is divided. Without searching for allowable points, the above judgment-unnecessary data is stored as judgment-unnecessary data. Each of the determination unnecessary data other than the determination unnecessary data related to the last group of the original data among the determination unnecessary data stored in the determination unnecessary data storage area, and The necessary subsequent processing is executed after virtually linking the determined data, and the necessary subsequent processing is directly performed on the determination-unnecessary data related to the last group of the original data. It is a feature.

In the case of the file division system according to claim 1, the determined data cut out by each division processing unit is before the division allowable point, and this is the last of the determination-unnecessary data related to the previous division processing unit. Since it is a mechanism in which necessary subsequent processing is executed after being virtually connected to the tail, so-called “crying separation” does not occur between target data of subsequent processing.
Moreover, the sequential division permissible point search process by each division processing unit is limited to the time when the first division permissible point is found in each group, and the division permissible point search process is omitted for the subsequent data. Therefore, the processing can be greatly simplified as compared with the case where the same processing is executed for all of the original data.

  In the file division system according to the fourth aspect, the division allowable point search processing by each distributed processing unit can be parallelized, and the file division processing can be made more efficient.

In the case of the file dividing method according to claim 5, the determined data cut out from each group is before the division allowable point, and this is at the end of the determination unnecessary data related to the previous group. Since it is a mechanism in which necessary subsequent processing is executed after being virtually connected, so-called “crying separation” does not occur between target data of subsequent processing.
In addition, the sequential division permissible point search process for data is limited until the first division permissible point is found in each group, and the division permissible point search process is omitted for subsequent data. Therefore, the process can be greatly simplified as compared with the case where the same process is executed for all of the original data.

  FIG. 1 is a block diagram showing a functional configuration of a file division system 10 according to the present invention, in which a data allocation processing unit 12, an original data storage area 13, a determined data storage area 14, and a determination unnecessary data storage area 15 are shown. A total result storage area 16, a first division processing unit 21, a second division processing unit 22, a third division processing unit 23, a fourth division processing unit 24, and a fifth division process. Unit 25, sixth division processing unit 26, seventh division processing unit 27, first totalization processing unit (subsequent processing unit) 31, second totalization processing unit (subsequent processing unit) 32, The third totalization processing unit (subsequent processing unit) 33, the fourth totalization processing unit (subsequent processing unit) 34, the fifth totalization processing unit (subsequent processing unit) 35, and the sixth totalization processing unit (subsequent) A processing unit) 36 and a seventh tabulation processing unit (subsequent processing unit) 37.

The original data storage area 13, the determined data storage area 14, the determination-unnecessary data storage area 15 and the total result storage area 16 are provided in the auxiliary storage device 42 of the computer 40.
The data allocation processing unit 12, the first division processing unit 21 to the seventh division processing unit 27, and the first aggregation processing unit 31 to the seventh aggregation processing unit 37 are specified by the CPU of the computer 40. This is realized by operating according to an application program.

The computer 40 includes a plurality of CPU cores, and when the first division processing unit 21 to the seventh division processing unit 27 operate, separate CPU cores are assigned to each of them. As a result, the first division processing unit 21 to the seventh division processing unit 27 can execute separate processes in parallel.
In addition, when the first totalization processing unit 31 to the seventh totalization processing unit 37 operate, as a result of assigning a separate CPU core to each, the first totalization processing unit 31 to the seventh totalization processing unit 37 can execute separate processes concurrently.

  Next, processing contents of the file dividing system 10 will be described based on the flowchart of FIG. 2 and the conceptual diagram of FIG.

  First, the data allocation processing unit 12 checks the number of records of the original data 46 stored in the original data storage area 13, and allocates equally for the first division processing unit 21 to the seventh division processing unit 27 (S10 ).

The original data 46 is, for example, millions of pieces of large-capacity data including data items of suppliers, slip numbers, and amounts, and is arranged in advance in ascending order or descending order based on supplier codes.
The data allocation processing unit 12 divides this large-capacity data into seven equal parts in order from the top, and allocates each group to the first division processing unit 21 to the seventh division processing unit 27 in order. In the case where a fraction is generated without being accurately divided into seven, data is allocated more frequently than the other to an appropriate division processing unit.
FIG. 3 shows a state in which the allocation data 51 to 57 are associated with the first division processing unit 21 to the seventh division processing unit 27, respectively.

Next, except for the first division processing unit 21 in charge of the leading group of the original data 46, the second division processing unit 22 to the seventh division processing unit 27 respectively perform the allocation data associated with itself. The break determination process is executed in order from the top (S12), the break determination process is stopped when the first break point α is found, and the determined data located before the break point α is determined as the determined data storage area 14 Is output as a file (S14). Here, the break point corresponds to a “dividable point”.
FIG. 3 shows a state where the determined data 62 to 67 are arranged in the determined data storage area 14.

At the same time, the second division processing unit 22 to the seventh division processing unit 27 output all data existing after the first break point α as files to the determination unnecessary data storage area 15 as determination unnecessary data (S16). .
On the other hand, the first division processing unit 21 outputs its own allocation data 51 as it is to the determination unnecessary data storage area 15 as the determination unnecessary data without executing any break determination processing (S18).
FIG. 3 shows a state in which determination unnecessary data 71 to 77 are arranged in the determination unnecessary data storage area 15.

FIG. 4 shows a specific example of the break determination process.
First, the second division processing unit 22 recognizes a place where the supplier code changes from “003” to “004” as the first break point α, and all the data arranged before the first break point α ( Here, the data having the supplier code “003”) is output to the determined data storage area 14 as the determined data 62.
As shown in the figure, the next break point exists in the determination-unnecessary data 72 arranged after the first break point α, but the second division processing unit 22 makes the determination without restarting the break determination process. The unnecessary data 72 is collectively output to the determination unnecessary data storage area 14.

Next, the first tabulation processing unit 31 includes the determination unnecessary data 71 related to the first division processing unit 21 stored in the determination unnecessary data storage area 15 and the second data stored in the determined data storage area 14. The determined data 62 related to the division processing unit 22 is virtually merged into the first division file 81, and then the aggregation process is performed for each supplier (S20), and the result is stored as the aggregation result. Store in area 16 (S24).
Here, “virtual merge” means that the determination unnecessary data related to the preceding group and the determined data related to the next group are read as one continuous file.

  Similarly, the second aggregation processing unit 32 to the sixth aggregation processing unit 36 are also output by the next division processing unit with respect to the determination unnecessary data 72 to 76 stored in the determination unnecessary data storage area 15, respectively. The determined data 62 to 67 are virtually merged into the second divided file 82 to the sixth divided file 86, and the totaling process is executed (S20), and the result is stored in the totaling result storage area 16. Store (S24).

  On the other hand, since the determination unnecessary data 77 related to the seventh division processing unit 27 in charge of the last group of the original data 46 does not include the determined data to be merged, the seventh aggregation processing unit 37 treats the judgment-unnecessary data 77 as it is as the seventh divided file 87, executes the aggregation processing for this (S22), and stores the aggregation results in the aggregation result storage area 16 (S24).

  As described above, since the first aggregation processing unit 31 to the sixth aggregation processing unit 36 have the virtual merge function, the files of the determination unnecessary data stored in the determination unnecessary data storage area 15 and the determined data The determined data files stored in the storage area 14 can be physically combined to save time and disk capacity when writing to another area as a third file.

  However, the present invention is not limited to this. In other words, the first aggregation processing unit 31 to the sixth aggregation processing unit 36 physically merge the files of the determination unnecessary data 71 to 76 and the files of the determined data 62 to 67 that are in charge of each of them independently. The seventh aggregation processing unit 37 may be configured to output the determination unnecessary data 77 that it is in charge of as it is to the area as it is as a divided file, while outputting the divided file to a predetermined area on the disk. Thereafter, the first totalization processing unit 31 to the seventh totalization processing unit 37 execute the totalization processing for each divided file.

  The number of data included in each of the divided files 81 to 87 varies, and a single divided file includes a plurality of breakpoints, but at least the divided files having the same key value are different. There is no “crying farewell” that scatters inside.

  For example, as shown in FIG. 4, the end of the allocation data 51 of the first division processing unit 21 includes data with the supplier code “003”. Since all the judged data 62 is occupied by the data of the supplier code “003”, and these are virtually combined with the tail of the judgment unnecessary data 71 of the first division processing unit 21, the supplier Data of code “003” is collected in the first divided file 81.

  In each division processing unit, sequential break determination processing is executed. Since different CPU cores are assigned to the first division processing unit 21 to the seventh division processing unit 27, parallel processing is performed. Efficiency improvement is realized.

  In addition, the break determination process is performed until the first break point α is found, and thereafter, the break determination process is not performed as the determination unnecessary data and is output to the determination unnecessary data storage area 15 in a lump. Therefore, the process can be expected to be greatly simplified as compared with the case where all data are checked one by one.

  For each of the divided files 81 to 87, it is guaranteed that the same kind of data does not break up as described above. Parallel processing by the first totalization processing unit 31 to the seventh totalization processing unit 37 is possible.

  In the above, dividing the original data 46 into seven files is merely an example, and it is natural that the original data 46 can be divided into an arbitrary number of files by preparing a corresponding number of division processing units (CPU cores).

  Note that the present invention does not require parallel processing by the computer 32 having a multi-core CPU as described above, but a method for executing processing in order for each group of the original data 46 using a single-core CPU computer Can also be adopted.

In the above, the case where a file is divided into a plurality of parts as a pre-process of the key break process has been exemplified.
As an example, prior to executing parallel processing using multiple CPU cores to output a line containing a specific character string from a large-capacity text file, the text file is converted into multiple files. This is the case when splitting.

  In other words, when the keyword "Kiku" is given to the text file "... Aiueo \ nKakikukuko \ nSashisuseso \ n ..." The original file needs to be split immediately after the newline character (\ n). If it is divided into “Aiueo \ n Kaki” and “Kukeko \ n Sashisoseso”, “Kiku” will cry and will not be recognized as a search target.

  In this situation, if the file dividing system 10 is applied to the original data dividing process, the original data can be accurately divided at the position immediately after the line feed character as follows.

(1) The data allocation processing unit 12 divides the original data into seven groups on the basis of the number of characters, and allocates the original data to the first division processing unit 21 to the seventh division processing unit 27.

(2) On the other hand, the second division processing unit 22 to the seventh division processing unit 27 determine whether or not the assigned data corresponds to a line feed character in order from the first character. When the first line feed character is found, the determined character string including the line feed character is output to the determined data storage area 14 as determined data. Here, immediately after the line feed character corresponds to a “dividable point”.

(3) At the same time, the second division processing unit 22 to the seventh division processing unit 27 collectively output the character string after the first line feed character to the determination unnecessary data storage area 15 as the determination unnecessary data. To do.

(4) The first division processing unit 21 outputs the file as it is to the determination-unnecessary data storage area 15 without performing the above-described line feed character search process for its assigned data.

(5) As a result, seven determination unnecessary data are stored as files in the determination unnecessary data storage area 15 and six determination data are stored as files in the determined data storage area 14. .

(6) After that, seven search processing units (subsequent processing units) are started on the computer, and the process of extracting the line including the character string “KIKU” in units of lines is executed in parallel.

(7) At this time, the first search processing unit virtually merges the determination-unnecessary data related to the first division processing unit 21 and the determined data related to the second division processing unit 22 into one file. handle.

(8) Since the last character string of this judged data ends with a line feed character, if this is connected to the end of the above judgment-unnecessary data, no tearing will occur in the first divided file. .

(9) Similarly, the second search processing unit to the sixth search processing unit perform the second determination unnecessary data to the sixth determination unnecessary data, and the second determined data to the sixth determined data. Are virtually merged, and necessary search processing is executed.

(10) On the other hand, since the seventh determined data does not exist, the seventh search processing unit treats only the seventh determination unnecessary data as a search target.

It is a block diagram which shows the function structure of the file division | segmentation system which concerns on this invention. It is a flowchart which shows the process sequence in this system. It is a conceptual diagram which shows the process sequence in this system. It is explanatory drawing which shows the specific example of a break determination process. It is explanatory drawing which shows the outline | summary of a key break process. It is explanatory drawing which shows the problem which arises by the conventional file division | segmentation.

10 File division system
12 Data allocation processing part
13 Original data storage area
14 Judgment data storage area
15 Judgment unnecessary data storage area
16 Total result storage area
21 First division processing section
22 Second division processing section
23 Third division processing section
24 Fourth division processing section
25 Fifth division processing section
26 Sixth division processing section
27 Seventh division processing section
31 First tabulation processing section
32 Second tabulation processing section
33 Third tabulation processing section
34 Fourth tabulation processing section
35 Fifth tabulation processing section
36 Sixth tabulation processing section
37 Seventh tabulation processing section
40 computers
42 Auxiliary storage
46 source data
51-57 Allocation data
62 to 67 Judgment data
71 to 77 Judgment unnecessary data
81 ~ 87 Split file α First breakpoint

Claims (5)

A plurality of division processing units;
A data allocation processing unit that divides original data having a plurality of division permissible points into a plurality of groups based on the amount of data, and sequentially associates the data included in each group as the allocation data with each of the above division processing units When,
A plurality of subsequent processing units,
Except for the first division processing unit to which the first group of the original data is assigned among the above-mentioned division processing units, each division processing unit executes a process of searching for a division allowable point from the top for its own assigned data. When the first division allowable point is found, the data before the first division allowable point is output as determined data to the determined data storage area, and the data after the first division allowable point is determined as unnecessary data Output to the unnecessary data storage area,
The first division processing unit outputs the determination unnecessary data as it is to the determination unnecessary data storage area as it is without executing a process of searching for a division allowable point for its assigned data.
If there is already determined data related to the division processing unit next to the division processing unit in charge of each of the subsequent processing units, the above determination is made at the end of the determination unnecessary data related to the division processing unit in charge of the self. When necessary data is virtually connected and the necessary subsequent processing is executed, and if the above-mentioned determined data does not exist, the necessary subsequent processing is performed only for the unnecessary determination data related to the division processing unit in charge. A file division system characterized by executing The data allocation processing unit divides the original data consisting of a plurality of records sorted based on the value of the key item into a plurality of groups based on the number of records,
2. The file division system according to claim 1, wherein the division processing unit searches for a break point at which a value of a key item of each record included in the original data changes as a division allowable point. The data allocation processing unit divides the original data composed of text files including a plurality of line feed characters into a plurality of groups based on the number of characters,
The file division system according to claim 1, wherein the division processing unit searches as a division allowable point immediately after each line feed character included in the original data.   The file division system according to claim 1, wherein the processing by each of the division processing units is executed by a separate CPU core. A file division method executed by a computer,
Dividing the original data having a plurality of division allowable points into a plurality of groups based on the amount of data;
For each group other than the first group of the original data among the plurality of groups, a search is made for a division allowable point from the beginning, and when the first division allowable point is found, a point before the first division allowable point is found. Outputting data to the determined data storage area as determined data;
Outputting data after the first division permissible point to the determination unnecessary data storage area as determination unnecessary data;
For the top group, without searching for division allowable points, outputting the determination unnecessary data as it is to the determination unnecessary data storage area as it is,
Of the determination unnecessary data stored in the determination unnecessary data storage area, for each determination unnecessary data other than the determination unnecessary data related to the last group of the original data, the determined data related to the next group is virtually To perform necessary subsequent processing, and for the determination-unnecessary data related to the last group of the original data, performing the necessary subsequent processing as it is,
A file dividing method comprising:

Images