JP2011186852A - File dividing device, method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce the maximum size of a divided file and reduce file dividing time, to reduce the memory consumption and accesses to an external storage device. <P>SOLUTION: The number of divisions and the number of duplicates are acquired. The start position of an input file is set for a pointer storage area. A line corresponding to a pointer in the pointer storage area is read from the input file. A distribution file for the number of duplicates (wherein, the number of duplicates<the number of divisions) of the number of divisions is determined so that the lines of the same main key are stored in the same distribution file. The processing of distributing the lines by duplication according to the number of duplicates and adding them into the division file is repeated until the end position of the input file is pointed. The sizes (the number of lines) of division files are compared with each other after division, to eliminate the top N(N=the number of duplicates-1) division files having a large number of lines. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a file dividing apparatus, method, and program, and more particularly, to a file dividing apparatus, method, and program for dividing an input file to be processed (input file) into a plurality of files.

  In order to perform file division, there is a method (round robin) in which a sorting destination is determined in order based on a circulating boundary condition (see, for example, Non-Patent Document 1).

  In addition, there is a method in which a plurality of key duplications are added to an input file and sorted from a small divided file in the descending order of the number of key duplications (see, for example, Patent Document 1).

  In addition, there is a method (integer programming) that constructs an optimal uniform distribution plan based on statistical information of an input file (see, for example, Non-Patent Document 2).

JP 2007-86951 A

Edited by Hideo Aiso and Hidehiko Tanaka “Second Edition Computer Dictionary”, p502, Asakura Shoten Edited by Hideo Aiso and Hidehiko Tanaka “Second Edition Computer Dictionary”, p731, Asakura Shoten

  However, the above round robin method has a problem that the size of the divided files is not equalized when the distribution of the key overlaps of the input file is not uniform.

  In addition, a technique that distributes from a small-sized divided file in descending order of the key duplication number or a technique that uses integer programming cannot start sorting until the entire reading of the input file is completed (key duplication information creation). Therefore, since the distribution is started after the entire reading of the input file is completed, there is a problem that extra memory usage and external storage device access occur.

  The present invention has been made in view of the above points, and an object of the present invention is to provide a file dividing apparatus, method, and program capable of reducing the maximum size of a divided file and shortening the time of file division. .

  It is another object of the present invention to provide a file dividing device, method, and program capable of reducing memory usage and external storage device access.

  FIG. 1 is a principle configuration diagram of the present invention.

The present invention (Claim 1) divides the input file 1 to be processed in which duplication of the primary key is permitted, and stores the file in the divided file 3 so that the rows having the same primary key value are the same distribution destination. A splitting device,
A division number information storage means 6 in which the division number is stored;
A division duplication number information storage means 7 in which the division duplication number is stored;
The division number is obtained from the division number information storage means 6, the division duplication number is obtained from the division duplication number information storage means 7, the head position of the input file is set in the pointer storage area, and the pointer is stored from the input file 1. Read the row corresponding to the pointer of the area, determine the division destination of the division overlap number (however, the division overlap number <the division number) out of the division number so that the same primary key row is the same assignment destination, A file output unit 23 that repeats the process of performing row distribution by performing copying according to the division overlap number and additionally storing the divided file 3 until the pointer reaches the end position of the input file;
After the division is completed, the divided file exclusion unit 24 compares the size (number of lines) of the divided file 3 and excludes the upper N (N = division overlap number-1) divided files having the largest number of lines.

Further, according to the present invention (Claim 2), a distribution destination information storage unit having a distribution destination list for storing a distribution destination for each primary key;
A distribution destination candidate list distribution destination candidate information storage means for storing a distribution destination candidate for each primary key;
The file output means 23
Means for reading a row from the storage area and extracting a primary key from the row;
An element from 1 to the number of divisions ({1, 2,..., The number of divisions}) is set as an initial value in the distribution destination candidate list corresponding to the primary key, and the division weight selected fairly from the distribution destination candidates Means for determining a plurality of distribution destinations and storing them in a distribution destination list corresponding to the primary key;

  FIG. 2 is a diagram for explaining the principle of the present invention.

The present invention (Claim 3) divides a processing target input file in which duplication of a primary key is permitted, and stores the file in the divided file so that rows having the same primary key value are assigned to the same distribution destination. Because
In an apparatus having division number information storage means for storing a division number and division overlap number information storage means for storing a division overlap number,
A division number obtaining step (step 1) for obtaining a division number from the division number information storage means;
A division duplication number acquisition step (step 2) for obtaining the division duplication number from the division duplication number information storage means;
The division number is acquired from the division number information storage means (step 1), the division overlap number is acquired from the division overlap number information storage means (step 2), and the head position of the input file is set in the pointer storage area (step 3). Configuration steps;
The row corresponding to the pointer in the pointer storage area is read from the input file (step 4), and the division number of the division number (provided that the division number is the same) so that the same primary key row is the same distribution destination. The distribution destination of <number of divisions> is determined (step 5), the rows are distributed by copying according to the division overlap number (step 6), and additionally stored in the divided file (step 7) Is repeated until the pointer reaches the end position of the input file (step 8), and a file output step;
After the division is completed, the size (number of lines) of the divided files is compared, and a divided file exclusion step (step 9) for eliminating the top N (N = divided multiples-1) divided files having the largest number of lines;
I do.

The present invention (Claim 4) provides a file output step including:
Means for reading a row from the storage area and extracting a primary key from the row;
An element from 1 to the number of divisions ({1, 2,..., The number of divisions}) is set as an initial value in the distribution destination candidate list corresponding to the primary key, and a plurality of division duplicates selected fairly from the distribution destination candidates The distribution destination is determined and stored in the distribution destination list corresponding to the primary key.

  The present invention (Claim 5) is a file division program for causing a computer to function as each means constituting the file division apparatus according to Claim 1 or 2.

  Duplicate primary key values in the input file are allowed in the entire input file, and lines with the same primary key value are divided in units of lines based on the division duplication number under the constraint that the same distribution destination is used when dividing the file. The division candidate is generated by allocating the copy of the file, and after the division is completed, the division file of the top N (division multiple -1) having the largest division file size is excluded, thereby reducing the maximum size of the division file. In addition, the file division time can be shortened.

  Also, the input file can be scanned once by setting the start position of the input file in the pointer storage area, reading the line from the start position of the pointer storage area, and allocating the line to the end position of the input file. As a result, the memory usage and the number of accesses to the external storage device can be reduced.

It is a principle block diagram of this invention. It is a figure for demonstrating the principle of this invention. It is a block diagram of the file division | segmentation apparatus in one embodiment of this invention. It is the format of the input file in one embodiment of this invention. It is an example of the distribution destination information storage part in one embodiment of this invention. It is an example of the allocation candidate information storage part in one embodiment of this invention. It is a block diagram of the file division | segmentation process part in one embodiment of this invention. It is a flowchart of the file division part in one embodiment of this invention. It is a flowchart of the file output process (S120) in one embodiment of this invention. It is a detailed flowchart of row distribution processing (S123) in an embodiment of the present invention. It is a detailed flowchart of the allocation destination determination process (S1233) in one embodiment of this invention. It is a detailed flowchart of the division | segmentation file exclusion process (S130) in one embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 3 shows a configuration of the file dividing device according to the embodiment of the present invention.

  The file dividing apparatus shown in FIG. 1 includes an input file 1, a file dividing unit 2, a divided file 3, a distribution destination information storage unit 4, a distribution destination candidate information storage unit 5, a division number information storage unit 6, and a division overlap information. It comprises a storage unit 7 and a memory 8. Among these, the allocation destination information storage unit 4, the allocation destination candidate information storage unit 5, the division number information storage unit 6, and the division overlap multiple information storage unit 7 are stored in a storage device connected to the file division processing unit 2 via a data bus. Stored. The memory 8 also has a pointer storage area for storing based on the handle of the input file 1 that has been read.

  As shown in FIG. 4, the input file 1 is a collection of lines composed of a primary key a, a secondary key b, and a data part c, and the primary key a and the secondary key b can identify a line. Key values are allowed to be duplicated throughout the input file. Since the file division is performed for the purpose of performing the parallel calculation of the aggregation calculation based on the primary key value, the same primary key value row imposes a constraint condition that becomes the same distribution destination.

  The distribution destination information storage unit 4 has a distribution destination list as shown in FIG. 5, and a distribution destination for each primary key is set in the distribution destination list.

  The distribution destination candidate storage unit 5 has a distribution destination candidate list as shown in FIG. 6, and a distribution destination candidate is set for each main key.

  The division number information storage unit 6 stores the division number.

  The division overlap number information storage unit 7 stores the division overlap number.

  The file dividing unit 2 generates a divided file 3 from the input file 1.

  FIG. 7 shows the configuration of the file division processing unit in the embodiment of the present invention.

  The file division processing unit 2 includes a division number acquisition unit 21, a division duplication number acquisition unit 22, a file output unit 23, and a divided file exclusion unit 24. The file output unit 23 includes a pointer storage area setting unit 231, a distribution destination determination unit 232, and a divided file storage unit 233.

  The division number acquisition unit 21 acquires the division number from the division number information storage unit 6 and outputs it to the distribution destination determination 232.

  The division duplication number acquisition unit 22 acquires the division duplication number from the division duplication number information storage unit 7 and outputs it to the division file exclusion unit 24 and the distribution destination determination unit 232.

  The pointer storage area setting unit 231 of the file output unit 23 reads the input file 1 and stores the line position of the input file in the pointer storage area P in the memory 8.

  The distribution destination determination unit 232 acquires the position of the line of the input file from the pointer storage area P of the memory 8, reads the line, distributes it to the divided file 3, and stores it. At this time, the assignment destination determination unit 232 extracts the primary key from the read row, and if the primary key is registered in the assignment destination information storage unit 4, the same primary key row is assigned to the same assignment destination. Thus, among the number of divisions, lines corresponding to the division overlap number (however, the division overlap number <the division number) are additionally stored in the division file 3. If it is not registered, the distribution destination is determined by the processing described later in FIG. 11 and added to the distribution destination information storage unit 4.

  Below, the process which produces | generates the division | segmentation file 3 from the input file 1 in said structure is demonstrated.

  FIG. 8 is a flowchart of the file division unit according to the embodiment of the present invention.

  Step 100) The file dividing unit 2 reads the division number information file 6 and extracts the division number.

  Step 110) The division overlap number information storage unit 7 is read to extract the division overlap number.

  Step 120) As the file output process, the process shown in FIG. 9 is performed.

  Step 130) As the divided file exclusion process, the process of FIG. 12 is performed.

  Next, the process of step 120 will be described.

  FIG. 9 is a flowchart of file output processing according to an embodiment of the present invention.

  Step 121) Based on the handle of the input file 1, the input file head position is acquired and stored in the pointer storage area P of the memory 8.

  Step 122) The line corresponding to the pointer in the pointer storage area P is read from the input file 1.

  Step 123) The read lines are allocated. Detailed operation will be described below with reference to FIG.

  Step 1231) A primary key is extracted from the contents of the read line.

  Step 1232) The distribution destination information storage unit 4 is searched based on the primary key to determine whether it has been registered. If it has not been registered, the process proceeds to Step 1233, and if it has been registered, the process proceeds to Step 1235.

  Step 1233) A distribution destination is determined. Details will be described later with reference to FIG.

  Step 1234) The determined allocation destination is additionally stored in the allocation destination list of the allocation destination storage unit 4, and the process proceeds to Step 1236.

  Step 1235) When the primary key is registered in the allocation destination information storage unit 4, the allocation destination information storage unit 4 is searched based on the primary key to acquire the allocation destination.

  Step 1236) The contents of the line are additionally output to the divided file 3 of the distribution destination.

  Step 124) It is determined whether or not the pointer in the pointer storage area P has reached the end position of the input file 1. If not, the process proceeds to Step 125, and if it has reached, the process proceeds to Step 130.

  Step 125) The position of the next row is stored in the pointer storage area P, and the routine proceeds to Step 122.

  Next, the distribution destination determination process in step 1233 will be described with reference to FIG.

  FIG. 11 is a flowchart of the assignment destination determination process according to the embodiment of the present invention.

  The process in FIG. 5 is an example using the Monte Carlo method, but is not limited to this example as long as it is a method that can fairly select a plurality of division destinations from the number of division candidates.

  Step 12331) First, the distribution destination list corresponding to the primary key of the distribution information storage unit 4 is initialized to be empty.

  Step 12332) Searches the distribution destination candidate information storage unit 5 based on the primary key, and sets {1, 2,..., Number of divisions} as an initial value in the corresponding distribution destination candidate list.

  Step 12333) Obtain the number of elements of the distribution destination list corresponding to the primary key of the distribution destination candidate information storage unit 5, equally divide "Section 0 to 1" by the number of elements, and In this order, the assignment destination candidate list is assigned one-to-one. For example, when the number of divisions = 5, it is equally divided into five partial sections, and the number of each element (assignment destination) is assigned.

  Step 12334) A uniform pseudo-random number in “section 0 to 1” is acquired.

  Step 12335) A structure that expresses the assigned association by determining which partial section the pseudo-random number value is included in the partial section obtained in Step 12333 (which partial section contains the random number x) Decide where to distribute based on your body.

  Step 12336) The determined allocation destination (m-th) allocation destination candidate is added to the allocation destination list of the allocation destination information storage unit 4 corresponding to the primary key.

  Step 12337) The determined allocation destination (m-th) allocation destination candidate is removed from the allocation destination candidate list in the allocation destination candidate information storage unit 5 corresponding to the primary key.

  Step 12338) If the number of elements in the primary key distribution destination information storage unit 4 is greater than or equal to the division overlap number, the process is terminated; otherwise, the process proceeds to Step 12333.

  Next, the divided file exclusion process in step 130 will be described.

  FIG. 12 is a detailed flowchart of the divided file exclusion process according to the embodiment of the present invention.

  Step 131) The size (number of lines) of the divided file 3 is acquired, and the handle of the divided file of the top N items (divided multiples-1) having the largest size is acquired.

  Step 132) File exclusion (for example, file deletion) is executed based on the handle of the divided file acquired in Step 131.

  As described above, if the number of lines in the input file 1 is biased toward a part of the key set (assumed to be the upper L) and the number of rows is large, a division candidate is generated, and a large size division is performed after the division is completed. By eliminating files, it is possible to reduce the "probability that all the lines identified by the upper L key will be accommodated in the same distribution destination" by using the property of rapidly increasing the number of combinations. Therefore, the maximum size of the divided file size can be reduced and the file division time can be shortened.

  It is possible to construct the operation of the file division processing unit 2 as a program and install it on a computer used as a file division apparatus for execution, or distribute it via a network.

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed in a computer.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

1 Input file 2 File division processing (CPU)
3 Divided File 4 Distribution Destination Information Storage Unit 5 Distribution Destination Candidate Information Storage Unit 6 Division Number Information Storage Unit 7 Division Duplication / Multiple Information Storage Unit 8 Memory 21 Division Number Acquisition Unit 22 Division Duplication / Multiple Acquisition Unit 24 File Output Unit 24 Division File exclusion unit 231 Pointer storage area setting unit 232 Distribution destination determination unit 233 Division file storage unit

Claims (5)

A file dividing device that divides an input file to be processed for which primary key duplication is allowed, and stores the same primary key value row in the divided file so that the same primary key value row is the same distribution destination,
A division number information storage means in which the division number is stored;
A division duplication number information storage means in which the division duplication number is stored;
Setting means for obtaining a division number from the division number information storage means, obtaining a division duplication number from the division duplication number information storage means, and setting a head position of the input file in a pointer storage area;
The row corresponding to the pointer in the pointer storage area is read from the input file, and the row of the same primary key is the same distribution destination. ) Is assigned, and the row is distributed by performing duplication according to the division overlap number, and additional storage in the division file is performed until the pointer reaches the end position of the input file. Repeated file output means,
A divided file exclusion means for comparing the size (number of lines) of the divided files after completion of the division and for removing the top N (N = divided duplication number−1) divided files having the largest number of lines;
A file dividing device characterized by comprising: A distribution destination information storage unit having a distribution destination list for storing a distribution destination for each primary key;
A distribution destination candidate list distribution destination candidate information storage means for storing a distribution destination candidate for each primary key;
The file output means includes:
Means for reading a row from the storage area and extracting a primary key from the row;
An element from 1 to the number of divisions ({1, 2,..., The number of divisions}) is set as an initial value in the distribution destination candidate list corresponding to the primary key, and the division weight selected fairly from the distribution destination candidates 2. The file dividing apparatus according to claim 1, further comprising means for determining a plurality of distribution destinations and storing them in the distribution destination list corresponding to the primary key. A file splitting method that splits an input file to be processed that allows duplicate primary keys, and stores them in the split file so that rows with the same primary key value are assigned to the same destination.
In an apparatus having division number information storage means for storing a division number and division overlap number information storage means for storing a division overlap number,
A division number obtaining step of obtaining a division number from the division number information storage means;
A division duplication number obtaining step of obtaining the division duplication number from the division duplication number information storage means;
Obtaining a division number from the division number information storage unit, obtaining a division duplication number from the division duplication number information storage unit, and setting a head position of the input file in a pointer storage area;
The row corresponding to the pointer in the pointer storage area is read from the input file, and the row of the same primary key is the same distribution destination. ) Is assigned, and the row is distributed by performing duplication according to the division overlap number, and additional storage in the division file is performed until the pointer reaches the end position of the input file. Repeated file output steps;
A divided file exclusion step for comparing the size (number of lines) of the divided files after the completion of the division, and excluding the top N (N = divided multiple-1) divided files having the largest number of lines;
A file dividing method characterized by: In the file output step,
Means for reading a row from the storage area and extracting a primary key from the row;
An element of 1 to the number of divisions ({1, 2,..., The number of divisions}) is set as an initial value in the distribution destination candidate list corresponding to the primary key, and the number of division overlaps selected fairly from the distribution destination candidates 4. The file dividing method according to claim 3, wherein a distribution destination is determined and stored in a distribution destination list corresponding to the primary key.   A file division program for causing a computer to function as each means constituting the file division apparatus according to claim 1.

Images