JP4251725B2 - File management method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a file management method, and more particularly to an improved file management method capable of handling variable length data.
[0002]
[Prior art]
When retrieving desired data from a file in which data is stored in record units, it is necessary to write the entire record including the data to the input / output buffer and cut out only the corresponding field from the input / output buffer. That is, for example, when one record is formed with 512 bytes, even if the desired data is 4 bytes, one record, that is, 512 bytes must be read from the file. As an example, when creating an employee address book by extracting only the name and address from the employee database, the records of all employees are read to the input / output buffer according to the above data reading method. Only the name and address must be taken out from. With such a method, unneeded data must be read out, which is not efficient and the processing load increases unnecessarily.
[0003]
Therefore, the same applicant as the present application has proposed a file management method that allows data to be read in units of fields in each record instead of reading out in units of records (Japanese Patent Application No. 9-319527, hereinafter referred to as “Previous”). Literature ").
[0004]
This management method will be described with reference to FIG. When a plurality of records 3 composed of a plurality of fields 2 are stored in the original file 1, the record 3 is divided into a predetermined number, for example, N records. Next, in each divided group, the fields provided for storing the same position, that is, the same item are grouped by dividing the fields one by one from the beginning of each record to generate the block 4. If field 2 is arranged in the row direction in record 3, it can be said that block 4 is generated by grouping field 2 in the column direction in the group. Then, the group 4 is reorganized by sequentially connecting the blocks 4 divided in the divided groups in the row direction from the top. This is performed for all records 3 and then the group 5 is connected to generate the transposed file 6.
[0005]
By generating the transposed file 6 having such a configuration, for example, when only the name included in each record 3 in the above example is to be extracted, only the block 4 including the field 2 in which the name is stored is sequentially read from the transposed file 6. If issued, it is not necessary to read out data other than the name such as employee number and age from the employee database, so that efficient data reading processing with a small amount of input / output data can be performed.
[0006]
By the way, in the prior art, each field (hereinafter referred to as “internal field”) 2 constituting the block 4 has a fixed length in order to increase the processing speed. In particular, consideration is given to avoiding an increase in the number of physical inputs and outputs to the disk device by matching the internal field length with a fixed boundary such as a word boundary. Accordingly, when each field (hereinafter, “logical field”) constituting the original file 1 has a variable length, as shown in FIG. 9, the logical field is a fixed-length internal field according to one or more fixed boundaries. It was converted to. Then, padding is applied to the area in the internal field that is not filled with the logical field data.
[0007]
In addition, regarding the description of the generation process of the transposed file, the block 4 is generated from the field 2 included in the original file 1 as described above with reference to FIG. 8, but actually, the logical field as illustrated in FIG. To a fixed-length internal field, and a process of collecting internal fields and generating a block 4 in two steps.
[0008]
[Problems to be solved by the invention]
However, in the prior art, the internal field length for storing the same item is the same length for all records regardless of the data length stored in the field. Explaining according to the above example, the actual data lengths of the addresses stored in the employee database are not all the same, but in the prior literature, the same length and a fixed length are used to register the address data. internal Fields are assigned to all records. For example, 100 bytes for address data internal When a field is prepared, a field length of 100 bytes is assigned regardless of whether the address data is 40 bytes or 60 bytes.
[0009]
The present invention has been made to solve the above problems, and an object of the present invention is to provide an improved file management method capable of managing variable length data using a variable length field.
[0010]
[Means for Solving the Problems]
In order to achieve the above object, a file management method according to the present invention is a file management method for managing an original file that stores a plurality of records including at least one variable-length field. When converting each field constituting a record into an internal field on a one-to-one basis, a variable-length field containing variable-length data Each of the fixed boundaries indicating the data delimiters matches one delimiter. Convert to internal field Rufu A field conversion step, a record group generation step for generating a record group by dividing all records composed of converted internal fields into a plurality of groups, and a group in which the same field in each record is the same in each record group A block generation step of generating a block by dividing it so as to be included in For each record group, Blocks generated in the record group Side by side Group Generate and further That Generation Group Include files side by side Transposed file As And a transposition file generation step for generating block management information that can specify a storage position in a transposition file of a block including variable length data, and for storing the storage position information in response to a variable length data read request from the original file. By referencing, the corresponding variable length data is identified and read from the transposed file.
[0011]
A file management method according to another invention is a file management method for managing an original file in which a plurality of records including at least one variable-length field are stored, wherein each field constituting the record stored in the original file is an internal field. When converting to one-to-one, the variable length data is registered in the variable length data storage means for the variable length field including the variable length data, and the variable length data storage position information in the variable length data storage means is stored internally. A field conversion step for setting a field, a record group generation step for generating a record group by dividing all records composed of converted internal fields into a plurality of groups, and the same for each record in each record group The field is in the same group A block generating step of generating a block by dividing as, For each record group, Blocks generated in the record group Side by side Group Generate and further That Generation Group Include files side by side Transposed file As A transposed file generation step, and stored in the variable length data storage means by referring to the storage location information of the variable length data stored in the transposed file in response to the variable length data read request from the original file The variable length data is specified and read.
[0012]
In each of the above inventions, the field conversion step is to add a free area to the internal field so that the delimitation of each internal field matches the boundary that is a physical processing unit.
[0013]
In each of the above inventions, when the generated block size does not match an integer multiple of the physical minimum input / output unit, the block generation step allocates an empty area to the block so that the block size becomes an integer multiple of the minimum input / output unit. It is to be added.
[0014]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.
[0015]
Embodiment 1 FIG.
FIG. 1 is a schematic diagram for illustrating a transposed file generation method used in the first embodiment of the file management method according to the present invention, and FIG. 2 is a logical field constituting a record included in the original file. FIG. 6 is a diagram showing the structure of the internal field, the internal field structure handled by the file management system in the present embodiment, and the correspondence between each field. The transposed file generation process in the present embodiment will be described using these figures and the flowchart shown in FIG.
[0016]
The procedure itself in this embodiment is basically the same as the procedure described in the prior art. That is, the logical field of the original file is converted into an internal field, a block is generated by combining the same internal fields, and a transposed file is generated by concatenating the blocks. Hereinafter, each process will be described in detail.
[0017]
In step 101, first, as shown in FIG. 2, each of M logical fields Field0, Field1,..., Field (M-1) constituting the record is converted into M internal fields Field # 0, Field # 1. , ..., Convert to Field # (M-1). As shown in FIG. 2A, a field length is added to the logical field in the present embodiment in addition to the actual field data. Therefore, each logical field can be formed in a variable length. When converting each logical field into an internal field, an empty area is provided as necessary so that the delimiter of the internal field matches the word boundary, and padding is performed in the empty area as shown in FIG. Insert. This is to make the internal field delimiter coincide with a fixed boundary such as a word boundary in order to improve the processing efficiency. The size of this empty area corresponds to the difference between the logical field length and the internal field length taking into account the fixed boundary. Therefore, as is apparent from the relationship between Field1 and Field # 1 in FIG. 2B, the length of the corresponding internal field of the logical field does not always match that logical field. Although FIG. 2B shows an example in which padding is inserted after the data, the number and number of padding insertions in one internal field are not limited to this example.
[0018]
What is characteristic in this embodiment is that each internal field (Field # 0, Field # 1,..., Field # (M -1)) is not only variable, but the same internal field, for example, the field length at the same position in each internal record, such as Field # 1 shown in FIGS. Is also able to be handled as variable. Thereby, variable length data can be handled with variable length. In accordance with the above example, if the address data stored in the employee database is 40 bytes, an internal field having a length of 40 bytes can be allocated, and if the address data is 60 bytes, an internal field having a length of 60 bytes can be allocated.
[0019]
When a record (hereinafter also referred to as “logical record”) included in the original file is converted into an internal record as described above, a record group 11 is generated by dividing the internal record into a plurality of groups. (Step 102). In the case of the present embodiment, the record group 11 having the same size is generated by dividing the original file into predetermined N records in the same manner as in the prior art.
[0020]
Subsequently, a block 12 is generated in each record group 11 (step 103). This is the same processing for each record group 11. The number of generated blocks in each record group 11 is the same. In the case of the present embodiment, the internal records are sequentially divided in the row direction by one internal field from the top. Then, the block 12 is generated by collecting the same divided internal fields. That is, according to FIG. 1, for example, the first block 12 included in the first record group 11 composed of the first N records is generated by the same internal record Field # 0 in the first to Nth internal records. become. In short, in this block generation processing, each internal record is divided by one internal field in the row direction, and then the internal fields are combined in the column direction to generate a block 12. As a result, the same number of blocks 12 as the number of internal fields are generated in each record group 11. As in the present embodiment, the processing itself in the case of generating a block by dividing a record group composed of a fixed number N records into one internal group is the same as that of the prior art.
[0021]
By the way, the internal field in the present embodiment is adjusted so that the separation from the adjacent internal field matches the word boundary by padding at the time of conversion from the logical field. However, the size of the block 12 is not always an integer multiple of the physical input / output unit. That is, in a storage device such as a disk device, the minimum unit that can be physically input / output is often fixed. This minimum unit is hereinafter referred to as “sector size”. If the input / output unit for accessing a file stored in the disk device is an integral multiple of the sector size, direct input / output can be performed between the storage device and the input / output buffer. However, if the input / output unit is not an integral multiple of the sector size, input / output may not be performed unless data in the disk device is once read into the memory and then copied to the input / output buffer again. This also applies to the present embodiment, and it is desirable to make the size of the block 12 an integral multiple of the sector size in order to perform efficient input / output. In the present embodiment, the internal field length is matched with a fixed boundary such as a word boundary, but the size of the block 12 that collectively generates the internal fields is not necessarily an integral multiple of the sector size. . For example, if the sector size is 512 bytes and the word boundary is 4 bytes, the 10-byte logical field is converted into a 12-byte internal field. At this time, 12 × 42 + 8 = 512, so 12 bytes for 42 records The block 12 generated by combining the long internal fields is not an integral multiple of the sector size.
[0022]
Therefore, in the present embodiment, as a result of generating the block 12 by collecting the internal fields, if the size of the block 12 does not become an integral multiple of the sector size, padding is performed so that the block 12 becomes an integral multiple of the sector size. Adjust to. FIG. 4 is a diagram showing an example of the data structure of one block 12 having a variable length. When the block size when the internal fields are concatenated does not become an integral multiple of the sector size, the last internal field is followed. An empty area 16 is provided, and padding is performed there. In this way, the block size is adjusted to be an integral multiple of the sector size, and the efficiency of input / output processing can be improved by performing such block size adjustment.
[0023]
If padding is inserted in this block generation process, it is not necessary to perform padding in the internal field. That is, the block 12 may be padded with the number of differences between the sum of logical field lengths from which the block 12 is generated and an integer multiple of the sector size. If padding is not performed in the internal field and the boundary of the internal field is not matched with the word boundary, it is not necessary to map (convert) the logical field to the internal field.
[0024]
Finally, a transposition file is generated (step 104). Here, a group 13 is newly generated by first connecting the blocks 12 generated in each record group 11 in order from the top. Based on FIG. 1, the image is such that blocks 12 arranged in the horizontal (row) direction are additionally registered in a file called a group in order (column direction). In this process, several record groups 13 are created. When the block generation process is performed on each record group 11, the transposed file 14 is generated by connecting the groups 13. The record group 11 and the group 13 corresponding to the record group 11 have the same internal fields, but the internal structure is different depending on the arrangement order of the internal fields. Shown as an element.
[0025]
Further, since the size of each block 12 composed of variable length internal fields is also variable, it is necessary to generate information for specifying the size of each block 12 when the blocks 12 are connected. In this embodiment, the management file 15 is generated, and the management file 15 holds and manages the offset position in the transposed file 14 of each block 12 in each group 13 as block management information. Thereby, when accessing the inverted file 14, the storage position of the desired block 12 in the inverted file 14 can be specified based on the management file 15, and the field data to be accessed can be read. As described above, a single transposed file 14 in which fields are transposed from a single original file is generated.
[0026]
The present embodiment is characterized in that processing efficiency is improved by generating the inverted file 14 in advance as described above and accessing the inverted file 14 in response to a data read request from the original file. Next, data read processing based on the file management method in this embodiment will be described with reference to the flowchart shown in FIG.
[0027]
The file management system reads the requested field data from the transposed file 14. At this time, in the internal processing, the block including the requested data is read out to the input / output buffer. However, when the field length is variable, there is a problem as to how large the input / output buffer to be read out is. It becomes. Therefore, in the present embodiment, the maximum block size (the maximum length of all the blocks 12) is obtained in advance, and an input / output buffer having a size equal to (or an integer multiple of) the maximum block size is prepared.
[0028]
When the file management system secures the input / output buffer (step 201), the block position for storing the requested field data by referring to the offset of each block 12 set in the management file 15 and the storage in the block 12 are stored. The field data is sequentially read by specifying the position. That is, the file management system positions the read position of the management file 15 in the first group (step 202), and obtains an offset of each block included in the group (step 203). When the offset position of each block 12 is obtained based on the management file 15, since the structure and field length of the internal field included in the block 12 are known, the storage position of the field data to be read can be easily specified (step 204). . Thereafter, the field data to be read may be sequentially read from the block 12 specified in the specified group 13 of the transposed file 14 (steps 205 to 209). In the present embodiment, the reading process from the transposed file 14 is performed asynchronously. When the processing for the first group is completed, the processing is shifted to the next group, and finally, the above-described reading processing is performed on all the groups included in the transposed file 14 (steps 210 and 211). When the data read processing for each group 13 is completed as described above, the input / output buffer is released at this point (step 212).
[0029]
For example, when a request is sent to retrieve only the field data stored in the logical field Field1 “address” from the employee database, in this embodiment, the transposed file corresponding to the logical field Field1 is sent. The data is extracted from the internal field Field # 1 in FIG. At this time, since the internal field Field # 1 is stored in units of blocks, the file management system can read only the block including the internal field Field # 1 from each group 13 in the transposed file 14 from each group 13. That's fine. At this time, the storage position in the transposed file 14 of the block 12 of the internal field Field # 1 included in each group 13 can be specified by referring to the management file 15. If the internal field Field # 1 is read from the original file, the entire original file must be accessed as a result. However, in the transposed file 14, the internal field Field # 1 is collectively blocked. Therefore, the “address” can be efficiently read without reading unnecessary data.
[0030]
According to the present embodiment, it is possible to easily read out field data even if the length of each block is variable only by holding and managing block management information. In this embodiment, the management file 15 is provided as a separate configuration. However, the management file 15 may be incorporated in the inverted file 14, and information such as the number of records may be stored in both the management file 15 and the inverted file 14 as a countermeasure against the failure. You may make it have double.
[0031]
In the above description, the management file 15 is provided with a maximum block size so that the necessary buffer size can be ensured before the start of the reading process. If the maximum length of variable-length data is limited, the maximum block size may be obtained by calculation based on the maximum length without providing the maximum block size in the management file 15. Thereby, mounting of the management file 15 can be simplified. Alternatively, the necessary input / output buffer acquisition and release may be repeated each time field data is read from the block 12. As a result, the memory can be secured with a buffer having the minimum necessary size, so that the memory can be used efficiently. Processing to repeatedly acquire and release buffers when necessary is effective particularly when there is extreme variation in block sizes.
[0032]
In the above embodiment, the offset position of each block 12 is held and managed as block management information. However, a difference from the previous block size may be held. Thereby, the size of the management file 15 can be reduced.
[0033]
Further, in the present embodiment, the read processing from the transposed file 14 is performed asynchronously, but asynchronous issue control in consideration of this limited number assuming that the number of asynchronous read requests that can be issued simultaneously is limited. Need to do. Or you may perform as a synchronous process.
[0034]
Embodiment 2. FIG.
In the first embodiment, since the variable length data is directly incorporated into the block 12, the block itself has a variable length, and the management file 15 for managing the block size is required. Therefore, in this embodiment, when the field length is variable length that varies depending on the record, the variable length data is not incorporated into the block 12 but stored separately in the variable length data file, and the block 12 stores the variable length data file. The pointer information was set.
[0035]
FIG. 6 is a schematic diagram for showing a transposed file generation method used in the second embodiment of the file management method according to the present invention. In the present embodiment, the variable length data file 17 is provided as variable length data storage means for storing the variable length data separately from the transposed file 14 without providing the management file 15 shown in the first embodiment. Also, in the internal field in which field data is normally stored, when the data to be handled is variable length, offset information and the storage location of the corresponding variable length data stored in the variable length data file 17 are specified. Stores size information of variable length data. As described above, the variable length data is stored in a separate file, and the internal field in which the variable length data is to be stored is stored with offset and size data that can be expressed in a fixed length. It is possible to generate fixed-length data, that is, actual fixed-length data and an offset and size. In FIG. 6, the offset and size are set in different internal fields and generated as another block 12. However, the offset and size internal fields are collectively generated as one variable length data block 18. Alternatively, it may be provided in the same internal field having a large size. In the present embodiment, as in the first embodiment, the block 12 can be padded as necessary and stored in the transposed file 14.
[0036]
The transposed file generation process in the present embodiment is basically the same as that in the first embodiment. To explain only the different processing, when converting from a logical field to an internal field in step 101, if the data to be handled is variable length, the variable length data is not set in the internal field but additionally registered in the variable length data file 17 To do. At this time, offset information from the head of the variable length data file 17 is acquired as the storage position information of the variable length data in the variable length data file 17 and set in the internal field together with the size information (variable length data length). The subsequent processing is the same as that of the first embodiment except that the data included in the internal field is variable length data itself or is offset and size. As a result, the transposed file 14 includes a block 12 including fixed length data and a variable length data block 18 including offset information and size information.
[0037]
Next, data read processing based on the file management method in the present embodiment will be described with reference to the flowchart shown in FIG. Here, since the process of reading fixed-length data from the transposed file 14 is the same as that of the first embodiment, only the process for reading variable-length data will be described. Whether the internal field includes fixed-length data, offset information, or size information can be identified by providing an identification flag in the internal field.
[0038]
When the file management system receives the data read request, the file management system positions the pointer (read position) at the head group 13 of the transposed file 14 (step 221), and includes variable information including offset information and size information regarding the variable length data to be read. The long data block 18 is read into a transposed file block buffer (not shown) (step 222). Then, the pointer (processing position) is positioned at the first record of the transposed file block buffer (step 223), and the offset information and size information relating to the variable length data to be read are read (step 224). Subsequently, based on the read offset information and size information, data having the size indicated by the size information is read from the variable length data file 17 to a variable length data buffer (not shown) from the storage position specified by the offset information (step 225). ).
[0039]
Thereafter, the pointer is moved next in the transposed file block buffer, the next offset information and the size information corresponding to the offset information are acquired, and the process of reading the variable length data from the variable length data file 17 is repeated (step 226, 227, 224, 225).
[0040]
The above processing (steps 224 to 227) is repeated for the variable length data block 18 read into the transposed file block buffer, and when the processing based on all the offset information and size information is completed, the unprocessed file 14 is not processed. The above processing (steps 222 to 227) is repeated until the group 13 is exhausted (steps 228 and 229).
[0041]
According to the present embodiment, variable length data can be handled as described above. In particular, in the present embodiment, variable length data is stored in a storage means provided separately from the transposed file 14, and the offset information and size of the variable length data that can be expressed in a fixed length in the blocks constituting the transposed file 14 Since the information is stored, the transposed file can be generated based on the fixed-length block.
[0042]
【The invention's effect】
According to the present invention, variable length fields having different lengths can be assigned to the same field in each record in internal processing, so that variable length data can be handled with a variable length.
[0043]
In addition, since the actual variable length data is held by variable length data storage means provided separately, and the storage location information such as offset information and size information that can be expressed in a fixed length is stored in the transposition file, It can be generated based on fixed-length blocks.
[0044]
In addition, by adding the empty area to the internal field, the delimitation of each internal field can be matched with the boundary that is a physical processing unit, so that efficient reading processing can be realized.
[0045]
In addition, when the block size is not an integer multiple of the physical minimum input / output unit as a result of generating the block by combining the internal fields, padding is performed to adjust the block size to an integer multiple of the minimum input / output unit. As a result, input / output processing can be performed efficiently.
[Brief description of the drawings]
FIG. 1 is a schematic diagram for illustrating a transposed file generation method used in Embodiment 1 of a file management method according to the present invention.
FIG. 2 is a record structure diagram showing a correspondence relationship between a logical field and an internal field of the original file in the first embodiment.
FIG. 3 is a flowchart showing a flow of transposed file generation processing in the first embodiment.
4 is a diagram showing an example of a data structure of a variable-length block according to Embodiment 1. FIG.
FIG. 5 is a flowchart showing data read processing in the first embodiment.
FIG. 6 is a schematic diagram for illustrating a transposed file generation method used in the second embodiment of the file management method according to the present invention.
FIG. 7 is a flowchart showing data read processing in the second embodiment.
FIG. 8 is a schematic diagram for illustrating a method for generating a transposed file in a conventional file management method.
FIG. 9 is a record structure diagram showing the correspondence between logical fields and internal fields in the original file in the past.
[Explanation of symbols]
11 record group, 12 block, 13 group, 14 transposed file, 15 management file, 16 free area, 17 variable length data file, 18 variable length data block.

Claims (4)

In a file management method for managing an original file storing a plurality of records including at least one variable-length field,
When converting each field that constitutes a record stored in the original file into an internal field on a one-to-one basis, each variable-length field that contains variable-length data is one internal that matches the fixed boundary indicating the data delimiter. and the field conversion step that converts the field,
A record group generation step for generating a record group by dividing all records composed of converted internal fields into a plurality of groups;
In each record group, a block generation step for generating a block by dividing so that the same field in each record is included in the same group;
For each record group, a group including blocks generated in the record group is generated, a file including the generated groups is generated as a transposed file , and a block including variable length data is further transposed. A transposed file generation step for generating block management information capable of specifying a storage position in the file;
A file management method for identifying and reading out the corresponding variable length data from the transposed file by referring to the storage location information in response to a variable length data read request from the original file. In a file management method for managing an original file storing a plurality of records including at least one variable-length field,
When each field constituting the record stored in the original file is converted into an internal field on a one-to-one basis, the variable length data is registered in the variable length data storage means for the variable length field including the variable length data, and A field conversion step of setting the variable length data storage position information in the variable length data storage means in an internal field;
A record group generation step for generating a record group by dividing all records composed of converted internal fields into a plurality of groups;
In each record group, a block generation step for generating a block by dividing so that the same field in each record is included in the same group;
For each record group, generate a group that includes the blocks generated in the record group side by side , and further generate a file that includes the generated groups side by side as a transposed file; and
The variable length data stored in the variable length data storage means is specified by referring to the storage position information of the variable length data stored in the transposed file in response to the variable length data read request from the original file. A file management method characterized by being read out.   3. The file management method according to claim 1 or 2, wherein the field conversion step matches each internal field delimiter with a boundary which is a physical processing unit by adding an empty area to the internal field.   In the block generation step, when the generated block size does not match an integer multiple of the physical minimum input / output unit, an empty area is added to the block so as to be an integer multiple of the minimum input / output unit. The file management method according to claim 1 or 2.

Images