JP2001022622A - File managing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To manage variable length data by using a variable length field. SOLUTION: In this file managing method, a variable length logic record including variable length data constituting a record included in an original file is converted to a variable length internal field. The record constituting the internal field is divided into a plurality of record groups 11. A block 12 is generated by dividing each record for every plural internal fields inside each record group 11, and a group 13 is generated by linking these divided blocks 12. A inverted file 14 is generated by further linking these groups 13. An offset position in the inverted file 14 in each block 12 inside each group 13 is set to a managing file 15. In a file management system, the relevant variable length data are read out from the inverted file 14 by specifying the block 12 including the variable length data while referring to the managing file 15 for a data read request to the original file.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

TECHNICAL FIELD The present invention relates to a file management method,
In particular, the present invention relates to an improved file management method capable of handling variable length data.

[0002]

2. Description of the Related Art When extracting desired data from a file in which data is stored in a record unit, a process of writing an entire record including the data into an input / output buffer and cutting out only a corresponding field from the input / output buffer is performed. Was needed. That is, for example, when one record is formed by 512 bytes, even if the desired data is 4 bytes, one record, that is, 512 bytes has to be read from the file. To give an example, when creating an address book of employees by extracting only names and addresses from the employee database, according to the above data reading method, records of all employees are read to the input / output buffer, and Only the name and address must be retrieved. With such a method, unnecessary data must be read out, which is not efficient, and the processing load is unnecessarily increased.

In view of this, the same applicant as the present application has proposed a file management method in which data can be read out in units of fields in each record, instead of in units of records (Japanese Patent Application No. 9-319527,
Hereinafter, "prior art documents").

[0004] This management method will be described with reference to FIG. When a plurality of records 3 each composed of a plurality of fields 2 are stored in the original file 1, the records 3 are divided by a certain number, for example, N records. Next, in each of the divided groups, the field provided for storing the same position, that is, the same item is collectively generated by dividing one field at a time from the beginning of each record, and a block 4 is generated. If field 2 is arranged in the row direction in record 3, field 2
Are grouped in the column direction in the group to generate the block 4. Then, in each divided group, the divided blocks 4 are sequentially connected in the row direction from the head, and the group 5 is reorganized. This is performed for all the records 3, and then the group 5 is connected to generate an inverted file 6.

[0005] By generating the transposition file 6 having such a configuration, for example, when it is desired to retrieve only the names included in each record 3 in the above example, only the blocks 4 including the field 2 in which the names are stored are transposed to the transposition file 6. , It is not necessary to read data other than the name such as the employee number and the age from the employee database, so that an efficient data reading process with a small input / output data amount can be performed.

In the prior art, each field (hereinafter referred to as "internal field") 2 constituting the block 4 has a fixed length in order to increase the processing speed. In particular, by making the internal field length coincide with a fixed boundary such as a word boundary, consideration is given so that the number of physical input / output operations to / from the disk device does not increase. Therefore, when each field (hereinafter, “logical field”) constituting the original file 1 has a variable length, as shown in FIG.
Logical fields have been converted to fixed-length internal fields according to one or more fixed boundaries. Then, padding is performed on an area in the internal field that is not filled with the data of the logical field.

[0007] The description of the process of generating the transposed file has been described above with reference to FIG. 8 so that the block 4 is generated from the field 2 included in the original file 1, but actually, as shown in FIG. 9. And a process of converting a logical field into a fixed-length internal field and a process of collecting the internal fields to generate a block 4.

[0008]

However, in the prior art, the internal field length for storing the same item is the length common to all records regardless of the data length stored in the field. Explaining according to the above example, the actual data lengths of addresses stored in the employee database are not all the same, but in the prior art document, in order to register the address data, the logical fields of the same length and fixed length are used for all records. Assigned commonly. For example, if a logical field of 100 bytes is prepared for address data, even if the address data is 40 bytes, 60 bytes are required.
Even a byte is assigned a field length of 100 bytes.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and an object of the present invention is to provide an improved file management method capable of managing variable-length data using variable-length fields. is there.

[0010]

To achieve the above object, a file management method according to the present invention provides a file for managing an original file storing a plurality of records including at least one variable length field. In the management method, when each field constituting a record stored in the original file is converted to an internal field on a one-to-one basis, a variable-length field including variable-length data is converted to a variable-length internal field so that the same A field conversion step of making the length of the internal field variable, a record group generation step of generating a record group by dividing all records constituted by the converted internal fields into a plurality of groups, , The same field in each record A block generating step of generating a block by dividing to be included in each record -
A new group is formed by connecting all the blocks generated in the group, and an inverted file is generated by connecting the formed groups. Further, the storage position of the block including variable length data in the inverted file is specified. Generating a permissible block management information, and identifying and reading the corresponding variable-length data from the transposed file by referring to the storage location information in response to the variable-length data read request from the original file It is.

According to another aspect of the present invention, there is provided a file management method for managing an original file storing a plurality of records including at least one variable length field, wherein each field constituting a record stored in the original file is managed. Is converted into an internal field on a one-to-one basis, the variable length data including the variable length data is registered in the variable length data storage means, and the variable length data storage position in the variable length data storage means is changed. A field conversion step of setting information in an internal field, a record group generation step of generating a record group by dividing all records constituted by the converted internal fields into a plurality of groups, The same field in the record is the same group A block generating step of generating a block by dividing to be included in flops, each record -
And generating a transposed file by connecting all the blocks generated in the original group to form a new group and connecting the formed groups, and reading variable-length data from the original file. The variable length data stored in the variable length data storage means is specified and read by referring to the storage position information of the variable length data stored in the transposed file in response to the request.

In each of the above-mentioned inventions, the field conversion step is to add a free space to the internal field so that the delimiter of each internal field matches a boundary serving as a physical processing unit.

In each of the above inventions, when the generated block size does not match an integer multiple of the physical minimum input / output unit, the block generation step includes adding the block to the block so as to be an integral multiple of the minimum input / output unit. An empty area is added.

[0014]

Preferred embodiments of the present invention will be described below with reference to the drawings.

Embodiment 1 FIG. 1 is a schematic diagram for illustrating a method of generating an inverted file used in Embodiment 1 of the file management method according to the present invention, and FIG. 2 is a diagram illustrating a logical field constituting a record included in an original file. FIG. 3 is a diagram showing the structure of the internal fields handled by the file management system according to the present embodiment, and the correspondence between the fields. The transposed file generation processing according to the present embodiment will be described with reference to these figures and the flowchart shown in FIG.

The procedure of this processing in the present embodiment is basically the same as the procedure described in the prior art. That is, a logical field of the original file is converted into an internal field, a block is generated by combining the same internal fields, and an inverted file is generated by connecting the blocks. Hereinafter, these processes will be described in detail.

In step 101, first, as shown in FIG. 2, each of M logical fields constituting a record
Field0, Field1, ..., Field (M-1) is M internal fields
Convert to Field # 0, Field # 1, ..., Field # (M-1). As shown in FIG. 2A, a field length is added to the logical field in the present embodiment in addition to the actual field data, so that each logical field can be formed to have a variable length. When converting each logical field into an internal field, an empty area is provided as necessary so that the delimiter of the internal field coincides with a word boundary, and padding is performed in the empty area as shown in FIG.
Insert (padding). This is because the boundary of the internal field is matched with a fixed boundary such as a word boundary in order to improve processing efficiency. The size of this free area corresponds to the difference between the logical field length and the internal field length with a fixed boundary in mind. Therefore, as is clear from the relationship between Field1 and Field # 1 in FIG. 2B, the length of the corresponding internal field of the logical field does not always match the logical field. FIG. 2 (b)
Shows an example in which padding is inserted after data. However, the position and number of padding in one internal field are not limited to this example.

A feature of this embodiment is that each internal field (Fiel) included in a record (hereinafter also referred to as an "internal record") constituted by internal fields.
d # 0, Field # 1,..., Field # (M-1)) is not only variable in length, but also the same internal field, for example, FIG.
The field length at the same position in each internal record can be treated as variable, as in each of Field # 1 shown in FIGS. As a result, the variable length data can be handled with the variable length. Explaining according to the above example, if the address data stored in the employee database is 40 bytes, an internal field having a length of 40 bytes can be allocated. If the address data is 60 bytes, an internal field having a length of 60 bytes can be allocated.

When the records (hereinafter, also referred to as "logical records") included in the original file are converted into internal records as described above, the internal records are then divided into a plurality of groups to form a record group. 11 is generated (step 102). In the case of the present embodiment, it is assumed that the original file is divided into predetermined N records and a record group 11 having the same size is generated as in the case of the preceding document.

Subsequently, a block 12 is generated in each record group 11 (step 103). In this case, the same processing is performed on each record group 11. Further, the number of generated blocks in each record group 11 is the same, and in the case of the present embodiment, the internal records are sequentially divided in the row direction by one internal field from the head. Then, the block 12 is generated by combining the same divided internal fields. That is, for example, according to FIG. 1, the first block 1 included in the first record group 11 including the first N records
2 is generated by the same internal record Field # 0 in the first to Nth internal records. In short, this block generation process generates a block 12 by dividing each internal record by one internal field in the row direction and then grouping the internal fields in the column direction. As a result, the number of blocks 1 equal to the number of internal fields
2 will be generated in each record group 11. As in the present embodiment, the processing itself in generating a block by dividing a record group composed of a fixed number N of records into one internal group is the same as that of the preceding document.

By the way, the internal fields in the present embodiment are adjusted so that the boundary between adjacent internal fields matches the word boundary by padding at the time of conversion from the logical field. However, the size of the block 12 is not always an integral multiple of the physical input / output unit. That is, in a storage device such as a disk device, the minimum unit that can be physically input and output is often fixed. This minimum unit is hereinafter referred to as “sector size”. If the input / output unit for accessing a file stored in the disk device is an integral multiple of the sector size, direct input / output can be performed between the storage device and the input / output buffer. However, if the input / output unit is not an integral multiple of the sector size, data may not be input / output unless data in the disk device is once read into the memory and then copied to the input / output buffer. This can be said to be the same in the present embodiment.
It is desirable that the size of 2 be an integral multiple of the sector size. In the present embodiment, the internal field length is made to match a fixed boundary such as a word boundary. However, the size of the block 12 that collectively generates the internal fields is not always an integral multiple of the sector size. .
For example, a sector size is 512 bytes and a word boundary is 4
If it is a byte, the 10-byte logical field is 12 bytes.
Is converted to a byte internal field,
Since × 42 + 8 = 512, the block 12 generated by collecting 12-byte internal fields for 42 records does not become an integral multiple of the sector size.

Therefore, in the present embodiment, when the size of the block 12 does not become an integral multiple of the sector size as a result of generating the block 12 by grouping the internal fields, padding is performed to make the block an integral multiple of the sector size. Adjust so that FIG. 4 is a diagram showing an example of the data structure of one variable-length block 12. If the block size when the internal fields are concatenated does not become an integral multiple of the sector size, the data is added immediately after the final internal field. An empty area 16 is provided, and padding is applied thereto.
In this way, the block size is adjusted to be an integral multiple of the sector size, and the input / output processing can be made more efficient by adjusting the block size.

If padding is inserted in the block generation processing, it is not necessary to perform padding in the internal field. In other words, the block 12 may be padded with the number of differences between the sum of the logical field lengths from which the block 12 is generated and an integral multiple of the sector size. If padding is not performed in the internal field so that the delimiter of the internal field does not match a word boundary, it is not necessary to map (convert) the logical field to the internal field.

Finally, an inverted file is generated (step 104). Here, the groups 13 are newly formed by connecting the blocks 12 generated in each record group 11 in order from the head. Generate. FIG. 1 shows an image in which blocks 12 arranged in the horizontal (row) direction are additionally registered in a file called a group in the order (column direction). In this process, several record groups 13 are created. When the block generation processing is performed on each record group 11, an inverted file 14 is generated by connecting the groups 13. Note that the record group 11 and the group 13 corresponding to the record group 11 have the same internal fields, but have different internal structures due to the difference in the arrangement order of the internal fields. It is shown as an element.

Further, since the size of each block 12 composed of variable-length internal fields is also variable, it is necessary to generate information for specifying the size of each block 12 when the blocks 12 are connected. There is. In the present embodiment, the management file 15 is generated, and the offset position of each block 12 in each group 13 in the transposed file 14 is held and managed as the block management information in the management file 15. As a result, the transposed file 1
When accessing the file 4, the storage location of the desired block 12 in the transposed file 14 can be specified based on the management file 15, and the field data to be accessed can be read. As described above, a single transposed file 14 in which fields are transposed from a single original file
Is generated.

The present embodiment is characterized in that the transposition file 14 is generated in advance as described above, and the processing efficiency is improved by accessing the transposition file 14 in response to a data read request from the original file. And Next, data read processing based on the file management method according to the present embodiment will be described with reference to the flowchart shown in FIG.

The file management system reads the requested field data from the transposition file 14. At this time, in the internal processing, the block containing the requested data is read out to the input / output buffer. However, if the field length is variable, the size of the input / output buffer to be read out is a problem. Becomes Therefore, in the present embodiment, the maximum block size (the maximum length of all the blocks 12) is obtained in advance, and an input / output buffer having a size equal to (or an integral multiple of) the maximum block size is prepared.

When the file management system secures the input / output buffer (step 201), the block position for storing the requested field data by referring to the offset of each block 12 set in the management file 15 and the corresponding block are stored. Then, the storage position is specified, and the field data is sequentially read. That is, the file management system positions the read position of the management file 15 in the first group (step 20).
2) An offset of each block included in the group is obtained (step 203). When the offset position of each block 12 is determined based on the management file 15, since the structure and the field length of the internal field included in the block 12 are known, the storage position of the field data to be read can be easily specified (step 204). . Thereafter, the field data to be read may be sequentially read from the blocks 12 specified in the specified group 13 of the transposed file 14 (steps 205 to 20).
9). In the present embodiment, the reading process from the transposed file 14 is performed asynchronously. When the processing for the first group is completed, the processing shifts to the next group, and finally, the above-described read processing is performed on all the groups included in the transposed file 14 (step 210,
211). As described above, when the data reading process for each group 13 is completed, the input / output buffer is released at this point (step 212).

For example, when a request to retrieve only all the field data stored in the logical field "Field 1" of "address" from the employee database is sent, in the present embodiment, the request corresponds to the logical field "Field 1". The data is extracted from the internal field Field # 1 in the transposed file 14 obtained. At this time, since the internal field Field # 1 is stored in block units, the file management system
Only blocks including the internal field Field # 1 from each group 13 in the transposed file 14 need be read from each group 13. At this time, the storage position of the block 12 of the internal field Field # 1 included in each group 13 in the transposition file 14 can be specified by referring to the management file 15. If the internal field Field # 1 is to be read from the original file, the entire original file must be accessed as a result. However, in the transposed file 14, the internal field Field # 1 is collectively blocked. Therefore, the "address" can be read efficiently without reading unnecessary data.

According to this embodiment, field data can be easily read even if each block length is variable only by holding and managing the block management information.
In the present embodiment, the management file 15 is provided as a separate configuration. However, the management file 15 may be incorporated in the transposition file 14, or information such as the number of records may be stored in both the management file 15 and the transposition file 14 as a measure against a failure. You may make it have and make it duplex.

In the above description, by providing the maximum block size in the management file 15, the necessary buffer size can be ensured before the start of the reading process. If the maximum length of the variable-length data is limited, the maximum block size may be calculated based on the maximum length without providing the maximum block size in the management file 15. This allows
The implementation of the management file 15 can be simplified. Alternatively, the necessary input / output buffer may be repeatedly acquired and released each time field data is read from the block 12. As a result, the memory can be secured with a buffer having a minimum necessary size, and the memory can be used efficiently. Processing to repeatedly acquire and release a buffer when necessary is effective especially when there is an extreme variation in the block size.

In the above embodiment, the offset position of each block 12 is held and managed as the block management information. However, the difference from the immediately preceding block size may be held. As a result, the management file 1
5 can be reduced.

In this embodiment, the transposed file 1
Although the reading process from No. 4 is performed asynchronously, it is necessary to perform asynchronous issuance control in consideration of the limited number, assuming that the number of asynchronous read requests that can be issued simultaneously is limited. Or you may perform as a synchronous process.

Embodiment 2 In the first embodiment, since the variable length data is directly incorporated into the block 12, the block itself has a variable length, and the management file 15 for managing the block size is required. Therefore, in the present embodiment, if the field length is variable length that varies depending on the record, the variable length data is stored in the block 12.
And is stored separately in a variable length data file, and pointer information to the variable length data file is set in the block 12.

FIG. 6 is a schematic diagram showing a method of generating an inverted file used in the file management method according to the second embodiment of the present invention. In the present embodiment, a variable length data file 17 is provided as variable length data storage means for storing variable length data separately from the transposition file 14 without providing the management file 15 shown in the first embodiment. When the data to be handled is of variable length, the internal field where field data is normally stored includes offset information and offset information for specifying the storage position of the corresponding variable length data stored in the variable length data file 17. The size information of the variable length data is stored. As described above, the variable length data is stored in a separate file, and the offset and size data that can be represented by the fixed length are stored in the internal field where the variable length data is to be stored. It can be generated with fixed-length data, that is, actual fixed-length data, offset, and size. In FIG. 6, the offset and the size are set in different internal fields and are generated as another block 12, but the internal fields of the offset and the size are collectively generated as one variable length data block 18. It may be provided in the same internal field having a large size. In this embodiment, similarly to the first embodiment, the blocks 12 can be padded as necessary and stored in the transposition file 14 as in the first embodiment.

The transposed file generation processing in the present embodiment is basically substantially the same as in the first embodiment. Explaining only the different processing, when the data to be handled is of variable length when converting from a logical field to an internal field in step 101, the variable length data file 1
7 is additionally registered. At that time, the variable length data file 17
, The offset information from the beginning of the variable-length data file 17 is acquired as the storage position information of the variable-length data, and set in an internal field together with the size information (variable-length data length). Subsequent processing is the same as that of the first embodiment except that the data included in the internal field is the variable length data itself or the offset and the size. As a result, the transposed file 14 includes the block 12 including the fixed length data and the variable length data block 18 including the offset information and the size information.

Next, data reading processing based on the file management method according to the present embodiment will be described with reference to the flowchart shown in FIG. Here, the process of reading fixed-length data from the transposed file 14 is the same as in the first embodiment, and thus only the process of reading variable-length data will be described. Whether the internal field contains fixed-length data or offset information or size information can be identified by providing an identification flag in the internal field.

When the file management system receives the data read request, the first group 1 of the transposed file 14
Position the pointer (reading position) at 3 (step 22)
1) The variable-length data block 18 including offset information and size information relating to the variable-length data to be read is read out to a non-illustrated inverted file block buffer (step 222). Then, a pointer (processing position) to the first record of the transposed file block buffer
(Step 223), and reads offset information and size information relating to the variable-length data to be read (step 224). Then, based on the read offset information and the size information, data of the size indicated by the size information is read from the storage position specified by the offset information from the variable length data file 17 to a variable length data buffer (not shown) (step 225). ).

Thereafter, the pointer is moved to the next in the transposed file block buffer, and the next offset information and the size information corresponding to the offset information are obtained.
The process of reading variable-length data from the variable-length data file 17 is repeatedly performed (steps 226, 227, and 22).
4,225).

The above processing (steps 224 to 227) is repeated for the variable length data block 18 read out to the transposed file block buffer, and when the processing based on all the offset information and size information is completed, the transposed file 14 The above processing (steps 222 to 227) is repeated until there are no unprocessed groups 13 (steps 228, 229).

According to the present embodiment, variable-length data can be handled as described above. In particular, in the present embodiment, the variable length data is stored in a storage means provided separately from the transposition file 14, and the blocks constituting the transposition file 14 have offset information and size of the variable length data which can be represented by a fixed length. Since the information is stored,
An inverted file can be generated based on fixed length blocks.

[0042]

According to the present invention, variable length fields having different lengths can be assigned to the same field in each record in internal processing, so that variable length data is handled with variable length. Can be.

Also, since the actual variable length data is held by the separately provided variable length data storage means, and the transposition file stores the storage position information such as offset information and size information that can be expressed in a fixed length. An inverted file can be generated based on fixed length blocks.

Further, by adding a free area to an internal field, the delimiter of each internal field can be matched with a boundary serving as a physical processing unit, so that efficient read processing can be realized.

If the block size does not become an integral multiple of the physical minimum input / output unit as a result of generating the block by grouping the internal fields, padding is performed so that the block size becomes an integral multiple of the minimum input / output unit. The input / output process can be performed efficiently.

[Brief description of the drawings]

FIG. 1 is a schematic diagram showing a method of generating an inverted file used in a file management method according to a first embodiment of the present invention.

FIG. 2 is a record structure diagram showing a correspondence relationship between a logical field and an internal field of an original file according to the first embodiment.

FIG. 3 is a flowchart showing a flow of a transposed file generation process according to the first embodiment.

FIG. 4 is a diagram showing an example of a data structure of a variable-length block according to the first embodiment.

FIG. 5 is a flowchart showing a data read process according to the first embodiment.

FIG. 6 is a schematic diagram showing a method of generating an inverted file used in a file management method according to a second embodiment of the present invention.

FIG. 7 is a flowchart showing a data read process according to the second embodiment.

FIG. 8 is a schematic diagram showing a method of generating an inverted file in a conventional file management method.

FIG. 9 is a record structure diagram showing a correspondence relationship between a logical field and an internal field of an original file in the related art.

[Explanation of symbols]

11 record groups, 12 blocks, 13 groups, 14 transposed files, 15 management files, 16
Free space, 17 variable length data files, 18 variable length data blocks.

Claims (4)

[Claims] 1. A file management method for managing an original file storing a plurality of records including at least one variable-length field, wherein each field constituting a record stored in the original file is assigned to an internal field on a one-to-one basis. At the time of conversion, a variable length field including variable length data is converted into a variable length internal field so that the length of the same internal field in each record is variable. A record group generation step of generating a record group by dividing records into a plurality of groups; and a block generation of generating a block by dividing each record group so that the same field in each record is included in the same group. Steps and each A new group is formed by connecting all the blocks generated in the data group, and an inverted file is generated by connecting the formed groups, and the storage position of the block including the variable length data in the inverted file is determined. Generating an identifiable block management information, including the steps of: A file management method characterized by reading. 2. A file management method for managing an original file storing a plurality of records including at least one variable-length field, wherein each field constituting a record stored in the original file is assigned to an internal field on a one-to-one basis. At the time of conversion, the variable length data including the variable length data is registered in the variable length data storage means, and the variable length data storage position information in the variable length data storage means is set as an internal field. A conversion step, a record group generation step of generating a record group by dividing all records composed of the converted internal fields into a plurality of groups, and in each record group, the same field in each record is in the same group. Split to be included Generating a block by generating a transposed file by linking all the blocks generated in each record group to form a new group and connecting the formed groups. And the variable length data stored in the variable length data storage means by referring to the storage position information of the variable length data stored in the transposed file in response to the variable length data read request from the original file. A file management method characterized by identifying and reading a file. 3. The field conversion step according to claim 1, wherein in the field conversion step, a delimiter of each internal field is matched with a boundary serving as a physical processing unit by adding a free area to the internal field. File management method. 4. The block generating step includes, when the generated block size does not match an integral multiple of a physical minimum input / output unit, adding an empty area to the block so as to be an integral multiple of the minimum input / output unit. 3. The file management method according to claim 1, wherein: