JP5010958B2 - Data management method, program and apparatus - Google Patents

Description

  The present invention relates to a data management method, program, and apparatus for a large-scale database using a computer, and more particularly to a technique for reading and storing tabular data.

  Conventionally, a large-scale database uses a relational database (RDB) that manages a large number of data in a table format and associates and operates a plurality of tables. Since RDB is suitable for transaction processing, it is used in various backbone systems, but it takes time to sort, update, and search all cases.

  In order to overcome such weaknesses of RDB, Patent Document 1 discloses a FAST (Filter Array Structure) structure based on a component decomposition method. In the FAST structure, data that is an array of records in a tabular format is managed by being decomposed (compiled) into order, position, and value components, thereby enabling high-speed processing for all cases.

Japanese Patent No. 3581831

  However, in the data management method adopting the FAST structure described above, since all items included in the tabular original data need to be compiled into the FAST structure, the data registration cost increases. In addition, since compilation is also required when adding, updating, and tabulating data, there is a problem that these processes take time.

  The present invention has been made in view of the above-described problems of the prior art, and can reduce the processing time for data registration, addition, update, etc., while using a FAST structure capable of high-speed search. It is an object (problem) to provide a data management method, program, and apparatus that can be used.

  In the data management method according to the present invention, the computer receives an input reception procedure for receiving input of original data in a table format including a plurality of records each having an item value for a plurality of items, and the input original data is used as a key for data processing. For items to be used, an item value table that is an array in which unique item values are sorted, and an index value that is an array that records the positions in the item value table in which the item values of the records are stored in the order in which the records are arranged Compile procedure for disassembling and saving to table, for input source data, for items not used as data processing key, execute tabular format save procedure to save as tabular data including multiple items in one record It is characterized by doing.

  The computer further regards the data saved by the tabular saving means, assuming that all items saved in the tabular format are regarded as one item, an item value table that is an array in which unique item values are sorted, and the order in which the records are arranged You may make it perform the 2nd compilation procedure which decomposes | disassembles into the index value table which is the arrangement | sequence which recorded the position in the item value table in which the item value of the said record is stored.

  The data management program of the present invention is characterized by causing a computer to function as a means corresponding to each procedure of the above method, and the data management apparatus of the present invention is equivalent to a computer that functions as such.

  According to the present invention, the FAST structure is compiled only for the items used as keys in the input table format original data, and the other items are stored in the table format. Compared to the conventional method of compiling all items by compiling all the items by saving them in tabular form without compiling them, converting items used in the FAST structure to enable high-speed search Processing time for data registration or record addition / update can be shortened.

  Embodiments of a data management method according to the present invention will be described below. First, an outline of a system to which the data management method of the present embodiment is applied will be described based on FIG. The system 1 includes a single computer 10 and peripheral devices. The computer 10 includes a CPU 11, a hard disk (HD) 20, a memory (RAM) 12, and an interface 13 connected to the CPU 11.

  Note that a display 30 and a keyboard 31 are connected to the CPU 11 of the computer 10 via the interface 13 as peripheral devices.

  In addition to an operating system (not shown), a data management program 21 is installed in the HD 20 and a database (DB) 22 is constructed. When the CPU 11 is activated, it reads the operating system from the HD 20 onto the RAM 13 and executes it, and executes the data management program 21 on this operating system.

  The data management program 21 includes an input reception function 21a that accepts input of tabular source data including a plurality of records each having item values for a plurality of items, and items used as data processing keys for the input source data. Is broken down into an item value table that is an array in which unique item values are sorted, and an index value table that is an array that records the positions in the item value table in which the item values of the records are stored in the order in which the records are arranged And a table format saving function 21c that saves the input original data as items of a table format including a plurality of items in one record for items that are not used as data processing keys. It is.

  The DB 22 stores FAST structure data 22a that is compiled and saved for items that are keys, and tabular data 22b that is saved without being compiled for items that are not keys.

  FIG. 1 shows an example in which the work management system is configured by a single computer, but a plurality of terminals are connected via a network using this computer as a server so that input and browsing can be performed from each terminal. May be.

  Next, the contents of the data management process executed in the system 1 will be described based on the flowchart shown in FIG. The flowchart of FIG. 2 shows a procedure for processing the data according to the input command after storing the input text data in a predetermined format. When the process starts, the CPU 11 accepts input of text data in step S001. Text data is CSV format or tabular data such as tab-delimited text and is output from an external relational database or input by a user.

  At the time of input, a selection screen as shown in FIG. 3 is displayed on the display 30, for example. The user selects a text file on this screen and checks the item name as a key using a check box. For example, in this example, a text file called master.txt is selected as the input file. The contents of this file are shown in FIG. Items include ID, name, and gender. In the selection screen of FIG. 3, the item “sex” is designated as an item to be used as a key.

  When text data is input in step S001, the CPU 11 identifies each item from the structure of the text data in step S002, and determines whether the item is used as a key for each item. Items used as keys are compiled into a FAST structure in step S003, and items not used as keys are added to the table-format relay data item array in step S004. While it is determined in step S005 that there are other items, the processes in steps S002 to S004 are repeated. When the processes for all the items are completed, the process proceeds to step S006.

  The processed text data is stored in a format as shown in FIG. That is, the item “gender” used as a key is compiled into a FAST structure, and as shown in FIG. 5A, an item value table ValueList which is an array in which unique item values are sorted, and the order of records in the order set And an index value table ValueNo that is an array in which the position in the item value table in which the item value of the record is stored is recorded and saved.

  On the other hand, items “ID” and “name” that are not used as keys are not compiled, but are stored as relay data items in a table format including a plurality of item values in one record as shown in FIG. However, an index value table ValueList indicating the position of the record is generated.

  Next, when a command is input in step S006, the CPU 11 checks items according to the type of command in steps S007 to S009. That is, if the command is for a compiled item, it is determined whether all the items to be compiled have been compiled. If there is even one item that has not been compiled, the command terminates abnormally. If all the target items have been compiled, or if the target of the command does not target the compiled item, the command is executed in step S010. When there are a plurality of commands, the process returns from step S011 to step S007, and the processes of S007 to S010 are repeatedly executed for all commands.

  When the execution of all the commands is completed, the CPU 11 outputs text data as a processing result in step S012. Here, the compiled items are converted into text format character strings, combined with the tabular relay data items, and output line by line. When all the lines have been output, the process ends.

  For example, when the data shown in FIG. 5 previously registered as a command is input to be sorted and output in the order of gender, a selection screen as shown in FIG. 6 is displayed on the display 30 in order to select an output option. . Here, the output file name is specified as master_new.txt, and items that are not keys are output as tab-delimited data.

  Since sorting by gender is specified, it is determined in step S007 that it is a command to be compiled. In step S008, it is confirmed that this item has been compiled. Since there is no other processing target, step S009 is set to No. In step S010, sorting by gender is executed. The result of sorting is as shown in FIG. That is, for the item “gender”, as shown in FIG. 7A, the array order OrderSet is rearranged so that the values in the index value table ValueNo are in ascending order. Here, “2” and “3” of OderSet are interchanged. Items that are not used as keys are not operated and are stored in the state shown in FIG.

  If there is no other command and step S011 is skipped with No, the results sorted in step S012 are output line by line. At this time, the compiled item “gender” shown in FIG. 7A is converted into a pre-compiled character string, combined with the relay data item shown in FIG. 7B, and output. FIG. 8 shows the contents of the output fill. As shown in this figure, records including ID, name, and gender items are sorted and output in the order of gender.

  In the above embodiment, items that are not key items are stored in the same table format as the input text data without being compiled into the FAST structure. However, it can be further compiled into a FAST structure. That is, for data saved in tabular format, all items saved in tabular format are regarded as one item, an item value table that is an array in which unique item values are sorted, and the item value of the record in the order in which the records are arranged May be decomposed into an index value table that is an array in which positions in the item value table in which are stored are recorded.

  FIG. 9 is an example of a selection screen at the time of input when compiling relay items. In addition to the selection screen shown in FIG. 3, a check box “Compile relay items” is also added. Here, a case where text data as shown in FIG. 10 is input will be described as an example. Compared with the data shown in FIG. 4, the record (row) of the name “Taro Yamada” with ID “0001” has four rows.

  When this text data is input and compiled using gender as a key item as in the above embodiment, a table as shown in FIG. 11A is obtained. The relay item includes an ID and a name, which are further compiled into a FAST structure.

  That is, all items included in the relay items are regarded as one item, and unique item values are sorted. In this example, two items of ID and name are regarded as one item, and records whose values match each other are determined to be the same and are regarded as one record in the item value table. That is, as shown in FIG. 11B, four records having the item value “0001 Taro Yamada” are combined into one record.

  When relay items are also compressed (compiled to FAST structure) as in this modification, it takes more time to compile than the method of the embodiment in which relay items are not compiled, but a plurality of items are compiled only once. Therefore, the time can be shortened as compared with the case where all items are compiled one by one, and the amount of data is compressed after the compilation, so that the consumption of the hard disk and the memory can be reduced.

1 is a block diagram showing a computer system including a data management apparatus according to an embodiment of the present invention. It is a flowchart which shows the content of the process by the data management apparatus of FIG. It is explanatory drawing which shows the selection screen at the time of the file input of the data management apparatus of FIG. It is explanatory drawing which shows the content of the text file input into the data management apparatus of FIG. It is explanatory drawing which shows the structure of the data preserve | saved by the data management apparatus of FIG. It is explanatory drawing which shows the selection screen at the time of the file output of the data management apparatus of FIG. It is explanatory drawing which shows the structure of the data operated by the data management apparatus of FIG. It is explanatory drawing which shows the content of the text file output from the data management apparatus of FIG. It is explanatory drawing which shows the selection screen at the time of the file input of the data management apparatus in the modification of embodiment of this invention. It is explanatory drawing which shows the content of the text file input into the data management apparatus in the modification of embodiment of this invention. It is explanatory drawing which shows the structure of the data preserve | saved by the data management apparatus in the modification of embodiment of this invention.

Explanation of symbols

1 System 10 Computer 11 CPU
12 RAM
13 Interface 20 HD
21 Data management program 21a Input reception function 21b Compile function 21c Tabular format storage function 22 DB
22a FAST structure data 22b tabular data 30 display 31 keyboard

Claims (3)

Computer
An input acceptance procedure for accepting input of original data in a tabular format including multiple records each having an item value for multiple items,
For items used as data processing keys for the input original data, an item value table that is an array in which unique item values are sorted, and the item value in which the item values of the record are stored in the order in which the records are arranged Compile procedure to decompose and save the index value table which is an array that records the position in the table,
For items that are not used as data processing keys for the input original data, a table format storage procedure for storing data as a table format including a plurality of items in one record, and
Regarding the data saved in the table format saving procedure, all items saved in the table format are regarded as one item, an item value table that is an array in which unique item values are sorted, and the items of the record in the order in which the records are arranged. A data management method comprising: executing a second compile procedure that decomposes and stores an index value table that is an array that records positions in the item value table in which values are stored . Computer
Input accepting means for accepting input of original data in a table format including a plurality of records each having an item value for a plurality of items;
For items used as data processing keys for the input original data, an item value table that is an array in which unique item values are sorted, and the item value in which the item values of the record are stored in the order in which the records are arranged A compiling means for disassembling and saving an index value table that is an array in which positions in the table are recorded;
With respect to items that are not used as data processing keys for the input original data, table format storage means for storing the data as tabular data including a plurality of items in one record, and
For the data saved by the tabular format saving means, all items saved in the tabular format are regarded as one item, an item value table that is an array in which unique item values are sorted, and the items of the record in the order in which the records are arranged A data management program that functions as a second compiling unit that decomposes and stores an index value table that is an array in which positions in the item value table in which values are stored are recorded . An input receiving means for receiving input of original data in a tabular format including a plurality of records each having an item value for a plurality of items;
For items used as data processing keys for the input original data, an item value table that is an array in which unique item values are sorted, and the item value in which the item values of the record are stored in the order in which the records are arranged A compiling means for disassembling and storing an index value table that is an array in which positions in the table are recorded;
With respect to items that are not used as data processing keys for the input original data, a table format storage unit that stores data as a table format including a plurality of items in one record;
For the data saved by the tabular format saving means, all items saved in the tabular format are regarded as one item, an item value table that is an array in which unique item values are sorted, and the items of the record in the order in which the records are arranged A data management apparatus comprising: a second compiling unit that decomposes and stores an index value table that is an array in which positions in the item value table in which values are stored are recorded .