JP2001291048A - Data tabulation method and storage medium stored with program for the data tabulation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a data tabulation method for reducing the using amount of a memory and realizing cross tabulation at a high speed. SOLUTION: This method for executing a sum-up processing to plural table form data respectively expressed as the array of records including items and item values included in them is provided with a step for constituting the table form data so as to be respectively divided into one or more information blocks composed of a value list storing the item values in the order of item value numbers corresponding to the item values belonging to a specified item and a pointer array storing pointer values for instructing the item value numbers in the order of unique record numbers, a step for providing a presence number array in which a presence number for indicating the number of the records provided with the equal item value number is arranged corresponding to the value in the value list in the information block, a step for finding the item value in which the presence number is 0 in the presence number array and a step for preparing a sum-up table composed of the presence number corresponding to the item value in the item in the state of eliminating the found item value.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a data processing method and a data processing apparatus for processing a large amount of data by using an information processing apparatus such as a computer. And a method for compressing data obtained when cross-tabulation is performed.

[0002]

2. Description of the Related Art A database (DB) for storing various data by associating a value for each of a plurality of items (name, age, address, school division, occupation, etc.) with each record, particularly a relational database (RD
B) is widely used to manage a series of data groups, such as customer management, order management, and inventory management. In such an RDB, a technique called cross tabulation is used to create a matrix for a plurality of, for example, two items, and to present a table (view) indicating the number of records corresponding to an area in each matrix. For example, as shown in FIG. 21A, corresponding to each record, the item values relating to the items of school classification and age are:
Consider the case where each is given. Here, when cross-tabulation is performed on the two items of the school division and the age, a table (view) as shown in FIG. 21B can be obtained.

[0003] Here, in the example of FIG.
It can be seen that there is no record in the row corresponding to the age of 9 or 9 and no record exists in the column corresponding to the junior high school student. As described above, in the cross tabulation, the elements constituting the matrix (for example, the age is 8 years old and
Depending on the element indicating a record that is a junior high school student), it is possible to see a case where the number of records corresponding to the element is 0. Therefore, in the conventional cross tabulation, when elements other than “0” are scattered (sparse), data compression of elements in the matrix is realized by the following method.

[0004] An item whose number of item values is "R";
In the case of cross tabulation of items whose number of item values is “C”, first, an “R × C” matrix is created, and a table (view) of the cross tabulation is created. Next, paying attention to the columns and rows of the matrix, the sum of the number of elements (the number of records existing) included in the columns is calculated, and the columns and / or rows whose sum is “0” are detected. From the created cross tabulation table (view), a row or column with the total sum of “0” deleted is recreated (see FIG. 21C).

[0005]

However, according to the conventional cross tabulation, even when a large number of rows or columns in which the number of records corresponding to an element is all 0 is found, first, one of the items Number of values related to “R”) ×
A matrix consisting of elements (the number of values “C” for the other item) has been generated.

[0006] In particular, when there are a large number of values "R" and "C" related to an item (item value), for example,
When both “R” and “C” are 10,000, a matrix including 100 million elements is generated by cross tabulation. Therefore, as the number of item values increases, more memory area is required, and there is a problem that the processing time for cross tabulation increases significantly. In addition, there is a problem in that once a cross tabulation table, which is a matrix including “R × C” regions, is created, the total number of the columns and rows is calculated, so that more processing time is required.

An object of the present invention is to provide a processing method capable of reducing the amount of memory used and realizing high-speed cross tabulation.

[0008]

SUMMARY OF THE INVENTION An object of the present invention is a data summarizing method for summarizing a plurality of tabular data, each of which is represented as an array of records each including an item and an item value included therein. The tabular data is divided into a value list in which the item values are stored in the order of the item value numbers corresponding to the item values belonging to the specific item, and the item value numbers in the order of the unique record numbers. And a pointer array for storing pointer values for indicating
Providing, in the information block, a presence number array in which the number of records indicating the number of records having the same item value number is arranged in association with the value in the value list; Finding an item value such that is 0, and a step of creating a summary table comprising the number of items corresponding to the item value in the item with the found item value removed. This is achieved by a data aggregation method characterized by the following.

According to the present invention, with respect to item values, the number of occurrences indicating the number of records is calculated, and a totaling table excluding the item values in which the number of occurrences is 0 is created.
After the tally table is created, the procedure for removing those whose elements are 0 can be omitted.

Another object of the present invention is a data summarizing method for summarizing a plurality of items to tabular data represented as an array of records each including an item and an item value included therein. For each of a plurality of items used for aggregation, a value list in which the item values are stored in the order of the item value numbers corresponding to the item values belonging to the item, and the item list in the order of the unique record numbers Configuring to divide into one or more information blocks consisting of a pointer array storing pointer values for indicating value numbers, and the number of records having the same item value number for each of the items Providing an existence number array in which the number of existences corresponding to the values in the value list is arranged, and the existence number is 0 in each of the existence number arrays.
Finding the item value such that is, and the number of each element of the matrix-like summary table consisting of the plurality of items,
Projecting the multidimensional position of each element to a one-dimensional position to correspond to an address; and, in each of the existence number arrays, the element corresponding to the element whose existence number is not 0. Generating at least an offset array such that the element becomes an offset value based on the projection, and by referring to a corresponding pointer value in the pointer array for each item based on the record number, Specifying an offset value in a corresponding offset array, adding the obtained offset value, calculating an added offset value, and calculating an address by associating the added offset with the projection. Counting up the value of the position indicated by the obtained address, It is achieved by the data aggregation method characterized in that aggregate tables based on the value of the position indicated in dress is obtained.

According to the present invention, when a total table of a plurality of dimensions is created, a projection matching an address indicating an element in the total table is obtained based on the added offset value.
Therefore, once you create a multi-dimensional summary table,
A desired tabulation table can be obtained without having to find an area where the element is 0.

In a preferred embodiment of the present invention,
When the number of item values of each item is Ni (0 ≦ i ≦ I−1: I is
The number of elements whose value is 0 in the existence number array of each item is represented by ni ((0
≦ i ≦ when I-1) represented by the offset value Pi _m offset arrangement of each item (except, 0 ≦ m ≦ Ni-n
i) I _{am, Pi 0 = 0 Pi (m} + 1) -Pi m = Π (Nk-nk) (0 ≦ k
.Ltoreq.m-1).

[0013] In another embodiment of the present invention,
Prior to providing the existence number array, a step of selecting a record number that can be counted from among the record numbers, and a step of generating a record number set in which the selected record numbers are arranged in a predetermined order And, based on the record numbers included in the record number set,
Generating the existence number array by referring to a corresponding pointer value in the pointer array. Further, the object of the present invention is also achieved by a computer-readable storage medium storing a program for executing the above method.

[0014]

Embodiments of the present invention will be described below with reference to the accompanying drawings. FIG. 1 is a block diagram showing a hardware configuration of a computer system capable of realizing cross tabulation and data compression according to an embodiment of the present invention. As shown in FIG.
The computer system 10 has a configuration similar to that of a normal computer. By executing a program, the computer system 10 controls the entire system and individual components, and a RAM (Random Access Memory) for storing work data and the like.
y) 14, ROM (Read Only Mem
ory) 16, fixed storage medium 18, such as hard disk, CD
A CD-ROM driver 20 for accessing the ROM 19; an interface (I / F) 22 provided between the CD-ROM driver 20 and an external terminal connected to an external network (not shown); a keyboard and a mouse Input device 24 and a CRT display device 26. CPU 12, RAM 14, ROM 16, external storage medium 18, I / F 22, input device 24, and display device 26
Are connected to each other via a bus 28.

A program for realizing cross tabulation based on tabular data, a program for compressing data obtained by cross tabulation, and a program for creating a cross tabulation (view) according to the present embodiment are CD-RO.
It may be stored in the M19 and read by the CD-ROM driver 20, or may be stored in the ROM 16 in advance. The data once read from the CD-ROM 19 may be stored in a predetermined area of the external storage medium 18. Alternatively, the program may be supplied from the outside via a network (not shown), an external terminal, and the I / F 22.

In the present embodiment, it is necessary to generate an information block having a predetermined data format, as described later, in order to realize the above-described cross tabulation and data compression at a high speed. Similarly, this information block generation program may be stored in the CD-ROM 19, stored in the ROM 16, or stored in the external storage medium 18. Alternatively, these programs are stored on a network (not shown)
Needless to say, it may be supplied from the outside via the. In the present embodiment, the data (information block) generated by the information block generation program is:
It is stored in the RAM 14 or in a predetermined area of the external storage medium 18.

Next, the data format and the principle of tabulation, which are the premise of the present invention, will be described. The present inventor has devised a method of constructing tabular data having a specific data format and a method of searching, counting, and sorting in order to achieve ultra-high-speed processing (PCT WO00 / 10913). Also in the present invention, basically, tabular data is constructed as a set of predetermined information blocks based on the method disclosed in this application, and tabulation including cross tabulation is realized using this. .

FIG. 2 is a diagram showing information blocks used in the present embodiment. As shown in FIG. 2, the information block 100 includes a value list 110 and an array of pointers 120 to the value list. The value list 110 stores, for each item of the tabular data, the item values 111 corresponding to the item value numbers in the order of the item value numbers in which the item values belonging to the item are ordered (converted into integers). It is a table. The pointer array 120 to the value list is an array in which the item value numbers of a certain column (that is, item) of the tabular data, that is, the pointers to the value list 110 are stored in the order of the record numbers of the tabular data.

By combining the above-mentioned pointer array 120 to the value list and the value list 110, when a certain record number is given, the pointer array 120 to the value list relating to a predetermined item corresponds to the record number. By taking out the stored item value number and then taking out the item value stored in the value list 110 corresponding to the item value number, the item value can be obtained from the record number. Therefore, similarly to the conventional data table, all data (item values) can be referred to using the coordinates of the record number (row) and the item (column).

For example, consider the tabular data shown in FIG. In this example, various item values are given to items such as customer ID, customer name, and telephone number. In the present embodiment, such tabular data is converted to the format shown in FIG.
Or (d) as an information block. For example, in FIG.
20-1 is associated with a value list 110-1 storing an item value indicating a customer ID. That is, the pointer value of the pointer array of the first record (record number “0”) is 0, and correspondingly, the item value “1” indicating the customer ID is obtained. In FIG. 3B, the pointer array 120-2 is associated with a value list 110-2 storing an item value indicating a customer name. For example, the pointer value in the pointer array of the first record (record number “0”) is “5”, and correspondingly, an item value “Yamada 男” indicating the customer name is obtained. 3C, similarly, it can be understood that the pointer array 120-3 is associated with the value list 110-3 storing the item values indicating the telephone numbers. Also, in each value list, it can be understood that the item values are ordered (in this example, ascending order).

Further, in the present embodiment, the value management table of the information block 100 stores, in addition to the value list 110, a class number flag array used for searching and counting, and a memory for storing pointers corresponding to item values. A start position array indicating the head address of the space and an existence number array are included. Each flag of the classification number flag array, and
Each existence number in the existence number array is associated with each item value. The flag value of the classification number flag is usually “0”, and is set to “1” corresponding to the item value to be found at the time of search or totalization. The number of existences corresponds to the number of records having the item value. The starting position is
It does not necessarily need to be provided because it corresponds to the sum of the number of existences corresponding to pointer values smaller than the corresponding pointer value.

FIG. 4 (a) shows another example of tabular data, and FIGS. 4 (b) and (c) each show "sex".
It is a figure which shows the information block regarding "age".
As shown in FIG.
In the value management table 210-1 of 00-1, the item values ("male" and "female") corresponding to each pointer value of the pointer array 220, the classification number corresponding to each item value, the start position, and the number of existences It is shown. For example, the number of records whose pointer value is “0” (that is, the item value of the value list is “male”) is 632564, while the pointer value is “1” (that is, the item of the value list is “male”). The number of records whose value is "female") is 369426
It is individual. The start position corresponding to each item value indicates the head address of a pointer array 230-1 to a record described later. The same can be understood from FIG.

An example of retrieval and counting using an information block having such a data structure and a process of generating an information block will be described below. FIG. 5 is a flowchart showing a search method for a single item. This processing is realized by the CPU 12 (see FIG. 1) executing a predetermined search program. In this example, "age"
A record whose item value is 16 years or 19 years is searched. First, of information blocks relating to tabular data, information block 2 relating to “age” shown in FIG.
00-2 is specified (step 501).

Next, the specified information block (hereinafter, referred to as the information block)
This is called a “specific information block”. ) Value list 210-
2. In the case where the item value matches the above search condition in (2)
The classification number of the row corresponding to (6 years old or 19 years old) is set to "1" (step 502). In the case of this example, the classification number of the row corresponding to the item value number “0” and the item value number “3” is set to “1”. Next, the start position and the number of occurrences corresponding to the line in which the classification number is set to "1" are obtained (step 503). Such information is referred to as pointer extraction information. In the array of pointers to records, a record number indicating a pointer to a record that matches the search condition is extracted based on the pointer extraction information acquired in step 503 (step 504). In this example, the pointer of the record corresponding to the item value number “0” is stored in the start position “0” of the pointer array to the record, that is, in the area from the beginning to the 45898th item. It can be seen that the pointer of the record corresponding to “3” is stored in the area of 1,89653 from the 2383137th position of the pointer array to the record. Finally, an array of the retrieved record numbers is created as a result set and stored, so that it can be used in the subsequent processing (step 505).

Next, a description will be given of a process of generating an information block for use in the above-described search process and the like.
FIG. 6 is a flowchart illustrating a process for creating an information block based on tabular data. First, the system 10 obtains original data in the form of a table and decomposes the data into items (step 601). This original data may be, for example, the data shown in FIG. 7A or the data shown in FIG. 7B. These original data may be supplied from the outside, or may be stored in the fixed storage medium 18. Steps 602 to 604 described below
The processing block 610 consisting of indicates the generation of an information block for a certain item. Therefore, when generating information blocks relating to a plurality of items, the processes corresponding to the processing blocks 610 are executed by the number of items. Hereinafter, an explanation will be given by taking an information block of an item relating to “sex” as an example.

First, an area for an information block of an item relating to “sex” is secured, for example, in the RAM 14 (step 602). Next, a value management table is generated in the secured area. More specifically, first, the value management table is initialized. Next, by manipulating the data relating to “sex” from the beginning to the end of the original data, it is possible to find out how many item names exist and how many. In this example, it is found that there are only 369436 and 632564 item names of “female” and “male”, respectively. As a result, the item values “female” and “male” are set in the value list, and a predetermined number is also set in the existence number array. Thereafter, the item values are sorted according to predetermined criteria. At the time of sorting, the number of items is also rearranged according to the rearrangement of the number of items. Next, the value of the start position array is determined. This is obtained by accumulating the number of existences higher than the self by sorting.
Further, the value of the start position array is assigned to the value of the corresponding classification number array. This value will be used in the next step.
After the value management table is generated in this way, an array of pointers to records is generated. The size of the area of the pointer array corresponds to the sum of the number of existences. In this way, it is possible to create an information block relating to a predetermined item. This information block is generated in advance, and processes such as search and counting are executed using the generated information block.

Next, a tallying process using the computer system according to the present embodiment will be described. In this tallying process, in principle, the value (existence number) of an element in the existence number array of the information block (for example, see FIGS. 4B and 4C) according to the present embodiment may be used. FIG. 8 is a flowchart illustrating the counting process according to the present embodiment. This processing is realized by the CPU 12 (see FIG. 1) executing a predetermined tallying program.
FIG. 9A is a diagram showing an information block related to an item “school division” in the tabular data shown in FIG. 21A. Here, for each item value of the school classification, the number of records that exist is counted.

In the totaling process, first, the storage position number is initialized, and an area 90 of the existence number array having the same number of elements as the value list 901 with respect to this information block.
2 is secured (step 801). Initially, “0” is given as a value of all elements of the existence number array. Note that this storage position number coincides with the record number in the case of aggregation for all records. On the other hand, when a predetermined record is selected from all the records by the search, the selected records are placed in a predetermined order (for example, in ascending order).
In the array arranged in the table, the number assigned to each element (the record number of the selected record) in ascending order corresponds to the storage position number.

Next, the pointer values in the pointer to the value list corresponding to the storage location number are referred to in ascending order of the storage location number (step 802), and the pointers in the value list indicated by the pointer value are referred to. The value of the corresponding element in the existence number array is incremented (step 803). For example, regarding the storage position number “0”, the pointer value of the pointer array is “0”. Therefore, the element of the number-of-existence array corresponding to the first row (No. 0) of the value list is changed from “0” to “1” (see FIG. 9B). As for the storage position number “1”, the pointer value of the pointer array is “2”. Therefore, the element of the number-of-existence array corresponding to the third row (No. 2) of the value list is changed from “0” to “1” (see FIG. 9B).

By executing the processing of steps 802 and 803 for all storage location numbers (steps 804 and 805), for each item value of the value list, the number of records having the item value is determined in the presence number array. It is represented as an element (see FIG. 9C). Then, CP
U12 refers to the existence number array, specifies an element whose value is “0”, and finds an item value related to the element (step 806). After that, the CPU 12 creates an aggregation table (view) for the item, excluding the item value for which the element (existence number) in the corresponding existence number array is 0 (step 807). FIG. 10 is a diagram illustrating an example of a view obtained by such a method. FIG.
As shown in, by the aggregation processing according to the present embodiment,
It will be appreciated that a view can be obtained in which columns for non-existent item values have been removed. According to the counting method according to the present embodiment, when the number of records is “N”, the addition operation (count-up) of the elements in the existence number array is performed only N times.
, It is possible to execute the totaling at a very high speed.

Next, a description will be given of the cross tabulation according to the present embodiment. Also in this embodiment, FIG.
Regarding the tabular data shown in (a), a case is considered in which cross tabulation is performed on the item values of the item “school category” and the item “age”. In the cross tabulation, processing substantially equivalent to that shown in FIG. 8 is executed for each item. FIG. 11 is a flowchart illustrating cross tabulation processing according to the present embodiment. FIG.
As shown in (1), in this process, an existence number array is generated for an information block including items related to cross tabulation in an arbitrary order (step 1101). This processing is shown in FIG.
Steps 800 to 805 are the same. Next, an element whose value is “0” is specified with reference to the element of the generated existence number array, and an item value related to the element is found (step 1102). This processing is shown in FIG.
Step 806 is the same. After this is executed for all items (steps 1103 and 1104), a summary table (view) for a plurality of items is created except for the item values for which the corresponding element of the number array (number of occurrences) is 0. Is performed (step 1105). FIG. 12 shows step 110 in the case of executing cross tabulation on two items.
5 is a flowchart showing the processing of No. 5 in more detail. In the cross tabulation table, a matrix for a plurality of items is created, and in each element, the number of records associated with a combination of item values of the plurality of items is arranged. For example,
Considering the above two items, "age" and "school classification", (any element in "age,
The corresponding number of records is arranged for each combination of any of the “school divisions”.

Since the address of the memory is actually one-dimensional, the number of records of each element of the total table of a plurality of items is also arranged in a one-dimensional array. FIG.
FIG. 3 shows an example of the correspondence between the tabulation table including the school divisions and the ages, and the addresses of the respective areas in the table. For example, FIG.
In the array shown in FIG. 3, assuming that the addresses are increased one by one as the age increases, the addresses become like the numbers shown in the array of FIG. Therefore, in this embodiment, the elements of the multi-dimensional matrix
One-dimensional addresses are projected, and items whose addresses increase one by one are set as lowest-order items, and items whose address offsets are larger are set as higher-order items. For example, in the example of FIG. 13, the lower order item is “age (vertical item)”, and the higher order item is “school category (horizontal item)”. Even in the case of three or more types of items, items having a larger address offset are considered to be higher-order items.

Therefore, as shown in FIG. 12, in the view creation processing, the dimension is determined by referring to the item in the summary table (step 1200), and the item of the lowest dimension is set as the first processing target. (Step 1201).

Next, an area for an offset array having the same number of elements as the number-of-existence array is generated for the item to be processed which is focused (step 1202). Next, the storage position number is initialized (step 1203),
A basic offset value is calculated to arrange the values of the elements in the area for the offset array corresponding to the storage position number (step 1204). The basic offset value is “1” in the lowest (lowest) dimension. On the other hand, in the next dimension, the offset value of the one lower (lower) dimension and the number of elements other than “0” in the existence number array for the item of the one lower (lower) dimension are set. Product.

Next, an offset value is arranged as an element of the position corresponding to the storage position number (step 120).
5). More specifically, the offset value is initially “0”, and as the storage position number increases,
The offset value is added. However,
If the element in the existence number array corresponding to the storage position number is “0”, no offset value is added.
When the element of the corresponding number array is “0”, it is not necessary to calculate the offset value and arrange the obtained value. Such processing is executed for all elements (particularly, elements whose elements in the existence number array are not “0”) (see steps 1206 and 1207). Step 1
The processes of 203 to 1207 are sequentially executed from a low dimension to a high dimension (step 120).
9, 1210). Thus, an offset array can be created for each item.

For example, regarding the tabular data shown in FIGS. 21A and 21B, consider an information block relating to the item "school division" (see FIG. 9C) and an information block relating to "age". In these items, "age" was defined as a lower (lower) dimension, and "school division" was defined as a higher (higher) dimension. As a result, first, the processing of steps 1202 to 1207 is executed for the item “age”. FIG.
In (a), reference numeral 1400 denotes an offset array obtained as a result of the processing in these steps. FIG.
As shown in (a), the values of the elements in the offset array are “0”, “1”, “2”,... So that the corresponding elements in the existence number array take values other than “0”. It can be seen that the number has increased by "1".

After the processing for the item "age" is completed, the processing of steps 1202 to 1207 is executed for the item "school division". In FIG. 14B, reference numeral 1410 denotes an offset array obtained as a result of the processing in these steps. As shown in FIG. 14B, the values of the elements in the offset array are “0”, “5”, and “10” such that the corresponding elements in the existence number array take values other than “0”. And "5"
It can be seen that it is increasing at a time. This is because, in the existence count array relating to only one lower-order (lower-order) item, there are only “5” whose elements are not “0”.
The offset value is increased by "5".

When the offset array for each item is generated in this way, a matrix area (cross tabulation area) for creating a cross tabulation table is secured (step 1210). Here, the size of the area to be secured is a product of the number of elements having a value other than “0” in the existence number array of each item. For example, in FIG.
4A and 4B, the size is “5 ×
3 = 15 ". In the cross tabulation area,
The area in each matrix can be specified according to the address. That is, at least the following items:
Incrementing and higher-order lowest items are:
If the lower order items are the same, an address whose address is increased by the number of lower order items is calculated.

Next, a storage position number relating to a set of record numbers is initialized (step 1211), and the following processing is executed sequentially from the first record number. First,
For each item, in the pointer array to the list of values,
The pointer value at the position corresponding to the record number is specified (step 1212). Then, for each item,
The element (offset value) at the position indicated by the pointer value in the offset array is extracted, and these offset values are added (step 1213). The added offset value (addition offset value) becomes an address indicating the matrix area in the cross tabulation area. Therefore, an element in the matrix area that indicates the added offset value as an address is incremented (step 1214).

By executing such processing for all storage position numbers (see steps 1215 and 1216), the number of records corresponding to each matrix area in the cross tabulation area is given. For example, a case will be described in which the offset arrays shown in FIGS. As shown in FIG.
First, from the set of record numbers, the record number “0” at the head (storage position number is “0”) is extracted, and the corresponding pointer value “0” in the pointer array related to “school division” is specified. , "Age", the corresponding pointer value "0" in the pointer array is specified.

Next, the corresponding offset value “0” in the offset array indicated by the pointer value “0” in the pointer array for “school division” and the corresponding offset value “0” in the pointer array for “age” Are added. The added offset value “0” thus obtained
“+ 0 = 0” represents the address of the cross tabulation area. Therefore, the value in the matrix area (see reference numeral 1500) of the address “0” of the cross tabulation area is incremented.

For the second record number "1" (storage position number "1") of the set of record numbers, the pointer value "2" and the "age" The pointer value “10” in the pointer array is specified, and the offset values “5” and “2” in the offset array indicated by these pointer values are specified, respectively. Therefore, the added offset value is “5
+ 2 = 7 ", and the matrix area (reference numeral 1) in the cross tabulation area using the added offset value as an address
(See 502) is incremented. Similarly,
Similarly, for the third (the storage position number is “2”) record number “2”, the addition offset value “5+
3 = 8 "is calculated. In this way, for all the record numbers stored in the set of record numbers, calculation of the addition offset value, designation of the matrix area in the cross tabulation area using the addition offset value as an address, increment of the elements of the matrix area Is repeated, it is possible to obtain a tabulation table as shown in FIG.

As described above, according to the present embodiment, the presence number array of the value list is generated, the value (item value) having the existence number other than “0” is specified, and the record is recorded for each item. Each offset value is given corresponding to the number, and the address of the cross tabulation area is indicated by the added value (added offset value) of the obtained offset value.

Further, another example of the cross tabulation will be briefly described. For example, consider the tabular data shown in FIG. In the present embodiment, the tabular data is composed of information blocks of “school division”, “sex”, “age”, and “name”. Each information block includes a "pointer array to a value list" and a "value list" as described above (see FIG. 17). In FIG. 17, no value is given to the elements in the existence number array.

First, by narrowing down only the records of “male” in the item of “sex”, a set of record numbers and a table (view) shown in FIG. 16B can be obtained. If this is sorted in the order of the Japanese syllabary, the number of the record number and the table (view) shown in FIG. 16C can be obtained. For the narrowing down (searching) and the sorting, a method disclosed by PCT WO00 / 10913 devised by the present inventor can be used. By such narrowing and sorting, a set of record numbers (see reference numeral 1601) shown in FIG. 16C can be obtained. Next, referring to each record number in the set of record numbers shown in FIG. 16 (c), an existence number array related to “school division” and “age” is created (see FIG. 18). Hereinafter, after creating an offset array as shown in FIG. 19, FIG.
The number of corresponding records is arranged in the matrix area in the cross tabulation area shown in FIG.

Finally, the processing speed of the cross tabulation according to the method according to this embodiment will be considered. The number of records to be cross-tabulated is “N”, the number of field values of one item (for example, “row” in the cross-tabulation table) in the case of two-dimensional cross-tabulation is “R”, and the other item ( For example, it is assumed that the number of item values of “column” in the cross tabulation table is “C”. In this case,
When the method according to this embodiment is used, (1) “2 × N” count-ups for “generation of array” and “R + C” for “generation of offset array”
It can be seen that only "N" count-ups are required for (3) "Creating a cross tabulation table". Therefore, although it depends on the number of R and C, it is sufficient to execute the count up about “3 × N” times. Worst case (ie, R = C = N)
However, it is sufficient to count up “5 × N” times.

On the other hand, according to the conventional method, “2 × C
× R ”calculations were required. Here, if “C ≒ N” and “R ≒ N”, the calculation amount is O
(N ² ). Therefore, it can be understood that the cross tabulation method according to the present embodiment is significantly faster than the conventional one, except when R and C are sufficiently smaller than N.

Further, according to the conventional method, it is unknown which element is “empty” (that is, the value is “0”) in the cross tabulation, so that the memory area for the huge cross tabulation is once used. Had to be secured. (For example,
If “100,000” values exist for one item and “1000” values exist for the other item, an area for storing only “100,000 × 1000 = 100 million” elements Need to be secured once. Here, if the memory area cannot be secured, the tallying itself may not be possible. On the other hand, according to the present invention, an area whose element (value) becomes “0” is not formed in the cross tabulation area, and the possibility that the memory area cannot be secured is significantly reduced. It becomes possible.

The present invention is not limited to the embodiments described above, and various modifications can be made within the scope of the invention described in the claims, and these are also included in the scope of the present invention. Needless to say, this is done. For example, in the above embodiment, one-dimensional aggregation and 2
Although the counting of dimensions has been described, it is clear that the technique of the present invention can be used for counting of three or more dimensions. Here, consider the projection of addresses in the case of multi-dimension.

When a total table for “I” items is obtained, the number of item values of each item is Ni (0 ≦ i ≦ I−1).
And the number of elements whose value is 0 in the existence number array of each item is represented by ni ((0 ≦ i ≦ I-1). Here, the offset array of each item is offset value Pi _m (although, 0 ≦ m ≦ Ni-ni ) _{is, Pi 0 = 0 Pi (m} + 1) -Pi m = Π (nk-nk) (0 ≦ k
.Ltoreq.m-1). That is, in a certain dimension "i",
The difference between adjacent offset values n is a value obtained by sequentially multiplying the number of elements whose corresponding number is not “0” among the elements of items of lower dimensions than the self.

Further, specific values given in the offset array are not limited to those shown in the above embodiment.
That is, when the value in the matrix area is specified by one or more addresses in the area reserved for the cross tabulation area (that is, when the difference between the addresses of adjacent matrix areas is 2 or more) Can be obtained by multiplying the offset value by “k” (k: the difference between the addresses).

Further, in the above-described embodiment, a predetermined program is read into the general computer system 10, and the program is executed, thereby joining a plurality of tabular data and processing the joined tabular data. However, the present invention is not limited to this. A board computer dedicated to database processing is connected to a general computer system such as a personal computer, and the board computer can execute the above processing. Needless to say, such a configuration may be adopted. Therefore, in this specification, means does not necessarily mean physical means, but also includes a case where the function of each means is realized by software. Further, the function of one unit may be realized by two or more physical units, or the function of two or more units may be realized by one physical unit.

[0053]

According to the present invention, it is possible to provide a processing method capable of reducing the amount of memory used and achieving high-speed cross tabulation.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a hardware configuration of a computer system capable of implementing cross tabulation and the like according to an embodiment of the present invention.

FIG. 2 is a diagram showing information blocks used in the present embodiment.

FIG. 3 is a diagram illustrating an example of tabular data and an example of an information block based on the tabular data.

FIG. 4 is a diagram illustrating another example of tabular data and another example of an information block based on the tabular data.

FIG. 5 is a flowchart illustrating a search method for a single item.

FIG. 6 is a flowchart illustrating a process for creating an information block based on tabular data.

FIG. 7 is a diagram illustrating an example of original data for creating an information block.

FIG. 8 is a flowchart illustrating a tallying process according to the embodiment;

FIG. 9 is a diagram showing an example of a certain information block according to the present embodiment.

FIG. 10 is a diagram illustrating an example of a view obtained by a tallying method according to the embodiment;

FIG. 11 is a flowchart illustrating a cross tabulation process according to the embodiment;

FIG. 12 is a flowchart showing the processing of step 1105 in a case where cross tabulation is performed on two items in more detail.

FIG. 13 is a diagram showing an example of a tabulation table composed of school divisions and ages generated according to the method according to the present embodiment.

FIG. 14 is a diagram illustrating an example of an information block including an offset array according to the present embodiment.

FIG. 15 is a diagram for explaining creation of a tally table using the offset array according to the present embodiment;

FIG. 16 is a diagram for explaining a process for creating another cross tabulation table according to the embodiment;

FIG. 17 is a diagram illustrating a process for creating another cross tabulation table according to the embodiment;

FIG. 18 is a diagram illustrating a process for creating another cross tabulation table according to the embodiment;

FIG. 19 is a diagram for explaining a process for creating another cross tabulation table according to the embodiment;

FIG. 20 is a diagram illustrating a process for creating another cross tabulation table according to the embodiment;

FIG. 21 is a diagram illustrating an example of a cross tabulation;

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 Computer system 12 CPU 14 RAM 16 ROM 18 Fixed storage device 20 CD-ROM driver 22 I / F 24 Input device 26 Display device

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G06F 17/30 170 G06F 17/30 170H 180 180D 340 340D

Claims (8)

[Claims] 1. A data summarizing method for summarizing a plurality of tabular data each represented as an array of records including an item and an item value included therein, wherein each of said tabular data is A value list in which the item values are stored in the order of the item values corresponding to the item values belonging to the specific item, and a pointer value for indicating the item value number in the order of the unique record number are stored. And arranging the number of records having the same item value number in the information block as the value in the value list. Providing an existence number array arranged in correspondence with the above, finding an item value whose existence number is 0 in the existence number array, and removing the found item value. In, data aggregation method characterized by comprising the steps of creating a schedule consisting existence number corresponding to the item value in the item. 2. Tabular data, each represented as an array of records, each containing an item and an item value contained therein,
A data aggregation method for performing aggregation processing on a plurality of items, and for each of a plurality of items used for aggregation, a value in which the item values are stored in the order of the item value numbers corresponding to the item values belonging to the item A step of dividing the data into one or more information blocks consisting of a list and a pointer array storing pointer values for indicating the item value numbers in the order of the unique record numbers; and Providing an existence number array in which the number of existences indicating the number of records having the same item value number is arranged in association with the value in the value list; and in each of the existence number arrays, A step of finding an item value that is 0; and setting the number of each element of the matrix-like tabulation table including the plurality of items to correspond to an address. Projecting a multidimensional position of the element onto a one-dimensional position; and in each of the existence number arrays, at least an element corresponding to an element whose existence number is not 0, wherein the element is included in the projection. Generating an offset array that becomes an offset value based on the record number, and referring to a corresponding pointer value in the pointer array for each item based on the record number, thereby obtaining an offset value in the corresponding offset array. Identifying, adding the obtained offset value to calculate an added offset value, calculating the address by associating the added offset with the projection, and calculating the address indicated by the obtained address. Counting up the value, based on the value of the position indicated by the finally obtained address. Data aggregation method characterized in that a total of Table obtained. 3. The number of item values of each item is Ni (0 ≦ i ≦
I-1: I is represented by the total number of items related to aggregation, and the number of elements having a value of 0 in the existence number array of each item is represented by ni ((0 ≦ i ≦ I−1)). when it is, the offset value Pi _m offset arrangement of each item (except, 0 ≦ m ≦ Ni-ni ) _{is, Pi 0 = 0 Pi (m} + 1) -Pi m = Π (nk-nk) (0 ≦ k
≦ m−1) 3. The data summarizing method according to claim 2, wherein: 4. A step of selecting a record number which can be counted from among the record numbers prior to providing the existence number array, and a record number in which the selected record numbers are arranged in a predetermined order. Generating a set, and referring to a corresponding pointer value in the pointer array based on a record number included in the record number set, generating the existence number array. The data aggregation method according to any one of claims 1 to 3, wherein: 5. A computer readable by a computer storing a program for implementing a data summarizing method for summarizing a plurality of tabular data represented as an array of records each including an item and an item value included therein. A storage medium, wherein the tabular data, each value list in which the item value is stored in the order of the item value number corresponding to the item value belonging to a specific item, and in the order of a unique record number, A step of dividing the data into at least one information block consisting of a pointer array storing a pointer value for indicating the item value number, and a record having the same item value number in the information block Providing an existence number array in which the number of existences indicating the number of items is arranged so as to correspond to the values in the value list; A program that includes a step of finding a certain item value; and a step of creating a tabulation table including the number of items corresponding to the item value in the item with the found item value removed. Storage medium. 6. Tabular data, each represented as an array of records each including an item and an item value included therein,
A computer-readable storage medium storing a program for implementing a data aggregation method for performing aggregation processing on a plurality of items, wherein each of a plurality of items used for aggregation corresponds to an item value belonging to the item. One or more information blocks including a value list in which the item values are stored in the order of the item value numbers, and a pointer array in which pointer values for indicating the item value numbers are stored in the order of the unique record numbers And a presence number array in which, for each of the items, the number of occurrences indicating the number of records having the same item value number is arranged in correspondence with the value in the value list. A step of finding an item value such that the number of occurrences is 0 in each of the existence number arrays; Projecting the multi-dimensional position of each element to a one-dimensional position, in order to make the number of each element of the liquor-based summary table correspond to the address, Generating an offset array including at least an element corresponding to an element whose number of existence is not 0, and the element having an offset value based on the projection; and, for each item based on the record number, the pointer Specifying an offset value in the corresponding offset array by referring to a corresponding pointer value in the array; adding the obtained offset value to calculate an added offset value; Calculating the address by associating with the projection, and counting up the position value indicated by the obtained address. To a step, the finally obtained based on the value of the position shown in the address summary table storage medium storing a program configured to obtain. 7. The number of item values of each item is Ni (0 ≦ i ≦
I-1: I is represented by the total number of items related to aggregation, and the number of elements having a value of 0 in the existence number array of each item is represented by ni ((0 ≦ i ≦ I−1)). when it is, the offset value Pi _m offset arrangement of each item (except, 0 ≦ m ≦ Ni-ni ) _{is, Pi 0 = 0 Pi (m} + 1) -Pi m = Π (nk-nk) (0 ≦ k
≦ m−1) The data summarizing method according to claim 6, wherein: 8. A step of selecting a record number that can be counted from among the record numbers prior to providing the existence number array, and a record number in which the selected record numbers are arranged in a predetermined order. Generating a set, and referring to a corresponding pointer value in the pointer array based on a record number included in the record number set, generating the existence number array. The storage medium according to claim 1, wherein the storage medium stores a program to be executed.