JPS6134626A - Method of executing external distribution and sorting - Google Patents

Abstract

(57) [Summary] This bulletin contains application data before electronic filing, so abstract data is not recorded.

Description

DETAILED DESCRIPTION OF THE INVENTION Field of the Invention The present invention relates to a computer-implementable distribution method for externally classifying extremely large files.

[Prior Art] Classification (sorting) is a method of arranging items in a certain "order." Of interest are records, each having an associated key value. The purpose of classification is to determine the permutation of records whose key values have been entered into non-reducing classes. A feature of the classification task is the manner in which the reordering is performed and that this is done entirely within internal memory that is local to the CPU performing the classification, i.e. where the data fits neatly into the CPU's random access internal (main) memory. It depends on whether it is done or not.

"Distribution classification" refers to dividing records into adjacent ranges such that all records in one range have a key with a smaller value than the key of the record in the next range. It is. On the other hand, "combination (merge) classification J is a list of two or more size-ordered lists 1 to 1.
The idea is to combine them so that the combined list is also arranged in order of size. A typical example is a two-way merge sort, where paired items are compared, each part is ordered, these pairs are then combined, and the resulting quadruple numbers are ordered, 4
The multiple numbers are combined into a sorted octuplet number, and this continues until no more combinations can be made.

"External classification" is a classification technique applied to files of data that exceed the capacity of primary or internal memory,
Reliance on secondary storage devices such as DASD (Direct Access Storage Devices), tapes and drums during the sorting process. In combinatorial classification, a form of external classification, parts of a file are read into internal memory, ordered internally, and
Then, it is written again to an external device, that is, a secondary storage device.

In the "exchange-select" technique, an intermediate file containing one or more ordered lists (strings) is created from an unordered "input file." Swap-select creates ordered strings of varying lengths, the average length of which is twice the capacity of internal memory. See US Pat. No. 2,983,904. Optimal combination of strings into ordered strings can be performed by forming a "minimal combinatorial Huffman tree." The minimum combinatorial tree is made up of terminal nodes that represent the lengths of strings, and is arranged so that the values of the combinatorial tree are as small as possible.

Most external classification methods used for data stored on disk drives are combination-based. 2, these require using swap-selection to generate several initial stored strings, and then repeating the combinations until only one string is left. US Patent No. 4
No. 210,961 "Sorting System" describes a typical combination-based external classification method.

13= On the other hand, associative memory devices are used for both classification and search applications. Chang et al,'As5oc
iative 5earch Bubble Devi
ces for Content Addressable
e Memories”, 18, IBM Technic
al Di, scl, osure Bulletin,
pp. 598-602, Jul, y 1975, describes a magnetic bubble memory device as a word- and bit-wise content-addressable memory. Also, U.S. Patent No. 4168535 “Non-Volatile”
Bubble Domain Memory System
m” has multiple small/large loops, bit storage,
array is described. Similarly, Dorty at
al,”Magnetic Bubble Memory
yArchjl: ectures for Suppo
rting AssocjatjveSearchjn
g of Re1atjonal Databases
”29, IEEE Transactjons on
Computers, pp, 957-970, N
Ovember l 980 substantially extends the magnetic bubble memory parallel architecture to support associative search in relational databases. C, S, Lj, n, “SOrting
ujth As5oactive 5econdar
y Storage Devices”, Processee
dings of AFIPS+ National
Computer Conference,
1977, pp. 691-695 describes a computer-implementable method for performing histogram-based distributional classification of keys and associative searches on head-per-track disks. This method only works if the key values are distributed on average.

[Problems to be solved by the invention] The conventional external classification method described above is based on combination (merging), and after generating several initial storage strings using exchange and selection, It was necessary to repeat the combination (merge) until there was only one column. For this reason, it takes a long time to classify, and it is difficult to apply this method to large files.

[Means for Solving the Problems] An object of the present invention is to devise an external classification method that can be applied to extremely long files that are accessed associatively through a secondary storage device and that can significantly reduce the operating time of the classification function. It's there.

The above objective is achieved by performing a distribution classification. The data to be rearranged includes records stored with keys that can be accessed in the content addressable secondary storage device. The method of the present invention utilizes random sampling of a predetermined number of keys and internal classification of the sampled keys to create a histogram of keys within a range of key values and combine less dense adjacent ranges. forming in a single pass partitions of the same size of records, each of which fits into the main memory of the CPU;
Perform an associative search for all records in the @concave circle with the key,
This is to internally classify searched records.

This method can be applied extremely quickly, regardless of the order of keys or the distribution of key values. After internal sorting of the sampled keys, a sequential pointer list is constructed that points to the keys in the order in which they were stored. In this context, creating histograms of keys and ranges of keys to form partitions of records involves performing a binary search of internal memory in a list of pointers for each key to determine the range associated with the key. Includes checking. The present distribution method can be advantageously optimized with respect to content addressable storage devices, and not with respect to mere conventional DASDs.

[Embodiment] The method of the present invention includes at least one CPU, each of which has an internal memory, an input/output channel, a control device,
It can be implemented on computer systems that have direct access storage devices and other input/output devices connected thereto. Such a system is called “Data Process
U.S. Patent No. 340 titled “sing System”
It is described in No. 0371.

The system described herein includes, as resources, all of the mechanisms required to carry out the process, either a computer system or an operating system running on this system.

Typical resources include internal memory, input/output devices, CPU, data sets, and control or processing programs.

A "classification" as a set of functions can be invoked by any executing application process or database management system. It is typical for an executive function to invoke a classification by file name. If the operating system locates the file and it does not fit into internal memory, the "external sort" function is called.

The method of the invention requires an associative secondary memory. This device is accomplished either by large DASD with logic-per-track functionality or by magnetic bubble memory (MBM).

The method is divided into three phases: (1) sample
(2) Range FHI (bucket 1-) formation phase, and (3) internal classification phase.

In the sample phase, a predetermined number of key values are randomly sampled and the sampled keys are internally classified. During the range formation phase, in a single pass through the file, a count is taken of how many records belong to each range as defined by the classified sample. Using this histogram, adjacent ranges are combined to form large ranges such that each range contains approximately the number of records that will fit in internal memory. Finally, in the internal classification phase for each range in increasing key value order, an associative search of the file is performed to search internal memory for all records with key values within that range. The records are then classified internally and
appended to the output file.

An analysis of the modified method to improve time/space efficiency, use of magnetic bubble memory as an associative secondary storage, and improvement of the effectiveness of the internal classification phase is also detailed.

In this specification, we first assume that each record has a fixed length of R bytes, while the key field consists of K (R bytes). The CPU has an internal memory size of M bytes. and the number of files classified is N
shall consist of records. Furthermore, it is assumed that the number F of records that can be classified internally is approximately M/R. Finally, assume that the number S of keys sampled is approximately M/.

Samples of the keys in eight sample phases (where S is approximately M/) are taken using a random sampling method. The choice of the value of S, which also determines the amount of internal memory M required, is
This is the core of the method and will be explained below. This sample stored in internal memory is AV
It is stored using a balanced tree, such as an L or RB tree. D. E. Knuth, “The Art.
of Computer Programming”
, Vol, ume3: “Sorting and S
Addison-Wesley
, 1973. Classification by balanced trees allows the classification time to partially overlap with sampling. For completeness, a "tree" is preferably a connected graph with no cycles.

Furthermore, a directed tree is a directed graph that does not contain any cycles or alternative paths. A directed tree is a directed tree whose descendant set consists of all other nodes. have.

In the present invention, each node of the balanced tree requires storage space for the key as well as two pointers for classification. Incidentally, the predicate ancestor-descendant is used to describe the relationship between nodes. Therefore, it can be said that the key value of the descendant on the left is less than or equal to the key value of the parent. Similarly, the key value of the right descendant is greater than or equal to the parent's key value. An example of a balanced tree configuration of keys and pointers is shown in FIG. The final part of the sample phase requires a sequential traversal of the balanced tree to create a sequential list of pointers pointing to the keys of the tree in ascending order. A list of pointers corresponding to the balanced tree of FIG. 1 is shown in FIG.

As mentioned above, this step includes random sampling of eight keys and internal classification of the sampled keys. In content addressable storage in the form of magnetic bubble memory, each record is stored in a random array of MBMs. The array is shown in FIG. 3 and will be described below.

Sampling is done while records are being written to the MBM. If the records of the input file already reside in the MBM, they are assumed to be stored in a random array. If not resident, they must be randomly encrypted within the array.

Sampling in such a device is performed as follows.

First, the number of records selected as samples from each array is randomly selected based on a multinomial distribution. The generation method of independent polynomial random variables is D, E, Knuth
,”The Art of ComputerProg
ramming”, Volume 2: “Seminu
mericalAlgorisms”, Additio
son-Wasley, 2nd edition.

1981. An ordered random sampling method is then used to select the desired number of keys from each array. In this regard, J.S.

Vitter, “Faster Method for
r Random Sampling”, Technj
cal Report C5-82-21, Brown
University.

See August 1982. The sampled =12- key values are then read into internal memory and inserted into the balanced tree as described above.

(Packet Multi Phase This phase is divided into a computation sub-phase and a join sub-phase. The sampled key values sorted in the sample phase serve to partition the file into S+1 range partitions. In this case, The classified key values are expressed as Xl, X2..., XS. Also, xo is set to -, and X841 is set to +.1
For each value i in the closed interval from to S+1, the i-th range is defined to be the set of records whose key values are in the semi-closed interval xt through

Each range contains an average of N/S records with a standard deviation of approximately N/S. The value of S is chosen such that N/S is much smaller than the number of records F that can be placed in internal memory. This will be discussed below.

The storage order of keys can be represented by a sequential pointer list created at the end of the sample phase.

Importantly, the counting subphase processes each key in associative memory and performs either a Fibonacci search or an equibinary search on the pointer list to see which of the smaller ranges contains the key. This is what happens.

The count in this range is 1. Increased by increments.

Since the pointer field of the balanced tree is no longer needed, the range count can be stored in the tree of pointer fields of the keys that define the upper bounds of the range. This is true except for the highest order range counts, which require a separate storage location.

The order of steps in the counting subsystem is explained with reference to the Pascal pseudocode in Table 1.

Table 1 Subphases of Range Phase (Init
(jalize the Counts) for i:=
1 to S+1 do P[il, count:=O
;(Process each key and 1n
cre+nent its range cc+unt
) for aach key in the file
d.

egin Perform a binary 5earch o
n the pointer ust P in or
der to find the value i 5u
ch that P[i-1], key<“key v
al,ue”<P[il,key;if P[il,c
count<V then P mouth], count:=P[
il, count+1 end; In this case, the pointer list created at the end of the sample phase is denoted by P. Each element of P is the address of a record with a key field and a count field. For each i between 1 and S+1, the P[il key is the i-th sampled key in the stored order. The key with value P [S+1] is
+ω, which is any number greater than the maximum possible value of the key. For each 1, the maximum value V that can be stored in P[il is greater than F, which is the number of records that can fit into internal memory during the internal classification phase.

The purpose of the coefficient subphase is to calculate the value of P[i]Kakarato for each i between 1 and S+1, if the number of records is at most V, the key value is in the interval (P[1-11 key The key value is set to the number of records where key value≦P[i cokey), and otherwise to the value V.

In the merging subphase, adjacent ranges are combined to form larger ranges. Each grouping combines as many small ranges as possible, such that the resulting combined range contains at most F records. Each resulting range will most likely be equal to F. Incidentally, a record containing more than F records is called an "overflow range." It is then established that the mean and standard deviation of the overflow range are less than one. Also, the keys defining the partitions for the =16- range are output sequentially to known disk or tape if there is no space in the internal memory. In the case of an overflow range, a count or special marker is output along with the key that defines the upper limit of the range.

How to perform the join subphase is explained with reference to the Pascal language pseudocode sequence in Table 1.

■Tabular form phase Musabfeez end: In this Pascal-like order, each j between 1 and S
The value P[il for is the sampled key value in the stored order. P[0] key: =-ω and P
It is assumed that [S+1] key=10 (1), which is a number that is both smaller and larger than any possible value of the key. The maximum number that can be stored in P[i] for each i is represented by ■.

The count of P [il for a value of j between 1 and S+1 is the maximum number of . The value of ■ is greater than F, which is the number of records that can fit into internal memory during the internal classification phase. To simplify programming, a special count field P[
S+2] count.

The output of the join phase is a list of key values that define the endpoints of the joined ranges.

If the records are of variable length rather than fixed length, the method can be modified as follows. In the coefficient subphase, the count for each range counts the total space occupied by records within this range, rather than the number of records within this range. In the merge subphase, as many adjacent ranges as possible are merged so that each record of the resulting range can fit into internal memory.

In this phase, the keys defining the range are processed in sequence. For each range, the records within it are searched by an associative search, sorted, and then appended to the output file. If the range is not in the overflow range, all of its records fit into internal memory and can be sorted internally. In this case, exchange selection is used to classify the records within the range, since each range does not need to be reinitialized. Once the records within the range are retrieved from the secondary storage, the search for the next range of records begins. This depends on all key values in the first range being less than or equal to all key values in the second range, and on the records in the range being able to fit into the exchange selection tree. In the rare case that a range is an overflow range, the range classification can be performed either by applying the method of the invention recursively or by using known classification combinations. As mentioned above, the number of such overflow ranges is always less than one.

Sample Su 4 Nu Sing J1 Pyin 1 Shi Sen 艷 u” 1 Ren Su y □
The number of sampled keys S is chosen to minimize the amount of internal memory required, with the objective that the standard deviation of the average number and the number of overflow ranges should be less than one. Ru. The values of M and S are related by the following equation.

(],) S= (M-2B)/(K+3P) The values of M and S are expressed as two variables M and r (where r=E (
It can be calculated by solving two equations: S+1)(N+1). The first equation that ensures that the above objective can be achieved is:

However, B and B' are the sizes of human output buffers,
P is the number of bytes per pointer field. The second equation is:

(3) (+3P to M-2B+) (M-2(B+B'
))=r(N+1)(R+P)(K+3P) This formula ensures that the sampled key S can be stored internally. The approximate value of M obtained by solving these two equations is as follows.

If more internal memory is available than the value of M calculated above, for example kM bytes of internal memory, the total classification time is reduced by approximately a factor.

An associative secondary memory implemented in a magnetic bubble memory processes queries of the following form (programming language) called associative searches or range queries.

Gjven values a and b+retr
jeve a], 1 recordssuch tha
t a “key value”≦b This query is performed once for each range during the internal classification phase of the method. This results in a significant reduction in execution time. Magnetic bubbles and associative secondary storage devices in database applications
For more information on memory usage, see Chan, Doty, supra.
and Lin references.

FIG. 3 illustrates a magnetic bubble memory (MBM) secondary storage device that includes multiple carts forming a hierarchy of several levels. At the highest level, memory consists of several boxes with a capacity of several hundred megabytes. A box can be partitioned into boards, which contain several modules containing several chips. The key storage device is called an array. Figure 3 shows some Ares 15.17.19. Each array has a plurality of magnetic bubble loops 21.23.25.27. It is desirable that each array have a large storage capacity. R for array
AM buffer 3 is combined. Buffer 3 serves as a read or write buffer for several magnetic bubble arrays.

Approximately 2000 bits of RAM are used for each 1 megabit array. From array 15 or 17 or 19, R
By transferring each record to AM buffer 3,
An associative search is performed. Microprocessor 33 coupled to RAM 3 selects and transfers from pass 2 to pass 1 all records whose key values satisfy the range introduction. Path 3 coupling array 15 to RAM 3
0 and 32 are driven at a speed of 1 megabyte/second.

Assuming that array 15 can release data from the loop to R2H at this rate, as explained below, the entire contents of the array can be searched in one second. Advantageously, all arrays can be searched in parallel, resulting in a total time per associative search of 1 second.

A typical 1 megabit array 15 consists of up to 1000 synchronous small loops 21.23.25.27. Each loop contains a magnetic bubble coded to represent 1000 bits which rotates the loop in response to a varying magnetic field.

In this case, the records are stored in a small loop. That is, the bits of the record are stored at the same relative position in the loop, one bit per loop.

Records containing more than 1000 bits are stored by expanding them into several arrays or by partitioning them into contiguous 1000 bit sections. In either case, each bit in the section is stored in a separate small loop in the same relative position.

Assuming that the record is expanded into several arrays if necessary, all bits of the record will be synchronous and arrive at the same point in the loop at the same time. Each read access of a small loop is considered non-destructive.

Reading is done by copying each bit and loading the copied bits into a 1000 loop read buffer. If read shift register R8R29 is empty and available, bits can be loaded into R2H. Next,
The bits are shifted sequentially by the read head 9.

Write operations are performed in a similar manner using write shift register WSR31.

The time required to cycle through the small loop is equal to the time it takes to move the OOO bit to R2H by the read head. This is approximately 0.001 seconds. Shift is R8
When completed in R, the 1000 loop read buffer has:
This gives you time to load the next record. Therefore, the entire file can be loaded and read into R8R in 1000XO,0O1=1 seconds. This corresponds to the time it takes to transfer 1 megabit from array 15 to RAM 3. The time per associative search is therefore 1-second.

If two arrays, such as 15 and 17, share a single read path 30, the associative search time would be 2 seconds instead of 1 second. It was found that an associative search of about 1 to 2 seconds is extremely appropriate for classification performance. Using the marking techniques described below, the effective associative search time can be reduced to a fraction of a second.

Economics can be achieved both in terms of modified execution time and in the amount of internal memory required to support the external distribution classification method of the present invention. The method generally involves three steps. These steps are: (1) random sampling of the 8 keys and internal classification of the 8 sampled keys; (2) each can fit into internal CPU memory;
(3) performing an associative search for all records whose keys fall within the range and internally classifying these records; be.

In the random sampling and internal classification steps of the sampled keys, which assume a uniform distribution of keys, speed can be increased by the fast splitting that occurs when the keys are uniformly distributed. This is accomplished by performing the following operations on a predetermined portion of internal memory. First, the range of 256 that a key can take is divided into approximately c N R/M equally sized intervals. However, C is a constant of c>1. As each key is processed, the counter for the corresponding interval is incremented. If the first few bytes of each key result in near-uniform partitions, the path to the file in the range formation phase can be omitted and replaced with a uniform interval count.

The above search required 1 to 2 seconds to perform an associative search for all records whose key values were within the range and to internally classify these records. However, the path coupling the magnetic bubble memory content addressable memory to the CPU can reduce the time it takes to transfer Mbytes (one load into internal memory) to a fraction of the above. Approximately one record for every NR/M record in the file is known to belong to the predetermined range. Therefore, most of the overall search for records within a given range involves processing records that do not satisfy the range query. Search speed can be increased by avoiding searching most of the records. M
It is advantageous to use one special bit per record stored in the BM and used as a "mark bit". Even if you mark a record,
You don't have to attach it. change the associative memory architecture,
It must be ensured that only marked records are transferred from the large loop to the read shift register 29. The programming language for queries to obtain keys within a predetermined range of key values takes the form:

Given values a and b retr
ieve all marked records in
the semi-closed range a“
keyvalue”≦b.

For purposes of illustration, assume that there are 11 ordered key values that divide the file into approximately equally sized regions. The K-1 ordered key values are obtained during the join subphase of the range formation phase. The internal classification phase can be divided into K consecutive subphases.

In this case, the i-th sub-face includes searching and classifying each range within the i-th region formed by K-1 adjacent key values. At the beginning of the i-th subface, if all records in the i-th region are marked and all other records are unmarked, approximately N/ records are marked at once. , and thus the number of records processed in each range search will be reduced by a factor. Each of the K processing steps requires one complete associative search. The total processing time is negligible.

A further improvement can be made if during each associative search, the marked bits of all records satisfying the range query are turned off. Therefore, at the beginning of each subphase, N/ records are marked, but the number of marked records decreases linearly until it reaches zero at the end of the subphase. Become. This is shown in FIG. However, the average number of records processed in each associative search will be approximately N/(2K).

Note that the range query can be rewritten as the following query, which only requires one comparison instead of two.

Given values b, retrieve
and unmark allmarked
records 5uch that “ke y
v a ], ue ”≦b+ This form of query is
This is equivalent to the previous query because while the range (a "key val, ue"<= b) is being processed, the mark is removed from key values with key values less than or equal to a.

[Effects of the Invention] Although preferred embodiments of the invention have been described, it is to be understood that various modifications may be made in accordance with the principles of the invention. For example, experience has shown that it is preferable for RAM 3 to have extra storage for extra marked bits or for queuing addresses. Additionally, each logical record in the input file is
MBM associations can be created by either placing very small logical records in the MBM, such that they span several physical records, or by placing very large physical records, such that each physical record contains multiple logical records. Maximize the utilization of storage space. The above marking techniques are more efficient when the size of the physical record is small compared to the size of the logical record.

If the time per associative search and the final output time for each stored range are short enough, the running time of the internal classification phase is dominated by CPU time and not by I/O time. In this case, the internal classification phase can be made much faster by allocating approximately twice as much space in internal memory and splitting it in two. The capacity of the internal memory is M'=2 (M-2(B+
B' )) increases. However, M is the previous internal memory capacity.

Since the internal range of each range is static rather than dynamic, instead of an exchange selection, almost two
Note that you can use static internal classification methods like quicksort or Radix, which are twice as fast. CPU time for the internal classification phase is reduced by approximately 50 percent.

Currently, large databases are stored on DASD in a run-32-dumb access fashion, which uses hashing and indexing techniques to speed up search times. The MBM technology described above allows databases to be stored in the MBM and take advantage of faster random access. The same hash and indexing techniques can be used. This type of database system is much faster and faster than database systems stored on DASD.
- Able to process ranzagions.

Furthermore, the database stored in the MBM can quickly execute general relational database queries in about one second by performing associative searches. This could not be achieved if the database was stored on DASD. In other words, the MBM can be used as either a random access device or an associative device for the current transaction or query, whichever is more efficient. Taxonomy can be considered as one of the common operations of relational databases.

[Brief explanation of the drawing]

FIG. 1 is a diagram of a balanced tree of sorted and sampled keys, including the necessary pointers created during the sample phase. FIG. 2 is a diagram of the pointer list created during the sample phase. FIG. 3 is a diagram of a typical associative memory card for carrying out the associative search of the method of the present invention. FIG. 4 is a diagram of the marking technique implemented. 3...RAM buffer, 5...Error code test, 9...Read, 11...Write, 1
5.17.19...Array. Applicant International Business Machines Corporation

Claims (1)

Claims: A method for performing external distribution classification for reordering data including keyed records accessed from an associative secondary storage device by a CPU with available internal memory, comprising: S By performing a random sampling of keys, internally classifying the sampled keys within the CPU, and obtaining a histogram of keys and a range of key values, we form partitions of equal size in a single pass, and each Section is C
combining adjacent low-density ranges to form a larger range to fit into the internal memory of the PU; performing an associative search for all records whose keys fall within the range of the partition and internally classifying the records; A method of performing external distribution classification consisting of and .