JPH06103459B2 - Sorting method and apparatus - Google Patents

Description

Description: TECHNICAL FIELD OF THE INVENTION The present invention relates to a method and apparatus for sorting records in a data set stored in a direct access external storage such as a disk pack device.

[Conventional technology and problems]

Sorting data on an external storage device according to a given order relational rule (sorting rule) is generally called an external sort. The external sort is a routine processing routinely performed in banking applications, clerical work, database applications, and other fields. A general-purpose sort / merge program is provided for external sort processing. However, no matter which sort / merge algorithm is adopted, the data recorded in the external storage will pass through the external storage control device and channel device. Need to be transferred to main memory and sent back again. The rate of channel use for sorting varies depending on the execution environment, but it is said that it may account for more than 20% in the field of business calculation and database-related processing. If the channel occupancy rate for external sorts is reduced, the channel utilization efficiency for other purposes is relatively increased.
In addition, the main factors that affect the performance of external sorting are
It is the number of data transfers (the number of accesses), and it is also considered that the algorithm that minimizes the number of transfers is a good external sort algorithm.

However, as long as the external sorting method uses the central processing unit and the main storage device, there is a limit in reducing the number of data transfers via the channel. Rather, if a device dedicated to external sort is placed between the channel and the external storage device and external sort can be achieved without using the channel, (1) speedup of sort processing by the dedicated device, (2) external of the channel It seems that the occupancy rate for sorting will be reduced. Moreover, even if such a dedicated device is made, if the device becomes complicated and expensive, efforts for speeding up sort processing and reducing the channel occupancy rate will not be realized. become. Therefore, if such a device could be made, it should be simple and inexpensive. Even more, it requires a huge amount of software development for the new device,
Don't expect a lot of software changes.

The current general-purpose external sort program opens three files, an input file, a work file, and an output file, and then performs read / write processing from the input file to the work file and from the work file to the output file. The sort process is realized while doing it. It is conceivable that the contents of this processing will be converted to hardware as it is, but the execution environment in which one device controls multiple files at the same time makes the logic complicated and lacks space when converted to hardware. In the case of, you have to take extra care of what to do, and you are out of the policy of realizing it as a simple device. Further, the design concept of the existing general-purpose sort program has an emphasis on responding to the request of the minimum number of data access times, and is somewhat different from the concept of simplification. In particular, file space management, which requires information exchange with a system originally supported by software, is likely to be a bottleneck when implementing hardware. Therefore, if the physical space of the data to be sorted is defined from the beginning to the end, it will be easier to implement the hardware on the assumption of a sorting method.

Now, suppose between the external storage device and the channel,
Suppose you have installed an external sorter. The operational situation in that case is that the data set group connected to the external sort device is a master file group that is always used as a sorted file, It may be another subfile group. That is, most of the files connected to such an external sort device will always be in the sorted state. If this is the case, the characteristic of the external sorter is that it is necessary to be able to easily add data to a file in a sorted state, or sort a partially unsorted state that is almost sorted. Ah

[Object of the Invention]

The present invention is based on the above consideration, that sort processing can be performed in a data set to be sorted, that the effect is great in the case of sorting a data set that has been substantially sorted, and the memory area used It is an object of the present invention to provide a sort processing method and device having features such as a small number of products and easy hardware implementation.

[Structure of Invention]

Therefore, the sorting method according to the first aspect of the present invention is such that N sorted blocks each consisting of a first block, a second block, ... Sorted n + 1 with records
A sort processing method of adding blocks and sorting the first block to the n + 1th block, wherein the nth block is stored in a first buffer, the n + 1th block is stored in a second buffer, and the contents of the first buffer are stored. And the contents of the second buffer are merged, the merged result is stored in the first buffer and the second buffer according to the sorting rule, and in the record string stored in the first buffer after the merge, the first buffer before the merge Check whether or not there is a record string located before the first record of the record string stored in. If it does not exist, the sorting is terminated, and if it exists, the contents of the first buffer are It is characterized by moving to two buffers, storing the previous block in the sorted block sequence in the first buffer, and performing the processing again. Than it is.

The sorting apparatus of the second invention of the present invention comprises an external storage unit for storing a data set having M blocks each consisting of N records, and the external storage unit when a buffer read command is received. A block I / O controller that reads the specified block from the unit, stores it in the specified buffer, and writes the data in the specified buffer to the specified external storage unit block when a write command is received, and an internal sort An internal sorter that internally sorts the contents of the specified buffer when a command is received, and a span slider that supports merging and determining the span length. To the p and q buffers specified when a command is received. Process to merge stored sorted data based on span length, before merging And the length of the record string positioned before the first record of the record string stored in the p-th buffer before merging in the record string obtained as a result of the process of outputting the presence or absence of data movement after merging Is provided with a two-block span slider for outputting a span length and a control means, and the control means performs internal sorting processing for all the blocks of the data set, and then the block number. Set i to the initial value, set i + 1 to i, check whether block number i is greater than M, and
In the case of, the process proceeds to the next process, and in the case of Yes, the process is terminated, and the span length is set to N, the block number i is set to k, and the data of the kth block should be stored in the qth buffer. Instruct the above block input / output controller to set k-1 to k, check whether k is less than 1, return to the process in the case of Yes, and execute the next process in the case of No. , Instructing the block input / output controller to read the data of the k-th block into the q-th buffer, merging the contents of the p-th buffer and the q-th buffer, and in the record string obtained as a result of the merging process. The length of the span should be set in the span length storage area,
In addition, the 2 block span slider is instructed to output the movement presence / absence information indicating whether or not the data has been moved before and after the merge, and it is checked whether or not the movement presence / absence information indicates no movement. , If there is no movement, the process shifts to, if there is a shift, shifts to the process, the data in the pth buffer should be written in the kth block, and the data in the qth buffer should be written in the k + 1th block. Instruct the above block I / O controller to write, check whether the span length is zero, move to the process of if it is Yes, move to the process of if it is No, p-th After the data in the buffer and the q-th buffer are exchanged, the above process is performed.

Example of Invention

Hereinafter, the present invention will be described with reference to the drawings. File (synonymous with data set) F has record r as shown below.
It can be considered as a sequence of [1], r [2], ... R [P], ... R [K].

Further, the file F can be considered as an array of blocks as described below.

Each block has a fixed length, and the block Bi is composed of N records as described below.

Here, ▲ R ⁱ _j ▼ means the j-th record of the i-th block. Further, ▲ R ⁱ _j ▼ is ▲ R ⁱ _j ▼ = r [N · (i-1) + j]. A block is an input / output unit.

Next, the definition of symbols used in this specification will be described.
Let r'and r be records, r '← r means that record r'should be located before record r. Further, r [p] r [q] means that records r [p] to r [q] are sorted. However, p <q.

Similarly, ▲ R ⁱ _j ▼ ▲ R ^{i ′} _{j ′} ▼, where N · (i−1) + j <N · (i′−1) + j ′
It is, which means that the record ▲ R ⁱ _j from ▼ to record ▲ R ^{i _'j'} ▼ has been sorted.

Next, the algorithm of the sorting method of the present invention will be described. The sort process can be written as given: file F target: ▲ R ¹ ₁ ▼ ▲ R ^M _N ▼. This is called problem 0. The solution level 1 in the present invention is to perform for i = 1, M ▲ R ¹¹ ₁ ▼ R ⁱ _N ▼. The above formula means that while executing ⁱ from 1 to M in increments of 1 while performing ▲ R ¹ ₁ ▼ ▲ R ⁱ _N ▼. FIG. 1 illustrates the processing of the solution level 1 described above. That is, first, the block B ₁ is brought into a sorted state. Next, blocks B ₁ to B ₂ are sorted. The same processing is repeated to bring blocks B ₁ to Bi _-1 into a sorted state. The blocks B ₁ to Bi _{−1 in} the sorted state and the block Bi are added, and then the blocks B ₁ to Bi are brought into the sorted state. The same process is repeated to bring blocks B ₁ to B _M into a sorted state.

In order to perform the above processing, the blocks B ₁ to Bi ₋₁ and the block Bi in the sorted state must be added, and then the blocks B ₁ to Bi must be put into the sorted state. If this is problem 1, problem 1 can be written as given: ▲ R ¹ ₁ ▼ ▲ R ^i-1 _N ▼, Bi target: ▲ R ¹ ₁ ▼ ▲ R ⁱ _N ▼. Solution level 2 of this problem 1
Then, the solution level 2 can be written as Internal Sort Bi Merge Bi and <B ₁ , B ₂ , ....., Bi _-1 >. The problem here is how to perform the above merge. If this is problem 2, problem 2 is written as given: ▲ R ¹ ₁ ▼ ▲ R ^i-1 _N ▼, ▲ R ⁱ ₁ ▼ ▲ R ⁱ _N ▼ target: ▲ R ¹ ₁ ▼ ▲ R ⁱ _N ▼ You can Assuming that the solution method for this problem 2 is the solution level 3, the solution level 3 is given by the following equation.

fot u = i, 2, -1 ▲ R ^u-1 ₁ ▼ ▲ R ⁱ _N ▼ The above formula realizes ▲ R ¹ ₁ ▼ ▲ R ⁱ _N ▼ by changing u from i to 2 in steps of -1. It means that. In order to carry out the above solution level 3, it is necessary to solve the following problems.

given: ▲ R ^u-1 ₁ ▼ ▲ R ^u-1 _N ▼, ▲ R ^u ₁ ▼ ▲ R ⁱ _N ▼ target: ▲ R ^u-1 ₁ ▼ ▲ R ⁱ _N ▼ If this is problem 3, problem 3 Means that the block Bu _-1 indicated by u-1 is sorted, and under the condition that the blocks Bu to Bi are sorted, the blocks Bu _-1 to Bi are sorted.

Since u = i in the initial state, the problem 3 is given: ▲ R ^i-1 ₁ ▼ ▲ R ^i-1 _N ▼, ▲ R ⁱ ₁ ▼ ▲ R ⁱ _N ▼ target: ▲ R ^i-1 ₁ ▼ ▲ R ⁱ _N ▼. FIG. 2 illustrates the solution of Problem 3 in the first step. It is assumed that the block Bi _-1 has already been sorted, and the content of the ^first record ▲ R ^i-1 ₁ ▼ is a and the content of the last record ▲ R ^i-1 _N ▼ is b. Similarly, it is assumed that the block Bi is already sorted, the content of the _first record ▲ R ⁱ ₁ ▼ is c, and the content of the last record ▲ R ⁱ _N ▼ is. Block Bi _-1
When Bi and Bi are merged, Cases 1 to 4 occur as shown in the figure. Case 1 shows a case where the merge result and the state before the merge have not changed. Case 2 shows a case where the record before merge ▲ R ⁱ ₁ ▼ has been moved forward in the merge result. Case 3 shows a case where the state of the merged result is a state in which the blocks Bi _-1 and Bi before the merge are exchanged. Case 4 shows a case where the record before merge ▲ R ⁱ ₁ ▼ is located at the beginning and the record before merge ▲ R ^i-1 ₁ ▼ is located behind it in the merged result. Case3 can be thought of as a special case of Case4. If the merge result is expected to be Case1, do nothing. That is, the blocks B ₁ to Bi have already been sorted. When the merged result is expected to be Case 2, the two blocks are merged with the blocks Bi ₋₁ and Bi, and the subsequent processing is not performed. That is, the above 2
When block merging is performed, blocks B ₁ to Bi are sorted. If the merged result is expected to be Case 3, it is assumed that the merged Bi has been completed, and then given: ▲ R ¹ ₁ ▼ ▲ R ^i-2 _N ▼, ▲ R ^i-1 ₁ ▼ Solve the problem ▲ R ^i-1 _N ▼ target: ▲ R ¹ ₁ ▼ ▲ R ^i-1 _N ▼. If the merge result is expected to be Case 4, perform the merge, and then use Bi and ▲ R after the merge.
It is assumed that ^i-1 _{k + 1} ▼ ▲ R ⁱ _N ▼ has been completed, and then given: ▲ R ¹ ₁ ▼ ▲ R ^i-2 _N ▼, ▲ R ^i-1 ₁ ▼ ▲ R ^i-1 _k ▼ target: ▲ R ¹ ₁ ▼ ▲ R ^i-1 _k ▼ solve the problem. Note that k + 1 is a record having the content a after merging (record before merge ▲ R ^i-1 ₁ ▼)
The position on the block Bi _-1 of is shown.

Next, the concept of span slide will be described. In FIG. 3, the records ▲ R ^u ₁ ▼ to ▲ R ^u _k ▼ are called spans. That is, the span is a group of records that may include records that should be located ahead of the record sequence (▲ R ¹ ₁ ▼, ..., ▲ R ^u-1 _N ▼) located in front (▲ R ^u ₁ ▼, ..., ▲ R ^u _k ▼). This is from C to a in Case 4 in Figure 2 (not including the a)
Up to. In Fig. 3, ▲ R ¹ ₁ ▼ ▲ R ^u-2 _N
▼, ▲ R ^u-1 ₁ ▼ ▲ R ^u-1 _N ▼, ▲ R ^u ₁ ▼ ▲ R ^u _k ▼, ▲
R ^u _{k + 1} ▼ ▲ R ⁱ _N ▼, and ▲ R ^u-1 _N ▼ ← ▲ R ^u _{k + 1}
▼ When block Bu _-1 and block Bu are merged, cases 1 to 4 in FIG. 2 occur. Case1, 2
In the case of, the span disappears, and in Cases 3 and 4, a new span is generated. Merging two blocks and finding the span is called span slide. Case
In the case of 3 and 4, after -1 the value of u, block Bu _-1
And slide span with Block Bu. Block Bu
_{When -1} and the block Bu are merged, the block Bu _-1 and the span portion (Ru or Ru portion) of the block Bu are merged.

FIG. 4 is a block diagram of one embodiment of the present invention. In FIG. 4, 1 is a sort processor, 2 is an external storage unit, 3 is a span sort supervisor, 4 is a 2 block span slider, 5 is a block internal sorter, 6 is a block input / output controller, and 7 is a record. Comparator, 8
-1 to 8-3 are block buffers, and BP is a buffer pool. The white thick line indicates block access, the black thick line indicates record access, and the solid line indicates the flow of control signals. The block input / output controller 6 is a span sort supervisor 3
Is instructed which block should be read into which block buffer, or the content of which block buffer should be written to which block, and input / output in block units is performed according to the instruction. Block internal sorter 5
Is instructed by the span sort supervisor 3 which block buffer contents should be internally sorted, and sorts records in the block buffer according to the instruction. The record comparator 7 is used to compare the records necessary for sorting. The two-block span slider 4 is instructed by the span sort supervisor 3 as to which of the two block buffer contents should be span-slipped (merged), and the span-slide is performed as instructed. When the two records are input, the record comparator 7 determines their order, and outputs a true value if the target order and a false value otherwise. Span Sort Supervisor 3
Gives a command to the 2-block span slider 4, the block internal sorter 5, and the block input / output controller 6. For this reason, it is necessary to preset the start and end of the block address of the data set to be sorted on the external storage unit 2 in the span sort supervisor 3 and to give the initial value of the span slide control. There is. When the sort processing device 1 is implemented by a processor, the span sort supervisor 3, the two block span slider 4, the internal block sorter 5, the block input / output controller 6, and the record comparator 7 are implemented by software. It Of course, all of these may be pure hardware configurations.

The sorting process of the present invention can be divided into a first phase and a second phase. The first phase is a phase for bringing all the blocks of the data set into a sorted state within the blocks. FIG. 5 illustrates the outline of the first phase. The block input / output controller 6 receives a read command from the span sort supervisor 3 to read a block in the external storage unit 2 and write it in the block buffer 8-1. After the writing of the block in the block buffer 8-1 is completed, the span sort supervisor 3 sends the block buffer 8-1 to the block internal sorter 5.
Specifies that the contents of should be sorted internally. Upon receiving this internal sort command, the block internal sorter 5 uses the record comparator 7 to perform an internal sort on the block data in the block buffer 8-1 and sends the sorted result to the span sort supervisor 3. Notice. When the internal sorting of the block data by the block internal sorter 5 is completed, the span sort supervisor 3 writes the block data of the block buffer 8-1 to the original block position of the external storage unit 6 as necessary. The block input / output controller 6 is instructed what to do. When receiving this write command, the input / output controller 6 writes the block data in the block buffer 8-1 to the original block position in the external storage unit 2. Hereinafter, the same process is repeated.

FIG. 6 explains the outline of the second phase of the sorting process. The span sort supervisor 3 instructs the block input / output controller 6 to first read two consecutive blocks in the external storage unit 2 into the block buffers 8-1 and 8-2. When the block input / output controller 6 receives this read command, it reads two consecutive blocks in the external storage unit 2 into the block buffers 8-1 and 8-2. After the execution of this read command is completed, the span sort supervisor 3
Instructs the two-block span slider 4 to span-slide the block data in the block buffers 8-1, 8-2. 2 blocks
When the span slider 4 receives this span slide command, it span-slides the two block data in the block buffers 8-1 and 8-2, and the result of the span slide is the span sort supervisor. Notify 3 After completing the execution of the span slide command, the span sort supervisor 3
Write the block data of the buffers 8-1 and 8-2 to the original block position in the external storage unit 2 and at the same time write these block data to the working storage area in the external storage unit 2. The block input / output controller 6 is instructed. Block input / output controller 6
When this write command is received, the block buffer 8
-1 and 8-2 block data to external storage unit 2
Write to. Hereinafter, the same process is repeated.

The commands from the span sort supervisor 3 to the block input / output controller 6 are as follows.

BIOC (Read, i, p) BIOC (Write, i, p) When the block input / output controller 6 receives the former command, it reads the data of the i-th block into the p-th buffer, and when it receives the latter command, it Write the data in the p buffer to the i-th block. The buffer is a block buffer. The command from the span sort supervisor 3 to the block internal sorter 5 has the following format.

When the BIS (p, c) block internal sorter 5 receives this command, it internally sorts the data in the p-th buffer, and if the data arrangement before and after sorting is unchanged, c = 0, change If so, c = 1 is output. The command from the span sort supervisor 3 to the 2-block span slider 4 has the following configuration.

2BSS (p, q, sp, c) When the 2-block span slider 3 receives this command, it merges the p-th buffer and the q-th buffer and outputs the span length after completion to the specified area sp for processing. If there is no movement of data in the buffer between before and after processing, c = 0 is output, and if there is movement, c = 1 is output. It is assumed that the q-th buffer is a span block with a span length sp immediately before the start and the p-th buffer is a span block after the end.

FIG. 7 shows the operation logic of the span sort supervisor 3. Span Sort Supervisor 3
Has operation logic for the first phase and operation logic for the second phase. First, the operation logic for the first phase will be described.

 The block number i is set to 0.

 Let i + 1 be i.

Check if i is greater than M. If Yes, start the second phase, and if No, perform.

 BIOC (Read i, 1) is issued.

 Issue BIC (1, c).

Check whether c is 0 or not. If yes, perform, and if no, perform.

Perform BIOC (Write, i, 1) and return to. The operation logic for the second phase operates as follows.

The block number i is set to 1, the buffer number p is set to 1, and the buffer number q is set to 2.

 Set i + 1 to i.

Check if i is greater than M. If Yes, the second phase ends, and if No, go.

Let N be the span length, i be the block number k, and
Issue BIOC (Read, k, q).

 Let k-1 be k.

Check if k is less than 1. If yes, perform, and if no, perform.

BIOC (Read, k, p) and 2BSS (p, q, sp, c) are issued.

Check whether c is 0 or not. If yes, perform, and if no, perform.

BIOC (Write M + 1, p), BIOC (Write, M + 2, q), BIO
Issues C (Write, k, p) and BIOC (Write, k + 1, q).
The two instructions are data copies for ensuring data integrity.

Check whether the span length is 0 or not. If Yes, perform. If No, perform.

The contents of the p-th buffer are transferred to the x-th buffer, the contents of the q-th buffer are transferred to the q-th buffer, and the contents of the x-th buffer are transferred to the q-th buffer. Then do the following:

FIG. 8 shows an example of transition of span sort. The sort according to the present invention is called span sort. In this example, the number of blocks is 6. The steps or steps are the first phase. The first block to the sixth block are internally sorted by the steps to. In the step, the first and second blocks are span-slide and the first block to the second block are sorted.
In the step, the second and third blocks are span slid. If the span continues to exist in the processing of the step, the span sliding of the first and second blocks is performed in the step. When the steps are completed, the first block to the third block are sorted. Similar processing is repeated after this.

〔The invention's effect〕

As is apparent from the above description, according to the present invention, (a) the processing can be performed in the same data set, so that another file is not required, and (b) the required memory area is small. , (C) Since it is a linear sort, it is easy to manage files, (D) It is very effective in sorting a roughly sorted data set, and (E) Because the execution environment can be limited to the target data set, In addition, it is possible to achieve remarkable effects such as easy hard fairing.

[Brief description of drawings]

FIG. 1 is a diagram for explaining the basic procedure for sorting blocks B ₁ to B _M , and FIG. 2 is a block Bi in a sorted state _{-1 A} block Bi in a sorted state is merged. FIG. 3 is a diagram for explaining the state after the step, FIG. 3 is a diagram for explaining the span slide, and FIG.
FIG. 5 is a block diagram of an embodiment, FIG. 5 is a diagram showing an outline of a first phase of sort processing in the embodiment, FIG. 6 is a diagram showing an outline of a second phase of sort processing in the embodiment, and FIG. FIG. 8 is a diagram showing the operation logic of the sort supervisor, and FIG. 8 is a state transition example of span sort. 1 ... Sort processing device, 2 ... External storage unit, 3 ...
… Span sort supervisor, 4 …… 2 blocks ・
Span slider, 5 ... Block internal sorter, 6 ...
Block input / output controller, 7 ... Record comparator,
8-1 to 8-3 ... Block buffer, BP ... Buffer pool.

Claims (2)

[Claims] 1. A sorted (n + 1) th block having N records is added to a sorted block sequence composed of a first block, a second block, ..., An nth block each having N records. From the 1st block to the nth
A sorting method for sorting up to +1 blocks, wherein the nth block is stored in a first buffer, the n + 1th block is stored in a second buffer, and the contents of the first buffer and the contents of the second buffer are merged. The merged result is stored in the first buffer and the second buffer according to the sort rule, and the first record of the record string stored in the first buffer before the merge in the record string stored in the first buffer after the merge It checks whether or not there is a record string located further ahead, and if it does not exist, sort is terminated, and if it exists, the contents of the first buffer are moved to the second buffer, and the sorted block string The method of sorting is characterized in that the block immediately before in 1 is stored in the first buffer and the processing is performed again. 2. An external storage unit for storing a data set having M blocks each consisting of N records, and reading a specified block from the external storage unit when a buffer read command is received. A block I / O controller that stores data in the specified buffer and writes the data in the specified buffer to the specified block in the external storage unit when a write command is received, and the buffer specified when the internal sort command is received Of the sorted data stored in the p-th buffer and the q-th buffer specified when the span sorter command is received, which supports merging and determining the span length. Merge based on length, move data before and after merge Processing for outputting the presence / absence, processing for outputting the span length indicating the length of the record string located ahead of the first record of the record string stored in the p-th buffer before the merge in the record string obtained as a result of the merge process And a control means, which performs internal sorting of all blocks of the data set, and then sets a block number i to an initial value, Set i + 1 to i, check whether block number i is greater than M, and
In the case of, the process proceeds to the next process, and in the case of Yes, the process is terminated, and the span length is set to N, the block number i is set to k, and the data of the kth block should be stored in the qth buffer. Instruct the above block input / output controller to set k-1 to k, check whether k is less than 1, return to the process in the case of Yes, and execute the next process in the case of No. , Instructing the block input / output controller to read the data of the k-th block into the p-th buffer, and merging the contents of the p-th buffer and the q-th buffer, in the record string obtained as a result of the merging process. The length of the span should be set in the span length storage area,
In addition, the 2 block span slider is instructed to output the movement presence / absence information indicating whether or not the data has been moved before and after the merge, and it is checked whether or not the movement presence / absence information indicates no movement. , If there is no movement, the process shifts to, if there is a shift, shifts to the process, the data in the pth buffer should be written in the kth block, and the data in the qth buffer should be written in the k + 1th block. Instruct the above block I / O controller to write, check whether the span length is zero, move to the process of if it is Yes, move to the process of if it is No, p-th A sort processing apparatus configured to perform the above processing after exchanging data in a buffer and a qth buffer.