JPH08286965A - Division parallel processing system of batch processing data - Google Patents

Abstract

PURPOSE: To improve the through-put as a whole of batch processing and to shorten a processing time by dividing batch processing data so as to execute a parallel processing. CONSTITUTION: When an operator command for dividing a data file 5 is supplied, a file division managing record generating means 7 divides data records 51-5m in the data file 5 into blocks at every data record with continuous record addresses and records a head record address and the last record address in the respective divided blocks in file division managing records 41-4n. When the operator command for starting the parallel processing is supplied, plural parallel processing means 31-3n read corresponding file division managing records 41-4n in the managing file 4 and successively read data records from the head record address to the last code address from the data file 5 so as to execute the processing.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a division parallel processing method for batch processing data, and in particular, data to be batch processed without affecting the processing result of the data records (hereinafter,
(Batch processing data)) and a parallel processing method of dividing batch processing data for parallel processing.

[0002]

2. Description of the Related Art Conventionally, when batch processing data is batch processed, batch processing data is sequentially processed by a single task. However, this method has a problem that if the data amount of the batch processing data is large, it takes a long time to complete the processing.

As an example of the collective processing, there is an automatic transfer processing of utility charges in banking business. This is a transfer of a predetermined amount of money from a contractor's account to a public institution account or the like on a predetermined date such as the end of the month, and is processed collectively for a large number of contractors. This automatic transfer process is generally performed on the transfer date, at night after the daytime work ends. This is because it is unknown whether the settlement can be made after the balance of the contractor's account on the day is fixed, and also to prevent other business from being hindered. For this reason, the amount of batch processing data becomes extremely large at the end of the month or at the beginning of the month when various batch processing is concentrated, and processing time may be required until the business start time of the next day. It is considered that this tendency will be further promoted as the extension of night service hours will be further promoted in the future, and that the working environment of staff involved in batch processing will deteriorate and the effect on the next day's work will be affected. It becomes a problem.

Therefore, the processing of individual cases included in the batch processing is performed in a tentative state by being distributed before the deadline, and the contents of the processing are fixed as they are if there is no change by the deadline. A processing date distribution method for batch processing has already been proposed (see Japanese Patent Laid-Open No. 5-143542).

[0005]

In the above-mentioned conventional method of distributing processing days of online batch processing, the processing of individual cases included in the batch processing is distributed before the deadline and performed in a provisional state. If there is no change until now, the contents of the process were fixed as it is, so when executing the normal business process, the process must be sorted according to whether it is before the deadline or after the deadline. There is a problem in that the application program that executes the process needs to be modified.

In view of the above points, an object of the present invention is to divide the batch processing data into parallel processing without modifying the application program for executing the normal business processing, and thereby to improve the overall throughput of the batch processing. It is an object of the present invention to provide a division parallel processing method for batch processing data that improves the processing time and shortens the processing time.

[0007]

The division parallel processing method of batch processing data of the present invention stores a plurality of data records to be batch processed, and a data file which can be simultaneously accessed by a plurality of tasks, and the data. Stores multiple file division management records that manage the first record address and last record address in each block obtained by dividing the data record in the file into multiple data records with consecutive record addresses, enabling simultaneous access from multiple tasks. Management file and a data record in the data file are divided into blocks for each of a plurality of data records having continuous record addresses, and the first record address and the last record address in each divided block are stored in the management file. File to be recorded in the split management record And a plurality of parallel processing means for reading the corresponding file division management record in the management file and sequentially reading the data records from the first record address to the last record address from the data file for processing. .

[0008]

According to the division parallel processing method of batch processing data of the present invention, the file division management record creating means divides the data record in the data file into blocks for each of a plurality of data records having continuous record addresses, and divides each of the divided data records. The first record address and the last record address in the block are recorded in the file division management record in the management file, and multiple parallel processing means read the corresponding file division management records in the management file, and read from the first record address to the last record address. Data records are sequentially read from the data file and processed.

[0009]

Next, the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of an online real-time processing system 1 to which a batch parallel processing system for batch processing data according to an embodiment of the present invention is applied. The online real-time processing system 1 includes a program control unit 2, a plurality of parallel processing units 31 to 3n (n is a positive integer) that operate as tasks, a management file 4, a data file 5, and a file division management record creating unit. And 7 are included. In addition, in FIG.
Reference numeral 6 indicates a command input device connected to the online real-time processing system 1.

The management file 4 is adapted to store file division management records 41 to 4n, and the file division management records 41 to 4n are each of the parallel processing means 31 to.
Start record addresses 411 to 4n1 and processed record address 412 of the data record processed in 3n
4n2 and final record addresses 413 to 4n3 are stored. Access to the management file 4 is exclusively controlled by an operating system (not shown) so that a plurality of tasks can access it substantially simultaneously.

The data file 5 has a data record 51.
Stores 5 m (m is the total number of data records). Access to the data file 5 is exclusively controlled by the operating system so that it can be accessed substantially simultaneously by a plurality of tasks.

Referring to FIG. 2, the process of the file division record creating means 7 is performed by a counter initialization step 201.
And data record counter end determination step 202
A data file reading step 203, a block top record determination step 204, a top record address and processing record address setting step 205
A block final record determination step 206, a final record address setting step 207, a file division management record counter adding step 208, a data record counter adding step 209, and a final record address setting step 210. Where a
Is the counter of the file division management records 41 to 4n, b
Is a counter for the data records 51-5m (same below). In addition, [] indicates that it is a Gaussian number.

Referring to FIG. 3, the parallel processing means 31 to 31.
The process of n is the data record reading step 301,
A data record processing step 302, a processing record address / final record address comparison step 303,
It comprises a final record address determination step 304, a next data record reading step 305, and a processing record address setting step 306.

Next, the operation of the divided parallel processing method for batch processing data of this embodiment having the above-mentioned configuration will be described.

From the command input device 6 to the data file 5
When an operator command for dividing the file is input, the program control means 2 activates the file division management record creating means 7. At this time, the division number n is specified in the operator command.

The file division management record creating means 7 is
The data records 51 to 5m of the data file 5 are read, the data records 51 to 5m are divided into n blocks having continuous record addresses, the head record addresses 411 to 4n1 of each block, and the processing record address 4
12-4n2 and final record address 413-4n
3 is a file division management record 41 to management file 4
4n respectively.

More specifically, the file division management record creating means 7 firstly, the file division management record counter a.
And the data record counter b are initialized to 1 (step 201).

Next, the file division management record creating means 7 compares the data record counter b with the total number m of data records (step 202). If the data record counter b is less than the total number m of data records, the data file 5 Read one data record (step 20)
3). The total number m of data records is registered in the management file 4 when the data file 5 is created.

Subsequently, in the file division management record creating means 7, the head record address [m / (n-1)] * (a-1) +1 of the block whose data record counter b is indicated by the file division management record counter a. Is determined (step 204), and if they are equal, the record address of the read data record 5b is set to the first record address 4a1 of the file division management record 4a.
And processing record address 4a2 (step 205). In step 204, the data record counter b sets the block start record address [m / (n-
1)] * (a-1) +1, the file division management record creating means 7 skips step 205.

Next, the file division management record creating means 7 determines whether the data record counter b is equal to the final record address [m / (n-1)] * a of the block indicated by the file division management record counter a. The determination is made (step 206), and if they are equal, the record address of the read data record 5b is set to the final record address 4a3 of the file division management record 4a (step 207), and the file division management record counter a
Is incremented by one (step 208). When the data record counter b becomes equal to the final record address [m / (n-1)] * a of the block in step 206, the file division management record creating means 7 skips steps 207 and 208.

Subsequently, the file division management record creating means 7 increments the data record counter b by 1 (step 209) and returns the control to step 202.

Steps 202 to 209 are repeated, and when the data record counter b becomes equal to the total number m of data records in step 202, the file division management record creating means 7 sets the record address of the last read data record 5b to file division management. Set to the final record address 4a3 of the record 4a (step 21
0), the process ends.

In this way, the file division management record creating means 7 causes the file division management records 41 to 4n of the management file 4 to start record addresses 411 to 4n1, process record addresses 412 to 4n2, and final record addresses of the blocks of the data file 5. Record address 41
When an operator command for starting parallel processing is input from the command input device 6 after 3 to 4n3 are respectively set, the program control means 2 simultaneously activates a plurality of parallel processing means 31 to 3n.

A plurality of parallel processing means 31 started at the same time
.About.3n operate in parallel and refer to the file division management records 41 to 4n of the management file 4 to refer to the data file 5
The data records 51 to 5m therein are processed in parallel in divided block units.

More specifically, the parallel processing means 3a (a = 1 to 1)
n) first reads the data record 5b in the data file 5 indicated by the processing record address 4a2 in the file division management record 4a (step 30).
1), an application program (not shown) is called for the data record 5b to perform business data processing (step 302).

Next, the parallel processing means 3a compares the processed record address 4a2 with the final record address 4a3 (step 303), and if they are not equal (step 30).
No in step 4), the next data record 5 (b + 1) in the data file 5 is read (step 305), and the current record address is set to the process record address 4a2 of the file division management record 4a (step 30).
6) Return control to step 302.

Steps 302 to 306 are repeated, and if the processed record address 4a2 and the final record address 4a3 are equal in step 304, the parallel processing means 3a.
Ends the process.

As described above, as a result of the data records 51 to 5m in the data file 5 being divided and processed in parallel by the plurality of parallel processing means 31 to 3n, the overall throughput of batch processing is improved and the processing time is significantly increased. Will be shortened to.

In the above embodiment, the data file 5
However, the present invention can be similarly applied to the case where there are a plurality of data files 5. Conventionally, if there is a large difference in the data amount of each of the plurality of data files 5, there is a time lag until the processing of the data records of each data file 5 is completed, which lowers the overall throughput of the computer system, However, according to the present invention, a plurality of data files 5
Even when the number of records in 1 is significantly different, the end time of the processing for each data file 5 can be made uniform, so that labor saving in computer operation can be realized.

[0031]

As described above, according to the present invention, the file division management record creating means divides the data record in the data file into blocks for each of a plurality of data records having continuous record addresses, and in each divided block. Record the first record address and the last record address of the file in the file division management record in the management file,
By the plurality of parallel processing means reading each file division management record in the management file and sequentially reading the data records from the first record address to the last record address from the data file,
The throughput of the batch processing of batch processing data can be improved and the processing time can be shortened.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an online real-time processing system to which a batch parallel processing method for batch processing data according to an embodiment of the present invention is applied.

FIG. 2 is a flow chart showing processing in a file division management record creating means in FIG.

FIG. 3 is a flowchart showing the processing in the parallel processing means in FIG.

[Explanation of symbols]

 1 Online Real Time Processing System 2 Program Control Means 4 Management File 5 Data File 6 Command Input Device 7 File Division Management Record Creating Means 31 to 3n Parallel Processing Means 41 to 4n File Division Management Records 51 to 5m Data Records 411 to 4n1 First Record Address 412 to 4n2 process record address 413 to 4n3 final record address

Claims (2)

[Claims] 1. A data file in which a plurality of data records to be collectively processed are stored and which can be simultaneously accessed from a plurality of tasks, and a data record in the data file is provided for each of a plurality of data records having continuous record addresses. A file that manages the first record address and the last record address in each divided block, and stores multiple management records that can be accessed simultaneously by multiple tasks.The data records in the data file have continuous record addresses. A file division management record creating unit that divides each of a plurality of data records into blocks, and records the first record address and the last record address in each divided block in the file division management record in the management file; Within A parallel processing method for batch processing data, comprising: a plurality of parallel processing means for reading a file division management record for reading data records from a first record address to a last record address and sequentially processing the data records from the data file. 2. The division parallel processing method for batch processing data according to claim 1, wherein the data files are plural.