JP2014089572A - Data processing system, data processing method and data processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent, when the delay of data processing using a part of a plurality of databases unexpectedly occurs in a data processing system in which a plurality of data processing is executed by a computer in a simultaneous parallel manner, and a plurality of databases are updated during the execution of the data processing, an outputtable processing result data output in a thread which has already completed processing from being put in a standby state.SOLUTION: A thread in which the delay of data processing has occurred is detected, and it is considered that the delay also occurs in another thread which performs data processing by the same data processing method to the same database as the database used by the thread in which the delay has occurred, and only the processing result data in the thread which has not received the influence of the delay are written first in a storage device in a batch without waiting for the processing completion of those delay threads.

Description

  The present invention relates to a data processing system, a data processing method, and a data processing program for executing a plurality of data processings simultaneously and in parallel on a computer and updating a plurality of databases during the execution of such data processing. The present invention relates to a data processing system, a data processing method, and a data processing program that collectively output the processing results of the above.

  Generally known is a data processing system in which a plurality of data processes are executed on a computer in parallel with a large amount of data generated in substantially the same time period, and a plurality of databases are updated during the data processing. Yes.

  An example of such a data processing system will be described with reference to FIG.

  Referring to FIG. 8, the data processing system includes a data processing device 500 such as a computer that executes various data processing, a storage device 700 that stores an input file 600, and a storage device 800 that stores an output file 620. And three databases 610-1, 610-2 and 610-3 each storing a plurality of files.

  Next, the operation of the data processing system will be described.

  First, it is assumed that the data processing execution process 510 is activated in advance on the data processing device 500 in order to process certain data on the data processing device 500.

  Next, the running data processing execution process 510 has four business threads 530a to 530d (in this case, “threads” that actually perform the data processing are “programs on an operating system that supports parallel processing”. The minimum execution unit ") and the control thread 520 that controls the business threads 530a to 530d are activated and made resident.

  Next, when an input file 600 including a large amount of input data to be processed from the storage device 700 is given to the data processing device 500, the control thread 520 reads all the data in the file 600 from the input file 600. Then, these data are grouped according to related tasks to create four processing data groups.

  Next, the control thread 520 passes the four processing data groups in a one-to-one correspondence to the four business threads 530a to 530d, and instructs the start of data processing for the processing data groups. Here, the data amount of each processing data group is determined so that the time required for completing the data processing in each of the business threads 530a to 530d is approximately equal.

  The business threads 530a to 530d instructed to start data processing simultaneously perform data processing on the received processing data group in parallel. In this way, performing data processing in parallel with a plurality of threads is generally referred to as multi-thread processing.

  During the execution of data processing in the business threads 530a to 530d, the business threads 530a to 530d may output the processing results to the output destination database at any time and update the contents of the databases 610-1 to 610-3 as needed. However, at that time, the commit of the databases 610-1 to 610-3 (confirmation of update contents) is not performed at the same time as each update, and the process waits for a commit instruction from the control thread 520.

  Here, committing a database is to formally reflect updates made to the database so that anyone can see the updated contents.

  Many databases are controlled so that other users cannot see or update the update location until it is committed, so that if there is a database update during a series of data processing, the update is abnormal. At that time, if the update location is not committed, the update can be canceled, and as a result, an erroneous reference to the update result by other users can be avoided. Therefore, a commit is performed for all update locations after it has been determined that all update processing of the database performed during a series of data processing in a certain business thread has been completed normally and cancellation of the update is not necessary. Implemented collectively.

  When the data processing for the given processing data group is completed, the business threads 530a to 530d send a processing completion notification to the control thread 520, and further pass the processing result data to be output to the file 620 to the control thread 520.

  When the control thread 520 receives processing completion notifications from all the business threads 530a to 530d, the control thread 520 collects the processing result data received from all the business threads 530a to 530d and outputs them to the output file 620.

  Here, after receiving processing completion notifications from all the business threads 530a to 530d, the processing result data received from these business threads 530a to 530c are output to the output file 620 in a batch, that is, the reason for writing is processing Rather than writing every time result data is generated, waiting until all the processing result data is collected and writing them in a batch reduces the number of times of writing processing to the file 620, and the total throughput of writing data to the output file 620 is comprehensive. This is because it becomes expensive.

  When the output of the processing result data to the output file 620 is completed, the control thread 520 sends a commit instruction to each business thread 530.

  In response to the commit instruction, each business thread 530a to 530d performs a commit process for confirming the update to the database that has been updated by itself.

  Thus, all data processing relating to the input file 600 is completed.

  FIG. 9a) shows the flow from the start of data processing in each business thread 530a-d to the commit of the databases 610-1 to 610-3.

  However, in data processing including database update in the data processing system shown in FIG. 8, it takes time to process data unexpectedly in a specific business thread due to concentration of update processing on a specific database. In some cases, there is a delay in the completion of data processing in the thread.

  FIG. 9b) shows an example when the completion of data processing in the business threads 530b and 530d is delayed. In this case, the data processing results of the other business threads 530a and c are finished, and the processing result data in the business threads 530b and d are stored even though the processing result data in the business threads 530a and c are available. Since they are not prepared, the output of the processing result data to the output file 620 is waited until the data processing in the most delayed business thread 530d is completed and all the processing result data is prepared. The data processing system shown in FIG. 8 has a problem that the processing time increases.

  An object of the present invention is to provide a data processing system, a data processing method, and a data processing program that can solve the above-described problems.

The data processing device of the present invention is a device that executes a plurality of data processings simultaneously and in parallel with a plurality of business threads generated on the device,
A thread status list, a delay determination list, a control unit, a delay determination unit, and an output control unit;
The status list has a plurality of thread status data corresponding one-to-one to the plurality of business threads,
Each of the thread status data is updated by ID information for identifying the corresponding business thread, status information indicating a current processing status of data processing performed by the corresponding business thread, and data processing performed by the corresponding business thread. Database identification information for identifying a database to be processed, data processing method information for identifying a data processing method in the corresponding business thread, and data of a processing target data group that is a target of data processing performed by the corresponding business thread Size information and start time information indicating the start time of data processing performed by the corresponding business thread,
The delay determination list includes each of the data processing method information, the database identification information for identifying the database updated by the data processing method identified by each of the data processing method information, the data processing method information, and the Delay reference value information obtained from the processing time per predetermined unit size of the processing target data group in the previous processing of each data processing corresponding to each combination with the database identification information,
The control unit reads a plurality of processing target data groups given from the outside and distributes them to the plurality of business threads, instructs each of the business threads to start data processing, and each of the business threads Before the start of the data processing, the status information, the database identification information, the data processing method information, the data size information, and the start time corresponding to the data processing to be performed are added to the thread status data corresponding to the own thread. Information, and when the data processing of the own thread is completed, the status information of the thread status data corresponding to the own thread is updated to information indicating that the data processing is completed,
The delay determination unit refers to each thread status data registered in the status list at a predetermined period, and for the business thread that has not yet been processed, the thread status data corresponding to the business thread From the delay reference value information registered in the delay determination list corresponding to the database identification information and the data processing method information, the data size information and the time information of the thread status data corresponding to the business thread, The presence / absence of a delay in data processing of the business thread is determined, and when the delay is detected, the status information of the thread status data corresponding to the business thread is updated to information indicating that the process is being delayed,
The output control unit refers to the status information, the database identification information, and the data processing method information registered in each thread status data after any of the plurality of business threads completes data processing. If no delay has occurred and there is no business thread that is still being processed, the processing result data of all the business threads that have been processed at that time are output to the outside in a batch.

Further, the data processing method of the present invention includes:
While executing multiple data processing concurrently in multiple business threads generated on its own device,
ID information for identifying the corresponding business thread, status information indicating the current processing status of data processing performed by the corresponding business thread, and a database to be updated by data processing performed by the corresponding business thread Database identification information, data processing method information for identifying a data processing method in the corresponding business thread, data size information of a processing target data group to be subjected to data processing performed by the corresponding business thread, and the corresponding A thread status list including a plurality of thread status data each corresponding to the plurality of business threads, each having start time information indicating a start time of data processing performed by the business threads;
Each of the data processing method information, the database identification information for identifying the database updated by the data processing method identified by each of the data processing method information, each of the data processing method information and the database identification information This is a method used in a data processing apparatus having a delay determination list having delay reference value information obtained from a processing time per predetermined unit size of a processing target data group in the previous processing of each data processing corresponding to a combination. And
A data processing start step for reading a plurality of processing target data groups given from the outside and distributing them to the plurality of business threads, and instructing the business threads to start data processing;
In each business thread instructed to start data processing, the status information and the database identification corresponding to the data processing to be performed are added to the thread status data corresponding to the own thread before starting the data processing of the own thread. Information, the data processing method information, the data size information, and the start time information are registered, and when the data processing of the own thread is completed, the status information of the thread status data corresponding to the own thread is completed. A data processing step for updating to information indicating that,
The database identification information and the data of the thread status data corresponding to the business thread for a business thread that has not yet been processed with reference to each thread status data registered in the status list at a predetermined cycle From the delay reference value information registered in the delay determination list corresponding to the processing method information, the data size information and the time information of the thread status data corresponding to the business thread, data processing of the business thread A delay determination step of determining whether there is a delay and detecting the delay, updating the status information of the thread status data corresponding to the business thread to information indicating that the delay is in progress;
After one of the plurality of business threads completes data processing, a delay occurs with reference to the status information, the database identification information, and the data processing method information registered in each thread status data. And an output control step for collectively outputting processing result data of all the business threads that have completed processing at that time when there is no business thread that is still being processed.

Furthermore, the data processing program of the present invention provides:
While executing multiple data processing concurrently in multiple business threads generated on the own computer,
ID information for identifying the corresponding business thread, status information indicating the current processing status of data processing performed by the corresponding business thread, and a database to be updated by data processing performed by the corresponding business thread Database identification information, data processing method information for identifying a data processing method in the corresponding business thread, data size information of a processing target data group to be subjected to data processing performed by the corresponding business thread, and the corresponding A thread status list including a plurality of thread status data each corresponding to the plurality of business threads, each having start time information indicating a start time of data processing performed by the business threads;
Each of the data processing method information, the database identification information for identifying the database updated by the data processing method identified by each of the data processing method information, each of the data processing method information and the database identification information A program executed on a computer having a delay determination list having delay reference value information obtained from a processing time per predetermined unit size of a processing target data group in a previous process of each data process corresponding to a combination, ,
A data processing start process that reads a plurality of processing target data groups given from the outside and distributes them to the plurality of business threads, and instructs each of the business threads to start data processing;
In each business thread instructed to start data processing, the status information and the database identification corresponding to the data processing to be performed are added to the thread status data corresponding to the own thread before starting the data processing of the own thread. Information, the data processing method information, the data size information, and the start time information are registered, and when the data processing of the own thread is completed, the status information of the thread status data corresponding to the own thread is completed. Information registration / update processing to update to information indicating that,
The database identification information and the data of the thread status data corresponding to the business thread for a business thread that has not yet been processed with reference to each thread status data registered in the status list at a predetermined cycle From the delay reference value information registered in the delay determination list corresponding to the processing method information, the data size information and the time information of the thread status data corresponding to the business thread, data processing of the business thread When determining whether there is a delay and detecting a delay, a delay determination process for updating the status information of the thread status data corresponding to the business thread to information indicating that the delay is in progress;
After one of the plurality of business threads completes data processing, a delay occurs with reference to the status information, the database identification information, and the data processing method information registered in each thread status data. If there is no business thread that is not yet processed, the computer is caused to execute output control processing that collectively outputs processing result data of all business threads that have been processed at that time to the outside. .

  According to the present invention, in a data processing system in which a plurality of data processes are simultaneously executed by a computer and a plurality of databases are updated during the execution of the data processes, a delay is expected in data processing using some databases. Even if it occurs outside, it is possible to batch write the processing result data in the other multiple data processing not affected by the delay to the storage device, and as a result, the other data not affected by the delay There is an effect that the processing result data in a plurality of data processing can be used without delay.

It is a block diagram which shows the structure of embodiment of this invention. It is a flowchart which shows the process in the business thread control part 121 in embodiment of this invention. It is a flowchart which shows the process in the delay determination part 122 in embodiment of this invention. It is a flowchart which shows the process in the output control part 123 in embodiment of this invention. It is a flowchart which shows the process which the business thread in embodiment of this invention performs. It is a figure which shows an example of the processing flow of the business thread in embodiment of this invention. It is a figure which shows the mode of a change with the business thread status list | wrist and the delay determination list | wrist in embodiment of this invention. It is a block diagram which shows the structure of the general data processing system which performs the process accompanying the update of a database. It is a figure which shows the processing flow of the business thread | sled of the general data processing system which performs the process accompanying the update of a database.

  Next, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing the configuration of the embodiment of the present invention.

  Referring to FIG. 1, the data processing system according to the present embodiment includes a data processing device 100, a storage device 200 storing an input file 210, a storage device 400 storing an output file 410, and a plurality of files. Are composed of three databases 300-1 to 300-3.

  The data processing apparatus 100 is configured by a general information processing apparatus (computer) that operates under program control.

  The storage devices 200 and 400 are realized by, for example, a disk device or a semiconductor memory device.

  Each of the at least one input file 210 includes a large amount of input data to be subjected to data processing performed by the data processing apparatus 100, information specifying a processing method for each input data, and the like.

  The storage device 400 stores an output file 410 generated by the data processing device 100.

  The data processing apparatus 100 includes a data processing execution process 110 and a memory 160.

  That is, in this embodiment, an operating system (not shown) mounted on the data processing apparatus 100 generates and starts a data processing execution process 110 in advance in order to perform various data processing on the data processing apparatus 100. Suppose that The “process” here is a concept well known in the computer field, and detailed description thereof is omitted.

  The active data processing execution process 110 activates and makes the four business threads 130a to 130d that actually perform data processing and the control thread 120 that controls the business threads 130a to 130d resident.

  The memory 160 stores a business thread status list 140 that stores the current status of data processing of all the business threads 130a to 130d, and a reference value for the time required for each process for each database 300-1 to 300-3. The delay determination list 150 is stored, and the control thread 120 and the business threads 130a to 130d can access the lists 140 and 150.

  The control thread 120 includes a business thread control unit 121, a delay determination unit 122, and an output control unit 123.

  The business thread control unit 121 reads input data from the input file 210 and controls the four business threads 130a to 130d to start data processing.

  The delay determination unit 122 periodically refers to the business thread status list 140 and determines whether or not there is a data processing delay in the business threads 130a to 130d based on information in the delay determination list 150. In the following description, a business thread that is determined by the delay determination unit 122 to have a delay in data processing is referred to as a “delayed business thread” or a “delayed business thread”.

  The output control unit 123 controls output of processing results in the four business threads 130a to 130d. That is, the business threads 130a to 130d do not directly output the processing results of these to the file 410, but temporarily pass them to the output control unit 123. The output control unit 123 to which the processing result is passed outputs the processing result to the output file 410 at a timing determined by itself. Further, the output control unit 123 instructs the business threads 130a to 130d to commit the database updated by the business threads 130a to 130d.

  The business threads 130a to 130d process the processing data groups given from the input file 210 via the business thread control unit 121 in parallel by the designated data processing method, and the business thread status list 140 and the delay determination list 150 is updated at any time.

  In the present embodiment, the number of business threads 130 is four and the number of databases is three. However, these are examples, and the present invention is not limited to these.

  FIG. 6 shows a flow of data processing in the business threads 130a to 130d.

  FIG. 7 shows the contents of the business thread status list 140 and the delay determination list 150 at each point in time T1 to T5 in FIG.

  Assume that the delay determination list 150 stores in advance a delay reference value obtained or arbitrarily determined from the data processing speed in the previous data processing.

  The delay determination list 150 registers a data processing method column for registering data processing method information indicating a data processing method, and database identification information for identifying a database updated by the data processing method identified by the data processing method information. The update DB field includes a delay reference value field for registering delay reference value information that is the delay reference value. The delay reference value is a value obtained by multiplying the processing time per 1 KB of processing data group in the previous data processing by (1 + acceptable delay rate).

  For example, looking at the second line of the delay determination list (at time T1) in FIG. 7, “processing method A” is displayed in the data processing method column, “DB-1” is displayed in the update DB column, and the delay reference value is displayed. 1.5 is registered in the column. This indicates that the data processing method is “processing method A” and the database to be updated is “DB-1”.

  In such data processing, when the processing time per 1 KB becomes 1.6, the value of 1.6 exceeds 1.5 which is the value indicated by the corresponding delay reference value column. Data processing of processing method A targeting -1 is regarded as “delayed”. That is, the delay reference value is a threshold value for determining a data processing delay.

  The allowable delay rate is an arbitrary value set in accordance with the contents of data processing work. In this embodiment, the allowable delay rate is 0.5.

  In this embodiment, it is assumed that the delay determination list 150 is set in advance as shown in the delay determination list (time T1) in FIG.

  Next, the operation of the present embodiment will be described in detail with reference to FIGS.

  First, when an input file 210 including a large amount of input data to be subjected to data processing is given from the storage device 200 to the data processing device 100, the business thread control unit 121 of the control thread 120 deletes all data in the input file 210. Read (step S101 in FIG. 2).

  Next, the business thread control unit 121 specifies a processing method of each input data read from the input file 210, and creates four processing data groups by collecting the read input data according to related processing methods according to the information. (Step S102 in FIG. 2).

  Next, the business thread control unit 121 passes the four processing data groups to the four business threads 130a to 130d in a one-to-one correspondence, designates a data processing method for each of the processing data groups, and executes the processing. The start of processing for the data group is instructed (step S103 in FIG. 2).

  Here, it is assumed that the business thread control unit 121 passes a processing data group having a data size of 500 KB to the business thread 130a and instructs the start of data processing. Furthermore, it is assumed that the business thread control unit 121 specifies processing method A that involves updating the database 300-1 (DB-1) as a data processing method.

  Similarly, the business thread control unit 121 instructs the business thread 130b to start data processing by passing the 300KB, 600KB, and 500KB processing data groups to the business threads 130b to 130d, respectively. Processing method B with update of the database 300-2 (DB-2) as a processing method B with update of the database 300-3 (DB-3) as a data processing method for the business thread 130c, and Assume that the processing method B accompanied by the update of the database 300-2 (DB-2) is designated as the data processing method for the business thread 130d.

  Next, the operation on the business thread side instructed to start the data processing will be described in detail with reference to FIG. 5 using the business thread 130a as a representative example. Further, the interrelationship of the operation flow of the four business threads 130a to 130d will be described with reference to FIGS.

  First, the business thread 130a registers information on data processing to be performed in the business thread status list 140 (step S401 in FIG. 5). The time at this time is T1. At this time, the business threads 130b to 130d also start data processing in parallel at time T1, as shown in FIG.

  As shown in FIG. 7, the business thread status list 140 includes a thread ID field for registering thread ID information for identifying a business thread for executing data processing, a status field for registering status information indicating the current status of data processing, and data Update DB column for registering database identification information for identifying a database to be updated during processing, data processing method column for registering data processing method information indicating a data processing method, and data size of a given processing data group in KB units The data size column for registering the data size information, which is the value represented by (2), and the start time column for registering the start time information indicating the time when the data processing is started.

  The second and subsequent lines of the business thread status list 140 have a one-to-one correspondence with a plurality of business threads, and the contents (information) of all the columns in each line are collectively referred to as thread status data.

  Here, in the thread ID column, “a”, “b”, “c”, and “d” are thread IDs for identifying the business threads 130a, b, c, and d, respectively, and “DB-1” in the update DB column. "," DB-2 "and" DB-3 "are database identification numbers for identifying the databases 300-1, 300-2 and 300-3, respectively.

  As shown in the business thread status list of FIG. 7 (at time T1), before starting data processing, the business thread 130a displays “a” indicating its own thread in the thread ID column, “processing” in the status column, and the update DB. Register "DB-1" indicating the database 300-1 in the column, "Processing method A" in the data processing method column, "500" in the data size column, and "T1" which is the current time in the start time column.

  At this time, the business threads 130b to 130d also register each piece of information corresponding to their own thread in each column of the business thread status list (at time T1).

  Next, the business thread 130a performs data processing of the designated “processing method A” (step S402 in FIG. 5). During the data processing, the business thread 130a updates the database 300-1 as needed. At this time, the commit of the database 300-1 is not performed, and the output control unit 123 waits for a commit instruction.

  In addition, after the start of data processing in the business threads 130a to 130d, the delay determination unit 122 performs a delay determination of each business thread 130a to 130d at a constant cycle, and the delayed business threads are listed in the business thread status list 140. Rewrite the status column to “Delayed”. In this embodiment, the cycle of the delay determination operation is every 100 ms. Detailed operation of the delay determination will be described later.

  Next, the operation at time T2 after 400 ms has elapsed from the data processing start time T1 will be described.

  It is assumed that the data processing of the processing data group to which the business thread 130a is given is completed at time T2. As shown in FIG. 6, it is assumed that none of the business threads 130b to 130d has yet completed data processing at time T2, and there is no business thread 130 that is determined to be “delayed” at this time. .

  When the data processing is completed, the business thread 130a confirms the status of its own thread in the current business thread status list 140 (that is, the business thread status list (time T1 in FIG. 7)) (step S403 in FIG. 5). As described above, at time T2, the business thread 130a is not determined to be “delayed”, and the status remains “processing”.

  Since the status column of the own thread is not “delayed”, the business thread 130a calculates the delay reference value from the processing time required for the current data processing and updates the delay determination list 150 (step S404 in FIG. 5). . Here, as described above, the delay reference value is obtained by multiplying the processing time per 1 KB of processing data group required for data processing by (1 + delay tolerance). Since the business thread 130a starts data processing at time T1 and ends processing at time T2 after 400 ms has elapsed, the required time is 400 ms, and the size of the processing data group is 500 KB. Accordingly, the delay reference value in the current data processing of the business thread 130a is obtained by the following calculation formula.

Processing time per 1 KB = required time / size of processing data group = 400/500 = 0.8 (ms / KB)
Delay reference value = processing time per 1 KB * (1 + delay tolerance) = 0.8 * 1.5 = 1.2 (ms / KB)
The business thread 130a is a value obtained by calculating the content of the delay reference value column that satisfies the condition that the data processing method column of the delay determination list 150 is “processing method A” and the update DB column is “DB-1”. It is rewritten to 1.2 (refer to the delay judgment list in FIG. 7 (time T2)).

  Next, the business thread 130a changes the status column of its own thread in the business thread status list 140 from “processing” to “completed” (see the business thread status list (at time T2 in FIG. 7)) (step in FIG. 5). In step S405, the processing result data is transferred and a processing completion notification is transmitted to the output control unit 123 of the control thread 120 (step S406 in FIG. 5).

  Thereafter, the business thread 130a waits until a commit instruction is received from the control thread 120 (step S407 in FIG. 5).

  Next, the operation of the output control unit 123 that has received the processing completion notification and the processing result data from the business thread 130a as described above will be described with reference to FIG.

  First, the output control unit 123 stores the processing result data passed from the business thread 130a in a buffer area (not shown) provided in the memory 160 (step S301 in FIG. 4).

  Next, the output control unit 123 checks whether or not there is a business thread “processing” at the present time (time T2) (step S302 in FIG. 4). That is, the output control unit 122 looks at the status column of the business thread status list 140 (business thread status list (at time T2 in FIG. 7)) in order from the top, and determines whether there is a “processing” business thread. Check. As a result, the output control unit 123 recognizes that the three business threads 130b to 130d whose thread IDs are “b” to “d” are “processing”.

  Since the “in-process” business threads 130b to 130d exist, the output control unit 123 next refers to the status column of the business thread status list 140 (the business thread status list (at time T2 in FIG. 7)). Each of the “in-process” business threads 130b to 130d is another business thread that updates the same update DB as the update DB updated by the “in-process” business thread, and is the “in-process” business thread It is confirmed whether the other business thread that is executing the same data processing method as that executed by is not “delayed” (step S310 in FIG. 4). Since there is no “delayed” business thread at time T2, the output control unit 123 performs data processing (that is, “DB-2” or “DB-3”) in the “processing” business threads 130b to 130d. It is recognized that there is no delay in the “processing method B” data processing).

  Thereafter, the output control unit 123 returns to the confirmation of the existence of the “processing” business thread in step S302 of FIG. 4 after a predetermined time has elapsed (step S320 of FIG. 4). That is, since the business threads 130b to 130d are normally processing without delay at the time T2, the output control unit 123 completes the processing of the business threads 130b to 130d or is regarded as a delay. Then, the output of the processing result data from the business thread 130a stored in the buffer area to the storage device 400 is awaited.

  Next, the operation at time T3 after 500 ms has elapsed from time T1 will be described.

  As described above, since the delay determination unit 122 performs the delay determination operation at a cycle of 100 ms after the data processing is started in the business threads 130a to 130d, the delay determination unit 122 also performs the delay determination at the time T3.

  As shown in FIG. 6, the business threads 130b-d have not yet finished data processing at this point.

  With reference to FIG. 3, the delay determination operation of the delay determination unit 122 at time T3 will be described.

  First, the delay determination unit 122 looks at the status column of the business thread status list 140 (the business thread status list (at time T2 in FIG. 7)) in order from the top, and searches for a business thread of “processing” (FIG. 3). Step S201). At this time, the business thread 130a is “completed” and therefore does not correspond, and the business thread 130b first corresponds. If there is no “processing” business thread, the delay determination unit 122 ends the delay determination operation.

  Next, the delay determination unit 122 displays the status column from the business thread status list 140 (business thread status list (time T2 in FIG. 7)) and the delay determination list 150 (delay determination list (time T2 in FIG. 7)). A delay reference value corresponding to the data size of the processing data group being processed by the business thread being “processing”, the processing start time, and the data processing method is acquired (step S202 in FIG. 3).

  Here, as the various information of the business thread 103b, the data size is “300KB” from the data size column of the business thread status list (at time T2), and the start time is “T1” from the start time column of the list. I understand that. Further, from the data processing method column and the update DB column in the business thread status list (at time T2), the data processing method and the update DB corresponding to the business thread 130b are “processing method B” and “DB-2”, respectively. As can be seen, the delay of the list that satisfies the condition that the content of the data processing method column of the delay determination list (at time T2) is “processing method B” and the content of the update DB column of the list is “DB-2” It can be seen from the contents of the row in the reference value column that the delay reference value is “1.5”.

  Next, the delay determination unit 122 determines whether or not the business thread performing the delay determination is delayed at time T3 (step S203 in FIG. 3). The method is as follows.

  First, the delay determination unit 122 calculates the processing time per tentative processing data group 1 KB on the assumption that the data processing in the business thread 130b has now ended. The calculation formula is as follows.

Temporary processing time per 1 KB = (current time−start time) / data size = 500 / 300≈1.7
Next, the delay determination unit 122 compares the calculated processing time with the delay reference value acquired in step S202 of FIG. 3, and if the calculated processing time is equal to or greater than the delay reference value, the delay determination unit 122 sets the business thread 130 to be used. It is determined that “delayed”. That is, the processing time per provisional 1 KB of the business thread 130b that has not finished processing at time T3 is longer than 1.7 and is the delay reference value of 1.5 or more. Is determined.

  When the determination result is “delayed”, the delay determination unit 122 changes the status column corresponding to the delayed business thread in the business thread status list 140 to “delayed” (step S204 in FIG. 3). Here, since it is determined that the business thread 130b is being delayed, the status column corresponding to the business thread 130b is updated to “delayed” as shown in the business thread status list (at time T3) in FIG.

  If it is determined in step S203 in FIG. 3 that there is no delay, the process proceeds to the next “in process” business thread without doing anything.

  The delay determination unit 122 performs these operations on all “processing” business threads. That is, in the present embodiment, the remaining business thread 130c and business thread 130d are also executed.

  In step S203 in FIG. 3, the provisional processing time per 1 KB is calculated to be about 0.8 for the business thread 130c and 1 for the business thread 130d. Therefore, the delay determination unit 122 determines which of the processing times at time T3. It is determined that the thread is not delayed. As a result, the contents of the business thread status list 140 are as shown in the business thread status list (at time T3) in FIG.

  The delay determination unit 122 performs the delay determination operation as described above periodically (every 100 ms).

  Next, the operation at time T4 after 700 ms has elapsed from time T1 will be described.

  It is assumed that the data processing of the processing data group to which the business thread 130c is given is completed at time T4. As shown in FIG. 6, at time T4, the business thread 130a has already completed processing, and the business thread 130b and the business thread 130d that have been determined to be delayed have not yet completed data processing. .

  First, when completing the data processing, the business thread 130c performs the processing up to step S407 in FIG. 5 as described above. That is, the business thread 130c satisfies the condition that the content of the data processing method column of the delay determination list 150 is “processing method B” and the content of the update DB column of the list is “DB-3”. The content of the row in the delay reference value column is rewritten to “1.8” (see the delay determination list (at time T4) in FIG. 7), and the status column of the own thread in the business thread status list 140 is changed to “completed” ( The processing thread status list (refer to the time T4 in FIG. 7) and the processing result data are delivered to the output control unit 123 and a processing completion notification is transmitted.

Hereinafter, the operation of the output control unit 123 that has received the process completion notification will be described only with respect to the operation at the time T2. First, the output control unit 123 receives the process completion notification and the processing result data from the business thread 130c. When received, the processing result data is stored in the buffer area in the memory 160 together with the processing result data of the previous business thread 130a (step S301 in FIG. 4).

  Next, the output control unit 123 checks whether or not there is a “processing” business thread at the current time (time T4), and recognizes that only the business thread 130d is “processing” at this time (FIG. 5). 4 step S302).

  Since there is a “processing” business thread 130d, the output control unit 123 searches for a “delaying” business thread, and this time recognizes that the business thread 130b is delayed. Further, the output control unit 123 refers to the business thread status list 140 (the business thread status list in FIG. 7 (at time T4)), and updates the update DB and the data to be updated in the “processing” business thread 130d. The processing method is compared with the update DB and data processing method to be updated in the “delayed” business thread 130b. In both cases, the update DB is “DB-2”, and the data processing method is “processing method B”, which coincides with each other. Therefore, the output control unit 123 performs the data processing performed in the “processing” business thread 130d. It is recognized that there is a delay in “the same data processing method for updating the same database”, ie, “processing method B” for updating “DB-2”. Therefore, the output control unit 123 considers that the processing of the business thread 130d is also delayed. Since there is no “in-process” business thread other than the business thread 130d, it is determined that there is no “in-process” business thread that has no delay in data processing and continues normal processing.

  In other words, the output control unit 123 is another business thread that updates the same update database as the update beta base that is updated by the “in-process” business thread for each “in-process” thread. It is determined whether the other business thread that executes the same data processing method as the data processing method executed by the “processing in progress” business thread is “delayed” (step S310 in FIG. 4).

  Therefore, the output control unit 123 outputs only the processing result data in the business threads 130a and 130c that have “completed” the data processing at this time (step S303 in FIG. 4). That is, the processing result data in the business threads 130 a and 130 c stored in the buffer area of the memory 160 are collected into one output file 410-1 and output to the storage device 400.

  Thereafter, the output control unit 123 transmits a commit instruction to the business threads 130a and 130c (step S304 in FIG. 4).

  Upon receiving the commit instruction (step S408 in FIG. 5), the business threads 130a and 130c perform commit processing on the updated databases 300-1 and 300-3, respectively, and confirm the database update (step S409 in FIG. 5). ).

  Finally, the business threads 130a and 130c delete the contents of the business thread status list 140 (the business thread status list in FIG. 7 (at time T4)) in the fields other than the thread ID column corresponding to the own thread. The process related to the processing data group instructed by the thread control unit 121 is terminated.

  In this way, the output control unit 123 does not wait for the completion of processing between the delayed business thread 130b and the business thread 130d deemed to be delayed, and the processing result data of the business threads 130a and 130c that have completed processing earlier. File output and database update commit are performed collectively, and the business threads 130a and 130c are released.

  Hereinafter, a description will be given of the delayed business thread 130b and the business thread 130d deemed to be delayed up to the completion of data processing.

  It is assumed that the business thread 130b completes the delayed data processing at time T5 after 900 ms has elapsed from time T1. At this time, as shown in FIG. 6, the business thread 130d has not yet finished data processing. Furthermore, the business thread 130d is determined that a delay has occurred by the delay determination unit 122 after 750 ms from the time T1 (in this embodiment, the determination cycle is every 100 ms and at 800 ms), and the business thread in the business thread status list 140 is determined. The contents of the status column corresponding to 130d have already been rewritten as “Delayed”.

  When the business thread 130b completes the data processing, the business thread 130b checks the content of the status column of its own thread in the business thread status list 140, and determines whether or not the content is “delayed” (step S403 in FIG. 5). ). Since the content of the status column at this time is “delayed”, the business thread 130b does not update the delay determination list 150 and performs the processing from step S405 of FIG.

  That is, the business thread b updates the content of the status column corresponding to its own thread in the business thread status list 140 to “completed” (see the business thread status list (at time T5 in FIG. 7)) (step S405 in FIG. 5). Then, the processing result data is transferred and a processing completion notification is transmitted to the output control unit 123 of the control thread 120 (step S406 in FIG. 5). Thereafter, the business thread 130a waits until a commit instruction is received from the control thread 120 (step S407 in FIG. 5).

  The output control unit 123 that has received the processing completion notification and the processing result data from the business thread 130b refers to the content of the status column of the business thread status list 140 (the business thread status list (at time T5 in FIG. 7)), and the content. It is determined whether or not there is a business thread whose status is “processing” (step S302 in FIG. 4). Since there is no business thread whose status is “processing”, the business processing whose data processing is “completed” is determined. Only the processing result data in the thread 130b is output to the storage device 400 as a new output file 410-2 (step S303 in FIG. 4), and a commit instruction is transmitted to the business thread 130b (step S304 in FIG. 4).

  Next, it is assumed that the last remaining business thread 130d completes the delayed data processing at time T6 after 1500 ms has elapsed from time T1. The operation of the business thread 130d after the completion of data processing is exactly the same as that of the business thread 130b described above. In addition, the operation of the output control unit 123 that has received the processing completion notification from the business thread 130d is exactly the same as the operation for the business thread 130b described above.

  With the above operation, at time T4 (700 ms from time T1), the output file 410-1 including the processing result data in the business threads 130a and 130c is output to the storage device 400, and time T5 (900 ms has elapsed from time T1). ), The output file 410-2 including the processing result data in the business thread 130b is output to the storage device 400, and the output file including the processing result data of the business thread 130d at time T6 (1500 ms elapses from time T1). 410-3 is output to the storage device 400. At time T4, the databases 300-1 and 300-3 are committed prior to the database 300-2.

  As described above, according to the present embodiment, even when a delay occurs in the data processing in the business thread 130b and the business thread 130d for executing the “processing method B” for updating the database 300-2, the delay Databases 300-1 and 300-3 unrelated to the delayed business thread 130b and business thread 130d are first output to the storage device 400 and output files containing the processing result data of the business threads 130a and 130c that are not affected. Can be committed first.

  In addition, the processing result data in the business threads 130a and 130c that are not delayed can be written into the storage device 400 in a lump, so that the writing process can be completed once. A reduction in writing throughput is suppressed as much as possible.

  As described above, in this embodiment, a part of databases is used in a data processing system in which a plurality of data processes are simultaneously executed by a computer and a plurality of databases are updated during the execution of the data processes. Even if a delay occurs in the data processing to be performed unexpectedly, it is possible to batch write the processing result data in the other multiple data processing that is not affected by the delay to the storage device. As a result, the influence of the delay Thus, there is an effect that it is possible to use the processing result data in the other plurality of data processing that has not been received without delay.

  The reason is that it detects a thread with a data processing delay, and assumes that other threads that perform data processing using the same data processing method for the same database as the database used by the thread in which the delay has occurred will also cause a delay. This is because, without waiting for the completion of the processing of these delay threads, only the processing result data in the threads that are not affected by the delay are written into the storage device in a batch.

  Note that the start time column of the business thread status list 140 may be time information for determining a delay, and may be, for example, a delay confirmation time that is regarded as a delay when the completion of processing is further delayed.

100, 500 Data processing device 110, 510 Data processing execution process 120, 520 Control thread 121 Business thread control unit 122 Delay determination unit 123 Output control unit 130a-d, 530a-d Business thread 140 Business thread status list 150 Delay determination list 160 Memory 200, 400, 700, 800 Storage device 210, 600 Input file 300-1 to 3 and 610-1 to 3 Database 410-1 to 3 and 620 Output file

Claims (9)

In a data processing apparatus that executes a plurality of data processes concurrently in a plurality of business threads generated on the own apparatus,
A thread status list, a delay determination list, a control unit, a delay determination unit, and an output control unit;
The status list has a plurality of thread status data corresponding one-to-one to the plurality of business threads,
Each of the thread status data is updated by ID information for identifying the corresponding business thread, status information indicating a current processing status of data processing performed by the corresponding business thread, and data processing performed by the corresponding business thread. Database identification information for identifying a database to be processed, data processing method information for identifying a data processing method in the corresponding business thread, and data of a processing target data group that is a target of data processing performed by the corresponding business thread Size information and start time information indicating the start time of data processing performed by the corresponding business thread,
The delay determination list includes each of the data processing method information, the database identification information for identifying the database updated by the data processing method identified by each of the data processing method information, the data processing method information, and the Delay reference value information obtained from the processing time per predetermined unit size of the processing target data group in the previous processing of each data processing corresponding to each combination with the database identification information,
The control unit reads a plurality of processing target data groups given from the outside and distributes them to the plurality of business threads, and instructs each business thread to start data processing,
Each of the business threads includes the status information, the database identification information, and the data processing method information corresponding to the data processing to be performed from now on, before starting the data processing of the own thread. The data size information and the start time information are registered, and when the data processing of the own thread is completed, the status information of the thread status data corresponding to the own thread is updated to information indicating that the data processing is completed And
The delay determination unit refers to each thread status data registered in the status list at a predetermined period, and for the business thread that has not yet been processed, the thread status data corresponding to the business thread From the delay reference value information registered in the delay determination list corresponding to the database identification information and the data processing method information, the data size information and the time information of the thread status data corresponding to the business thread, The presence / absence of a delay in data processing of the business thread is determined, and when the delay is detected, the status information of the thread status data corresponding to the business thread is updated to information indicating that the process is being delayed,
The output control unit refers to the status information, the database identification information, and the data processing method information registered in each thread status data after any of the plurality of business threads completes data processing. When no delay occurs and there is no business thread that is still being processed, the processing result data of all the business threads that have completed processing at that time is output to the outside at once. Data processing device. The output control unit regards that another business thread that performs data processing by the same data processing method as a database that is updated by the business thread in which the delay has occurred is also regarded as having a delay. Item 2. A data processing apparatus according to Item 1. In the output of the processing result data, the output control unit instructs all the business threads that have completed processing at that time to commit the database updated by each business thread and confirms the update result 3. The data processing apparatus according to claim 1, wherein the data processing apparatus is a data processing apparatus. While executing multiple data processing concurrently in multiple business threads generated on its own device,
ID information for identifying the corresponding business thread, status information indicating the current processing status of data processing performed by the corresponding business thread, and a database to be updated by data processing performed by the corresponding business thread Database identification information, data processing method information for identifying a data processing method in the corresponding business thread, data size information of a processing target data group to be subjected to data processing performed by the corresponding business thread, and the corresponding A thread status list including a plurality of thread status data each corresponding to the plurality of business threads, each having start time information indicating a start time of data processing performed by the business threads;
Each of the data processing method information, the database identification information for identifying the database updated by the data processing method identified by each of the data processing method information, each of the data processing method information and the database identification information A data processing method for use in a data processing apparatus comprising: a delay determination list having delay reference value information obtained from a processing time per predetermined unit size of a processing target data group in the previous processing of each data processing corresponding to a combination In
A data processing start step for reading a plurality of processing target data groups given from the outside and distributing them to the plurality of business threads, and instructing the business threads to start data processing;
In each business thread instructed to start data processing, the status information and the database identification corresponding to the data processing to be performed are added to the thread status data corresponding to the own thread before starting the data processing of the own thread. Information, the data processing method information, the data size information, and the start time information are registered, and when the data processing of the own thread is completed, the status information of the thread status data corresponding to the own thread is completed. A data processing step for updating to information indicating that,
The database identification information and the data of the thread status data corresponding to the business thread for a business thread that has not yet been processed with reference to each thread status data registered in the status list at a predetermined cycle From the delay reference value information registered in the delay determination list corresponding to the processing method information, the data size information and the time information of the thread status data corresponding to the business thread, data processing of the business thread A delay determination step of determining whether there is a delay and detecting the delay, updating the status information of the thread status data corresponding to the business thread to information indicating that the delay is in progress;
After one of the plurality of business threads completes data processing, a delay occurs with reference to the status information, the database identification information, and the data processing method information registered in each thread status data. And an output control step for collectively outputting processing result data of all the business threads that have completed processing at that time when there is no business thread that is not yet processed. Data processing method. In the output control step, another business thread that performs data processing by the same data processing method with respect to the same database as the database that is updated by the business thread in which the delay has occurred is regarded as having a delay. Item 5. A data processing method according to Item 4. In outputting the processing result data in the output control step, instructing all business threads that have completed processing at that time to commit the database updated by each business thread and confirming the update result The data processing method according to claim 4 or 5, wherein: While executing multiple data processing concurrently in multiple business threads generated on the own computer,
ID information for identifying the corresponding business thread, status information indicating the current processing status of data processing performed by the corresponding business thread, and a database to be updated by data processing performed by the corresponding business thread Database identification information, data processing method information for identifying a data processing method in the corresponding business thread, data size information of a processing target data group to be subjected to data processing performed by the corresponding business thread, and the corresponding A thread status list including a plurality of thread status data each corresponding to the plurality of business threads, each having start time information indicating a start time of data processing performed by the business threads;
Each of the data processing method information, the database identification information for identifying the database updated by the data processing method identified by each of the data processing method information, each of the data processing method information and the database identification information In a data processing program executed by a computer comprising a delay determination list having delay reference value information obtained from a processing time per predetermined unit size of a processing target data group in a previous processing of each data processing corresponding to a combination ,
A data processing start process that reads a plurality of processing target data groups given from the outside and distributes them to the plurality of business threads, and instructs each of the business threads to start data processing;
In each business thread instructed to start data processing, the status information and the database identification corresponding to the data processing to be performed are added to the thread status data corresponding to the own thread before starting the data processing of the own thread. Information, the data processing method information, the data size information, and the start time information are registered, and when the data processing of the own thread is completed, the status information of the thread status data corresponding to the own thread is completed. Information registration / update processing to update to information indicating that,
The database identification information and the data of the thread status data corresponding to the business thread for a business thread that has not yet been processed with reference to each thread status data registered in the status list at a predetermined cycle From the delay reference value information registered in the delay determination list corresponding to the processing method information, the data size information and the time information of the thread status data corresponding to the business thread, data processing of the business thread When determining whether there is a delay and detecting a delay, a delay determination process for updating the status information of the thread status data corresponding to the business thread to information indicating that the delay is in progress;
After one of the plurality of business threads completes data processing, a delay occurs with reference to the status information, the database identification information, and the data processing method information registered in each thread status data. If there is no business thread that is still being processed, the computer executes the output control process that outputs the processing result data of all the business threads that have completed processing at the same time to the outside. A data processing program characterized by that. In the output control processing, another business thread that performs data processing by the same data processing method with respect to the same database as the database that is updated by the business thread in which the delay has occurred is regarded as having a delay. Item 8. A data processing program according to Item 7. In outputting the processing result data in the output control process, for all business threads that have completed the processing at that time, instruct the commit of the database updated by each business thread to confirm the update result 9. A data processing program according to claim 7 or claim 8.

Images