CN102393822B - Batch scheduling system and method - Google Patents
Batch scheduling system and method Download PDFInfo
- Publication number
- CN102393822B CN102393822B CN201110391457.6A CN201110391457A CN102393822B CN 102393822 B CN102393822 B CN 102393822B CN 201110391457 A CN201110391457 A CN 201110391457A CN 102393822 B CN102393822 B CN 102393822B
- Authority
- CN
- China
- Prior art keywords
- task
- processing
- unit
- tasks
- channel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a batch scheduling system comprising the following devices: a parameter setting device, by which a user sets system parameters and task processing related parameters, and stores the set parameters in a data storage device; a preprocessing device for preprocessing batch tasks and outputting the tasks generated after processing into the data storage device; a scheduling engine device for periodically detecting the current quantity of the tasks concurrently processed by the system, calculating the number of distributable threads for processing the batch tasks submitted by each channel of the system, and outputting the result to a task processing device; and the task processing device for receiving the number of the batch task processing threads transmitted from the scheduling engine device, regulating internal threads, reading the preprocessed tasks according to the priority parameters of the tasks, and processing the tasks. Through the system, the efficiency of processing batch tasks can be improved, and the system stability can be improved.
Description
Technical field
The present invention relates to batch data processing technology field, be specifically related to a kind ofly promote calculation process efficiency, effectively utilize system resource to realize the system and method for batch processing.
Background technology
In large-scale online trading system, always there are a large amount of batch jobs.Generally speaking, the tupe of batch jobs mainly contains two kinds:
The one, take the mode of online real-time processing, the online short run of namely often saying.This kind of method is ageing very high, but online short run mode is when processing batch tasks with on-line mode, do not judge the busy of data handling system, in real time, send real-time processing, because task closeness in batch tasks is high, by taking rapidly a large amount of system resource, to system, bring great pressure, cause whole system processing speed to reduce, even have the machine risk of delaying; And batch tasks processes generally can to concentrate and use some or certain several user data, user data used is pinned for a long time, makes client's online online task need to wait for the releasing of data interlock, causes affecting the online task of client.
Another batch tasks processing mode is the mode of taking according to play Timing Processing, is generally arranged in the fixedly period that portfolio is less to focus on according to batch play, as morning, noon, night etc.
For this kind of mode, be that different pieces of information processing, statistics task in batch are carried out successively according to fixing order traditionally, new processing demands of every increase, just write a new task, add in the back, or the clear-cut handling procedure of directly revising a certain task, new processing code increased.Meanwhile, in order to improve the utilization rate of resource, for each task, according to the concurrent execution of data partition condition (as area, country, product classification etc.).But fixing task sequentially can be because indivedual tasks cause the execution time long, or the whole batch of impact is carried out because task is made mistakes; And minute Division is also more fixing, each subregion execution time length gap is larger, in execution, exists and waits for, time resource, system resource are not fully utilized.For conventional process mode, the patent No. is that ZL200610126976.9, denomination of invention have proposed a kind of dispatching method for " a kind of batch task scheduling engine and dispatching method ": the batch tasks of current play is carried out to static nature analysis, to obtain the monomer feature of each task of using Parametric Representation and the dependence between task; Dependence based between each task is divided into groups to task; According to the weights of each task, calculate the group weights of each group task, wherein organize weights and determine master scheduling order; According to subregion weights, determine the secondary dispatching sequence of the dispatching sequence of each subregion in group; Based on described master scheduling order and described secondary dispatching sequence, form the scheduling strategy of the batch tasks of current play; And utilize described scheduling strategy, carry out the scheduling of the batch tasks of current play.By the method, make batch processing task realize task, subregion two dimensions are concurrent, to time resource and the intensive utilization of hardware resource in the batch play of fixing period, promoted dirigibility, maintainability and the fault-tolerant ability of batch processing.
But the method for this patent is still batch tasks is arranged in the batch play of fixing period and is carried out, and exists following not enough: be first to process not in time, be difficult to meet ageing business need of having relatively high expectations.In addition,, along with increasing sharply of batch tasks, fixedly in the period, be difficult to maybe cannot complete whole batch tasks on the same day; Answer in contrast, within the period of non-batch play, there is the free time in system resource, is not fully used sometimes.If can break fixedly period batch play, not only can significantly increase the batch processing time, can also make data handling system resource be fully used.
Summary of the invention
In order to overcome the problems referred to above of prior art, the present invention proposes a kind of lot size scheduling system and method.
Lot size scheduling system of the present invention comprises: parameter setting apparatus, user, by this device, systematic parameter is set and task is processed correlation parameter, and by the Parameter storage arranging in data storage device, the time interval that wherein said systematic parameter comprises the maximum concurrent processing task amount of system and scanning system Real-time and Concurrent task amount, described task processing parameter comprises: task pretreatment parameter, for the batch tasks that determines to send on different channels that described system docking receives, the pretreatment mode carrying out according to task type; Priority of task order parameters, for determining the processing sequence between different task type; Pretreatment unit, for batch tasks is carried out to pre-service, and producing after processing of task is outputed in data storage device, it further comprises: batch tasks receiving element (301), be responsible for reading priority of task order parameters from the parameter storage unit (601) of data storage device (600), according to priority ranking from the batch tasks storage unit (602) of data storage device (600), read each channel and be submitted to the batch tasks in described system, export to task pretreatment unit (303); Task pretreatment parameter reads in unit (302), and this unit reads the pretreatment parameter of a certain task type from data storage device (600), exports to task pretreatment unit (303); Task pretreatment unit (303), obtain the batch tasks of batch tasks receiving element (301) input, and task pretreatment parameter reads in the pretreatment parameter corresponding to batch tasks of this kind of task type of unit (302) input, to batch tasks sort, aggregation process, producing after processing of task is outputed in the task location (603) of data storage device (600); Scheduling engine device, for the current concurrent processing task amount of timing detecting system, send the Thread Count of batch tasks on each channel of the assignable processing of computing system, and result is exported to Task Processing Unit; Task Processing Unit, for receiving the batch tasks processing threads number transmitting from scheduling engine device, adjusts internal thread, according to priority of task order parameters, reads through pretreated task, carries out task processing; Data storage device, for receive and the setting of stored parameter setting device or the parameter calculating, each channel on the batch tasks sent and the pretreatment unit task and the task that generate process daily record.
Wherein, said system also comprises task data analytical equipment, for recording the information relevant to task amount, calculates channel eigenwert and time point eigenwert in a certain period, and result of calculation is stored in data storage device.
Wherein, scheduling engine device, according to the maximum concurrent processing task amount of the system of obtaining from data storage device, the channel eigenwert of current period and time point eigenwert, calculates the Thread Count that send batch tasks on each channel of assignable processing.
Wherein, described scheduling engine device further comprises: task amount detecting unit, and according to the predefined time interval, the current concurrent processing task amount of system N described in timing acquisition, exports to scheduling decision unit, and wherein N is natural number; Scheduling decision unit, from the maximum concurrent processing task amount of system parameter setting unit reading system M, calculate M and whether be greater than N, if be greater than, calculate current period system and can distribute to the processing threads of the batch tasks that each channel uploads and count T, wherein M and T are natural number.
Wherein, Task Processing Unit further comprises: Thread control unit, the processing threads that system can be distributed to the batch tasks that each channel uploads counts T and the required processing threads number of Processing tasks amount N compares, if comparative result is that T is large, increases thread, and start the thread that should increase, if T is little, reduce thread, and the unnecessary thread of mark, after completing, its current task discharges this thread; Task receiving element obtains priority of task order parameters, according to priority sequence, reads T bar task; Task processing unit, processes described task according to reading order.
Wherein, the processing threads that system can be distributed to the batch tasks that each channel uploads is counted T and is calculated by following formula: T=(M-N) * Pa * Pb, wherein M represents the maximum concurrent processing task amount of system, N represents the current concurrent processing task amount of system, Pa=SUM (Sa)/SUM (S), Pa represents channel eigenwert, refer to the some periods in history, certain channel is to sending in described system task amount to account for the accounting of all task amounts that described system receives, Sa represents some day, the number of tasks that in a certain period, a certain channel is submitted to, SUM (Sa) represented in 1 year, the total task number that in the described period, described channel is submitted to, S represents some day, whole number of tasks that in a certain period, described system is received, SUM (S) represented in 1 year, the total task number that in the described period, described system is received, Pb=SUM (Sb)/SUM (S), Pb represents time point eigenwert, refer to the some periods in history, accounting in whole tasks that the batch tasks that described system receives receives in described system, Sb represents some day, the batch tasks number that in a certain period, described system receives, SUM (Sb) represented in 1 year, the batch tasks sum that in the described period, described system receives, S represents described some day, whole number of tasks that in a certain period, described system is received, SUM (S) represented in 1 year, the total task number that in the described period, described system is received.
Wherein, parameter setting apparatus further comprises: system parameter setting unit, and user arranges the maximum concurrent processing task amount of described system M by this element, and the time interval of scanning system Real-time and Concurrent task amount is set, and outputs in data storage device; Priority of task order parameters setting unit, user arranges the processing priority order of different task type by this element, output in data storage device; Task pretreatment parameter setting unit, user arranges the required pre-service done when receiving a batch tasks of described system by this element and works, and outputs in data storage device.
Lot size scheduling system and method provided by the invention, can make full use of system resource, and do not affecting under the prerequisite of online online task requests processing, round-the-clock processing batch tasks, can meet the ageing requirement for different business feature, can dynamically determine batch processing mode according to system load situation again, greatly improve system treatment effeciency, the risk of simultaneously effective control system overload.
Accompanying drawing explanation
Fig. 1 is lot size scheduling system construction drawing of the present invention;
Fig. 2 is the structural drawing of lot size scheduling system parameters setting device of the present invention;
Fig. 3 is the structural drawing of task data analytical equipment in lot size scheduling system of the present invention
Fig. 4 is the structural drawing of pretreatment unit in lot size scheduling system of the present invention;
Fig. 5 is the structural drawing of scheduling engine device in lot size scheduling system of the present invention;
Fig. 6 is the structural drawing of Task Processing Unit in lot size scheduling system of the present invention;
Fig. 7 is the structural drawing of memory storage in lot size scheduling system of the present invention;
Fig. 8 is that lot size scheduling system of the present invention is to task pretreatment process figure;
Fig. 9 is the process flow diagram to thread scheduling and task processing in lot size scheduling system of the present invention.
Embodiment
For making the object, technical solutions and advantages of the present invention clearer, below in conjunction with specific embodiment, and with reference to accompanying drawing, the present invention is described in more detail.
The invention provides a kind of batch data dispatching method and system, according to the task concurrency of data handling system, when system is idle, control and process the Thread Count that send batch tasks on each channel, thereby reach, both make full use of system resource, do not affect again the promptness that online online task requests is processed; Further, sequence, the account that first batch tasks carries out account when being submitted to native system such as gathers at the pre-service work, the complexity of processing with reduction system, improve the efficiency of processing batch tasks, thereby greatly reduce the risk of data handling system overload operation, improved the processing operational efficiency of system.
Fig. 1 is lot size scheduling system construction drawing of the present invention.As shown in Figure 1, this system comprises: parameter setting apparatus 100, task data analytical equipment 200, pretreatment unit 300, scheduling engine device 400, Task Processing Unit 500 and data storage device 600.
Wherein, parameter setting apparatus 100, task data analytical equipment 200, pretreatment unit 300, scheduling engine device 400, Task Processing Unit 500 are all connected with data storage device 600, and scheduling engine device 400 is also connected in Task Processing Unit 500.
Task data analytical equipment 200 is one group of software module, be deployed in data handling system, for being recorded in the different periods, the task amount of each channel, various task kinds, and according to the above-mentioned data analysis in a year, calculate channel eigenwert and time point eigenwert in a certain period, and result of calculation is stored in data storage device 600.Described channel eigenwert, refers to the some periods in history, and certain channel is to sending in described system task amount to account for the accounting of all task amounts that described system receives; Described time point eigenwert, in some periods of history, the accounting in whole tasks that the batch tasks that described system receives receives in described system; Described task kind comprises online task and batch tasks.
Scheduling engine device 400 is one group of software module, be deployed in data handling system, for the current concurrent processing task amount of timing detecting system, the maximum concurrent processing task amount of the system of obtaining from data storage device 600, the channel eigenwert of current period and time point eigenwert, calculate the Thread Count that send batch tasks on each channel of assignable processing, and result is exported to Task Processing Unit 500.
Task Processing Unit 500 is one group of software module, be deployed in data handling system, for receiving the batch tasks processing threads number transmitting from scheduling engine device 400, adjust internal thread, according to priority of task order parameters, read through pretreated task, carry out task processing, and result is recorded in daily record.
Above between each system module between corresponding physical equipment, by the local area network communication of enterprise, can be Ethernet (Ethernet), can be also other LAN (Local Area Network), as Fiber Distributed Data Interface (FDDI), token ring (Token-Ring) etc.
Fig. 2 is parameter setting apparatus 100 structural representations.With reference to Fig. 2, parameter setting apparatus 100 comprises: system parameter setting unit 101, priority of task order parameters setting unit 102, task pretreatment parameter setting unit 103 and period parameter set unit 104.
System parameter setting unit 101, user arranges the maximum concurrent processing task amount of described system M by this element, and the time interval of scanning system Real-time and Concurrent task amount is set, and outputs in data storage device 600.Wherein, M value can be used task that move, that the processing time is the longest in described system (for example using B2C to pay task) to carry out pressure test to described system, obtains the maximum concurrent processing task amount that described system can be born, namely maximum thread.M is actual is exceedingly the most odious in the situation that, the supported maximum thread of system.Priority of task order parameters setting unit 102, user arranges the processing priority order of different task type by this element, output in data storage device 600.Form below shows the priority ranking that some task types are corresponding by way of example.
| Task type | Processing order |
| Pay out wages | 1 |
| Repay | 2 |
| The remittance of transferring accounts | 3 |
| Buy fund | 4 |
| Other | 5 |
Task pretreatment parameter setting unit 103, user arranges the required pre-service done when receiving a batch tasks of described system by this element and works, and outputs in data storage device 600.In form below, show by way of example different task type is arranged to different pretreatment modes, wherein content of parameter can increase according to actual task processes or revise.
| Task type | Content of parameter |
| Pay out wages | By paying the square account single payment amount of money |
| Repay | By paying a square account single payment amount of money, by keeping accounts, account merges the amount of money that keeps accounts |
| The remittance of transferring accounts | By paying the square account single payment amount of money |
| Buy fund | By paying square account, fund title merging fund share |
| Other | Do not process |
Period parameter set unit 104, by this unit users according to the task treatment capacity of described system at same date not, the fluctuating of different time points, fluctuating characteristic, arrange several periods interval.
In conjunction with form below, this example shows based on certain Web bank of bank application transaction characteristics and formulates, monthly 1 day～9,11～19,21～31, trading volume is comparatively steady, its type of transaction is also similar with feature, wherein 0:00～7:59,11:59～13:29,17:00～0:00 are idle, are set to a period SD1; 8:00～11:59,13:30～16:59 are busy, are made as a period SD2.And within 10 days and 20 days, be payroll credit and bill payment day, trading volume there will be surge, specific to different time points, 0:00～7:59,11:59～13:29,18:00～0:00 are relative idle, be made as a period SD3,8:00～11:59,13:30～17:59 are busy, are made as a period SD4.4 periods as shown below are set like this, altogether.
As shown in Figure 3, Fig. 3 is task data analytical equipment 200 structural representations.Comprise: record cell 201 and computing unit 202.
Formula 1:Pa=SUM (Sa)/SUM (S)
Pa represents channel eigenwert, and in some periods of history, certain channel is to sending in described system task amount to account for the accounting of all task amounts that described system receives; Sa represents some day, the number of tasks that in a certain period, a certain channel is submitted to; SUM (Sa) represented in 1 year, the total task number that in the described period, described channel is submitted to; S represents above-mentioned some day, whole number of tasks that in a certain period, described system is received; SUM (S) represented in 1 year, the total task number that in the described period, described system is received.
Formula 2:Pb=SUM (Sb)/SUM (S)
Pb represents time point eigenwert, in some periods of history, and the accounting in whole tasks that the batch tasks that described system receives receives in described system; Sb represents some day, the batch tasks number that in a certain period, described system receives; SUM (Sb) represented in 1 year, the batch tasks sum that in the described period, described system receives; S represents above-mentioned some day, whole number of tasks that in a certain period, described system is received; SUM (S) represented in 1 year, the total task number that in the described period, described system is received.
Through calculating, can obtain channel eigenwert and the time point eigenwert of different periods on the same day, different channels.Form is below the concrete example of the time point list of feature values.
| Period sequence number | Task amount ratio (in batches: total amount) |
| SD1 | 1∶2 |
| SD2 | 5∶6 |
| SD3 | 1∶2 |
| SD4 | 1∶11 |
Upgrade and nearly 1 year of Dynamic Acquisition in data, for the analysis that makes historical data more accurately, have more reference value, and should periodically carry out for the analysis of historical data.
Fig. 4 is pretreatment unit 300 structural representations.With reference to Fig. 4, pretreatment unit 300 comprises: batch tasks receiving element 301, task pretreatment parameter read in unit 302 and task pretreatment unit 303 3 parts.Wherein, batch tasks receiving element 301 is connected with task pretreatment unit 303, and task pretreatment parameter reads in unit 302 and is connected with task pretreatment unit 303.
Batch tasks receiving element 301, be responsible for reading priority of task order parameters from the parameter storage unit 601 of data storage device 600, according to priority ranking from the batch tasks storage unit 602 of data storage device 600, read each channel and be submitted to the batch tasks in described system, export to task pretreatment unit 303;
Task pretreatment parameter reads in 302，Gai unit, unit and from data storage device 600, reads the pretreatment parameter of a certain task type, exports to task pretreatment unit 303;
Pre-service is not subject to the restrictions such as system condition, does not exist and conflicts with thread distribution, task processing etc., in real time batch tasks is processed.
Fig. 8 is the pretreatment process figure of task pretreatment unit 303.With reference to Fig. 8, in step 801, batch tasks receiving element 301 reads priority of task order parameters.In step 802, batch tasks receiving element 301 scanning batch tasks storage unit 602, have judged whether pending batch tasks.If any, enter step 803; As nothing, continue scanning.In step 803, batch tasks receiving element 301, according to priority of task order parameters, reads the batch tasks that priority is high, exports to task pretreatment unit 303.In step 804, task pretreatment parameter reads in unit 302 and reads pretreatment parameter, exports to task pretreatment unit 303.In step 805, task pretreatment unit 303 is received batch tasks and pretreatment parameter, and batch tasks is sorted, gathered.In step 806, pre-service result is outputed in task location 603, and log.In step 807, after completing, repeating step 802, and scanning batch tasks storage unit 602, carries out processing procedure next time.
Form below shows the batch tasks in batch tasks storage unit 602 by way of example.
According to priority of task order parameters, first process " paying out wages " task, according to pretreatment parameter " by paying the square account single payment amount of money ", pay square account and be AcctFrom, merge processing, obtain:
| Mission Number | Detailed sequence number | Task type | Pay square account | Produce the amount of money | Proceed to account | Treatment state |
| 100001 | 00001 | Pay out wages | AcctFrom | 300 -------- | Pre-service completes | |
| 100001 | 00002 | Pay out wages-------- | 100 | AcctTo | Pre-service completes | |
| 100001 | 00003 | Pay out wages-------- | 200 | AcctTo2 | Pre-service completes |
Process " repaying " task again, according to pretreatment parameter " by paying a square account single payment amount of money, account merges the amount of money that keeps accounts by keeping accounts ", pay a square account and be AcctFrom, debit's account is AcctTo3.Merge processing, obtain:
Finally, by result incoming task storage unit 603, complete the pre-service of a batch tasks.
| Mission Number | Detailed sequence number | Task type | Pay square account | Produce the amount of money | Proceed to account | Treatment state |
| 100001 | 00001 | Pay out wages | AcctFrom | 300 -------- | Pre-service completes | |
| 100001 | 00002 | Pay out wages-------- | 100 | AcctTo | Pre-service completes | |
| 100001 | 00003 | Pay out wages-------- | 200 | AcctTo2 | Pre-service completes | |
| 100002 | 00001 | Repay | AcctFrom | 900 -------- | Pre-service completes | |
| 100002 | 00002 | Repay-------- | 900 | AcctTo3 | Pre-service completes |
Fig. 5 is scheduling engine device 400 structural representations.With reference to Fig. 5, scheduling engine device 400 comprises: system task amount detecting unit 401, scheduling parameter are read in unit 402 and scheduling decision unit 403.Wherein system task amount detecting unit 401, scheduling parameter are read in unit 402 and are connected with scheduling decision unit 403 respectively.
System task amount detecting unit 401, for the set time interval of system parameter setting unit 101, the current concurrent processing task amount of system described in timing acquisition, exports to scheduling decision unit 403.
Scheduling parameter is read in unit 402, reads channel eigenwert and the time point eigenwert of current slot for the mission bit stream storage unit 604 from data storage device 600, exports to scheduling decision unit 403.
Formula 3:T=(M-N) * Pb * Pa
Wherein T represents that current period system is assignable, processes the processing threads number of the batch tasks that each channel uploads; M represents the maximum concurrent processing task amount of system; N represents the current concurrent processing task amount of system (being obtained by system task amount detecting unit 401); (M-N) be illustrated under the condition of the normal operation of assurance system the current task amount that can bear again of system; (M-N) although * Pb represents the current task amount that can bear again (M-N) of system, but the system of considering is also constantly receiving online and batch tasks, for reserving surplus, tackle the growth of online task, can only start wherein a part of thread and do issued transaction in batches.So multiply each other with time point eigenwert, in conjunction with in history, the batch tasks accounting of current period, increases rational Thread Count and goes to process batch tasks.(M-N) * Pb * Pa represents to distribute Thread Count in batches to multiply each other with channel eigenwert again, just can on each channel, send the accounting of task amount in conjunction with in history, obtains processing the Thread Count of the batch tasks of being uploaded by this channel.
By the current idle thread of above-mentioned formula capable of dynamic computing system, by the analysis to historic task data, calculate the preferred plan of the spendable system resource of each channel of this period.
Below by the mode of giving an example, introduce the course of work of scheduling decision unit 403: the maximum concurrent processing task amount of certain system is 80, detecting learns that current concurrent task amount is 50, at the corresponding period SD1 of current time, it is 0.4 that system is calculated time point eigenwert according to history value, for Web bank's channel, the channel eigenwert that system is calculated according to history value is 0.5, so current, system again Processing tasks amount is 80-50,30; Be multiplied by time point eigenwert 0.4, the current batch tasks Thread Count of reallocating of the system that obtains is 12, and remains 18 as reserved; For Web bank's channel, 12 are multiplied by channel eigenwert, show that the Thread Count of the batch tasks that the current You Gai of processing Web bank uploads is 6, and remaining 6 threads go to process the batch tasks that other channels are uploaded.
Fig. 6 is Task Processing Unit 500 structural representations.With reference to Fig. 6, Task Processing Unit 500 comprises: Thread control unit 501, task receiving element 502 and task processing unit 503.Wherein Thread control unit 501, task receiving element 502 are connected with task processing unit 503 respectively.
The data that parameter storage unit 601 arranges for preserving parameter setting apparatus 100: the time interval of the maximum task amount parameter of system, scanning system task amount, priority of task order parameters, task pretreatment parameter.Batch tasks storage unit 602, is submitted to the batch tasks of described system for storing other channels.Task location 603, for storing generating after pretreatment unit 300 is processed of task.Mission bit stream storage unit 604, exports the task amount of every day for preserving task data analytical equipment 200 record cells 201; And the computing unit 202 of store tasks data analysis set-up 200 is exported channel eigenwert, the time point list of feature values on the same day.Log unit 605: process daily record for storage system operation, storage pre-service, task.
Description based on above with reference to Fig. 1 to Fig. 7 lot size scheduling system, is described in detail lot size scheduling method provided by the invention below.
Fig. 9 is thread scheduling and task processing flow chart.With reference to Fig. 9, in step 901, system task amount detecting unit 401 is according to the predefined time interval (being time interval of scanning system task amount of storage in parameter storage unit 601), and the current concurrent processing task amount of system N described in timing acquisition, exports to scheduling decision unit 403.In step 902, scheduling decision unit 403 is the maximum concurrent processing task amount of 101 reading systems M from system parameter setting unit.In step 903, calculate M and whether be greater than N, if be greater than, represent that described system resource is available free, can start thread process batch tasks, setting up procedure 904, as are less than or equal to, described system resource is busy, and repeating step 901, proceeds system resource detecting.In step 904, scheduling parameter is read in channel eigenwert Pa and time point eigenwert and the Pb that unit 402 obtains the current period from mission bit stream storage unit 604, and result is exported to scheduling decision unit 403.In step 905, scheduling decision unit 403 is according to formula T=(M-N) * Pa * Pb, and the system described in the current period calculated can be distributed to the processing threads number of the batch tasks that each channel uploads, and result is exported to Thread control unit 501.In step 906, Thread control unit 501 detecting present batch task processing threads numbers, and with T comparison, if comparative result is that T is large, show to need to increase thread, start the thread that should increase, if T is little, showing needs to reduce thread, and the unnecessary thread of mark discharges thread after its current task completes.In step 907, task receiving element 502 reads priority of task order parameters from data storage device 600.In step 908, task receiving element 502 scan task storage unit 603, have judged whether waiting task, if had, enter step 909, if nothing continues scanning.In step 909, task receiving element 502, according to priority of task order parameters, according to priority sequence, reads T bar task.In step 910,503 pairs of tasks of task processing unit are processed (for example financial process).In step 911, logger task result, discharges thread, and returns to step 908.
The system and method for realizing batch data processing of the present invention is by dynamic detection mechanism, monitoring system resource at any time, when system pressure hour, distribute part thread process batch tasks, needn't wait for the batch processing play of set time, improved the ageing of batch tasks processing, and system resource is made full use of; When online concurrent task amount is high, system call is frequent, pressure is when excessive, and by dynamic detection mechanism, the concurrent Thread Count of control system, can not increase due to the frequent processing of batch tasks the burden of system thereby realize; By the customization of pretreatment parameter, in native system, in batch tasks advanced processing and preparation, reduced the complexity of task, improved processing speed.
Above-described specific embodiment; object of the present invention, technical scheme and beneficial effect are further described; institute is understood that; the foregoing is only specific embodiments of the invention; be not limited to the present invention; within the spirit and principles in the present invention all, any modification of making, be equal to replacement, improvement etc., within all should being included in protection scope of the present invention.
Claims (13)
1. a lot size scheduling system, this system comprises:
Parameter setting apparatus (100), user, by this device, systematic parameter is set and task is processed correlation parameter, and by the Parameter storage arranging in data storage device (600), the time interval that wherein said systematic parameter comprises the maximum concurrent processing task amount of system and scanning system Real-time and Concurrent task amount, described task is processed correlation parameter and is comprised: task pretreatment parameter, for the batch tasks that determines to send on different channels that described system docking receives, the pretreatment mode carrying out according to task type; Priority of task order parameters, for determining the processing sequence between different task type;
Pretreatment unit (300), for batch tasks is carried out to pre-service, and producing after processing of task is outputed in data storage device (600), it further comprises: batch tasks receiving element (301), be responsible for reading priority of task order parameters from the parameter storage unit (601) of data storage device (600), according to priority ranking from the batch tasks storage unit (602) of data storage device (600), read each channel and be submitted to the batch tasks in described system, export to task pretreatment unit (303);
Task pretreatment parameter reads in unit (302), and this unit reads the pretreatment parameter of a certain task type from data storage device (600), exports to task pretreatment unit (303);
Task pretreatment unit (303), obtain the batch tasks of batch tasks receiving element (301) input, and task pretreatment parameter reads in the pretreatment parameter of the described a certain task type of unit (302) input, to batch tasks sort, aggregation process, producing after processing of task is outputed in the task location (603) of data storage device (600);
Scheduling engine device (400), for the current concurrent processing task amount of timing detecting system, send the Thread Count of batch tasks, and result is exported to Task Processing Unit (500) on each channel of the assignable processing of computing system;
Task Processing Unit (500), for receiving each channel of the assignable processing of system transmitting from scheduling engine device (400), send the Thread Count of batch tasks, adjust internal thread, according to priority of task order parameters, read through pretreated task, carry out task processing;
Data storage device (600), the task and the task that for the batch tasks sent on the parameter receiving and stored parameter setting device (100) arranges or calculates, each channel and pretreatment unit (300), generate are processed daily record.
2. the system as claimed in claim 1, is characterized in that, this system further comprises:
Task data analytical equipment (200), for recording the information relevant to task amount, calculates channel eigenwert and time point eigenwert in a certain period, and result of calculation is stored in data storage device (600).
3. system as claimed in claim 2, it is characterized in that, described scheduling engine device (400), according to the maximum concurrent processing task amount of the system of obtaining from data storage device (600), the channel eigenwert of current period and time point eigenwert, send the Thread Count of batch tasks on each channel of the assignable processing of computing system.
4. system as claimed in claim 3, is characterized in that, described scheduling engine device (400) further comprises:
Task amount detecting unit (401), according to the predefined time interval, the current concurrent processing task amount of system N described in timing acquisition, exports to scheduling decision unit (403), and wherein N is natural number;
Scheduling decision unit (403), from the maximum concurrent processing task amount of system parameter setting unit (101) reading system M, calculate M and whether be greater than N, if be greater than, calculate the Thread Count T that send batch tasks on each channel of the assignable processing of current period system, wherein M and T are natural number.
5. system as claimed in claim 4, is characterized in that, Task Processing Unit (500) further comprises:
Thread control unit (501), to on each channel of the assignable processing of system, send the required Thread Count of the Thread Count T of batch tasks and Processing tasks amount N to compare, if comparative result is that T is large, increase thread, and start the thread that should increase, if T is little, reduce thread, and the unnecessary thread of mark, after completing, its current task discharges this thread;
Task receiving element (502) obtains priority of task order parameters, according to priority sequence, reads T bar task;
Task processing unit (503), processes described task according to reading order.
6. the system as described in claim 1～5 any one, is characterized in that, on each channel of the assignable processing of system, send the Thread Count T of batch tasks to calculate by following formula:
T=(M－N)×Pa×Pb，
Wherein M represents the maximum concurrent processing task amount of system, and N represents the current concurrent processing task amount of system,
Pa=SUM (Sa)/SUM (S), Pa represents channel eigenwert, refer to the some periods in history, certain channel is to sending in described system task amount to account for the accounting of all task amounts that described system receives, Sa represents some day, the number of tasks that in a certain period, a certain channel is submitted to, SUM (Sa) represented in 1 year, the total task number that in the described period, described channel is submitted to, S represents above-mentioned some day, whole number of tasks that in a certain period, described system is received, SUM (S) represented in 1 year, the total task number that in the described period, described system is received, Pb=SUM (Sb)/SUM (S), Pb represents time point eigenwert, refer to the some periods in history, accounting in whole tasks that the batch tasks that described system receives receives in described system, Sb represents some day, the batch tasks number that in a certain period, described system receives, SUM (Sb) represented in 1 year, the batch tasks sum that in the described period, described system receives, S represents described some day, whole number of tasks that in a certain period, described system is received, SUM (S) represented in 1 year, the total task number that in the described period, described system is received.
7. system as claimed in claim 6, is characterized in that, parameter setting apparatus (100) further comprises:
System parameter setting unit (101), user arranges the maximum concurrent processing task amount of described system M by this element, and the time interval of scanning system Real-time and Concurrent task amount is set, and outputs in data storage device (600);
Priority of task order parameters setting unit (102), user arranges the processing priority order of different task type by this element, output in data storage device (600);
Task pretreatment parameter setting unit (103), user arranges the required pre-service done when receiving a batch tasks of described system by this element and works, and outputs in data storage device (600).
8. a lot size scheduling method, the method comprises:
User arranges systematic parameter and task processing parameter by parameter setting apparatus (100), and by the Parameter storage arranging in data storage device (600), the time interval that wherein said systematic parameter comprises the maximum concurrent processing task amount of system and scanning system Real-time and Concurrent task amount, described task processing parameter comprises: task pretreatment parameter, for the batch tasks that determines to send on different channels that described system docking receives, the pretreatment mode carrying out according to task type; Priority of task order parameters, for determining the processing sequence between different task type;
By pretreatment unit (300), batch tasks is carried out to pre-service, and producing after processing of task is outputed in data storage device (600), wherein pretreatment unit (300) further comprises: batch tasks receiving element (301), be responsible for reading priority of task order parameters from the parameter storage unit (601) of data storage device (600), according to priority ranking from the batch tasks storage unit (602) of data storage device (600), read each channel and be submitted to the batch tasks in described system, export to task pretreatment unit (303); Task pretreatment parameter reads in unit (302), and this unit reads the pretreatment parameter of a certain task type from data storage device (600), exports to task pretreatment unit (303); Task pretreatment unit (303), obtain the batch tasks of batch tasks receiving element (301) input, and task pretreatment parameter reads in the pretreatment parameter of the described a certain task type of unit (302) input, to batch tasks sort, aggregation process, producing after processing of task is outputed in the task location (603) of data storage device (600);
Utilize the regularly current concurrent processing task amount of detecting system of scheduling engine device (400), on each channel of the assignable processing of computing system, send the Thread Count of batch tasks, and result is exported to Task Processing Unit (500);
By Task Processing Unit (500), receive the Thread Count that each channel of the assignable processing of system transmitting from scheduling engine device (400) send batch tasks, adjust internal thread, according to priority of task order parameters, read through pretreated task, carry out task processing;
Task and task that the batch tasks of sending on the parameter that is arranged or calculated by data storage device (600) reception stored parameter setting device (100), each channel and pretreatment unit (300) generate are processed daily record.
9. method as claimed in claim 8, is characterized in that, the method further comprises:
By task data analytical equipment (200) the record information relevant to task amount, calculate channel eigenwert and time point eigenwert in a certain period, and result of calculation is stored in data storage device (600).
10. method as claimed in claim 9, it is characterized in that, described scheduling engine device (400), according to the maximum concurrent processing task amount of the system of obtaining from data storage device (600), the channel eigenwert of current period and time point eigenwert, send the Thread Count of batch tasks on each channel of the assignable processing of computing system.
11. methods as claimed in claim 10, is characterized in that, described scheduling engine device (400) further comprises:
Task amount detecting unit (401), according to the predefined time interval, the current concurrent processing task amount of system N described in timing acquisition, exports to scheduling decision unit (403), and wherein N is natural number;
Scheduling parameter is read in unit (402), reads channel eigenwert and the time point eigenwert of current slot for the mission bit stream storage unit (604) from data storage device (600), exports to scheduling decision unit (403);
Scheduling decision unit (403), from the maximum concurrent processing task amount of system parameter setting unit (101) reading system M, calculate M and whether be greater than N, if be greater than, calculate the Thread Count T that send batch tasks on each channel of the assignable processing of current period system, wherein M and T are natural number.
12. methods as claimed in claim 11, is characterized in that, described Task Processing Unit (500) further comprises:
Thread control unit (501), to on each channel of the assignable processing of system, send the required Thread Count of the Thread Count T of batch tasks and Processing tasks amount N to compare, if comparative result is that T is large, increase thread, and start the thread that should increase, if T is little, reduce thread, and the unnecessary thread of mark, after completing, its current task discharges this thread;
Task receiving element (502) obtains priority of task order parameters, according to priority sequence, reads T bar task;
Task processing unit (503), processes described task according to reading order.
13. methods as described in claim 8～12 any one, is characterized in that, on each channel of the assignable processing of system, send the Thread Count T of batch tasks to calculate by following formula:
T=(M－N)×Pa×Pb，
Wherein M represents the maximum concurrent processing task amount of system, and N represents the current concurrent processing task amount of system,
Pa=SUM (Sa)/SUM (S), Pa represents channel eigenwert, refer to the some periods in history, certain channel is to sending in described system task amount to account for the accounting of all task amounts that described system receives, Sa represents some day, the number of tasks that in a certain period, a certain channel is submitted to, SUM (Sa) represented in 1 year, the total task number that in the described period, described channel is submitted to, S represents some day, whole number of tasks that in a certain period, described system is received, and SUM (S) represented in 1 year, the total task number that in the described period, described system is received
Pb=SUM (Sb)/SUM (S), Pb represents time point eigenwert, refer to the some periods in history, accounting in whole tasks that the batch tasks that described system receives receives in described system, Sb represents some day, the batch tasks number that in a certain period, described system receives, SUM (Sb) represented in 1 year, the batch tasks sum that in the described period, described system receives, S represents described some day, whole number of tasks that in a certain period, described system is received, SUM (S) represented in 1 year, the total task number that in the described period, described system is received.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110391457.6A CN102393822B (en) | 2011-11-30 | 2011-11-30 | Batch scheduling system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110391457.6A CN102393822B (en) | 2011-11-30 | 2011-11-30 | Batch scheduling system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102393822A CN102393822A (en) | 2012-03-28 |
| CN102393822B true CN102393822B (en) | 2014-03-12 |
Family
ID=45861151
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110391457.6A Active CN102393822B (en) | 2011-11-30 | 2011-11-30 | Batch scheduling system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102393822B (en) |
Families Citing this family (26)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103197970A (en) * | 2013-04-12 | 2013-07-10 | 中国银行股份有限公司 | Method for triggering jobs with multiple mechanisms running and bank data processing system |
| CN103197960B (en) * | 2013-04-12 | 2016-06-22 | 中国银行股份有限公司 | Dispatching method and system for batch job system |
| CN103645944B (en) * | 2013-12-25 | 2017-01-18 | 中国工商银行股份有限公司 | Batch data conflict detection method, device and system |
| CN105335231B (en) * | 2014-08-15 | 2020-01-31 | 阿里巴巴集团控股有限公司 | server thread dynamic allocation method and device |
| CN105718243B (en) * | 2014-12-01 | 2018-12-21 | 金蝶软件（中国）有限公司 | The method and system that batch data operates in ERP system |
| CN105389208B (en) * | 2015-11-10 | 2018-12-14 | 中国建设银行股份有限公司 | Job processing method and device |
| CN105389209B (en) * | 2015-12-25 | 2019-04-26 | 中国建设银行股份有限公司 | A kind of asynchronous batch tasks processing method and system |
| CN105912401B (en) * | 2016-04-08 | 2019-03-12 | 中国银行股份有限公司 | A kind of distributed data batch processing system and method |
| CN105930216A (en) * | 2016-04-15 | 2016-09-07 | 平安科技（深圳）有限公司 | Automatic scheduling method and system for electronic signature system and server |
| CN107491346B (en) * | 2016-06-12 | 2021-03-12 | 阿里巴巴集团控股有限公司 | Application task processing method, device and system |
| CN106204010A (en) * | 2016-07-26 | 2016-12-07 | 通联支付网络服务股份有限公司 | A kind of channel access system |
| CN108197912A (en) * | 2016-12-08 | 2018-06-22 | 阿里巴巴集团控股有限公司 | A kind of transaction processing system and method |
| CN108270603A (en) * | 2016-12-31 | 2018-07-10 | 中国移动通信集团陕西有限公司 | A kind of dispatching method and management system |
| CN107016536B (en) * | 2017-01-16 | 2018-06-22 | 平安银行股份有限公司 | The method and trading server of trading processing |
| CN107977206B (en) * | 2017-11-14 | 2020-11-27 | 北京思特奇信息技术股份有限公司 | Method and device for processing batch service data files |
| CN107943584A (en) * | 2017-11-15 | 2018-04-20 | 中国银行股份有限公司 | The processing method and processing device of batch transaction request |
| CN108829506A (en) * | 2018-07-04 | 2018-11-16 | 中国建设银行股份有限公司 | Batch tasks processing method, device and service system |
| CN109271252B (en) * | 2018-08-29 | 2021-07-16 | 华为技术有限公司 | Method and device for adjusting thread number |
| CN109492024A (en) * | 2018-10-26 | 2019-03-19 | 平安科技（深圳）有限公司 | Data processing method, device, computer equipment and storage medium |
| CN110018906B (en) * | 2019-02-26 | 2020-11-27 | 上海丽人丽妆网络科技有限公司 | Scheduling method, server and scheduling system |
| CN110297711B (en) * | 2019-05-16 | 2024-01-19 | 平安科技（深圳）有限公司 | Batch data processing method, device, computer equipment and storage medium |
| CN110362452B (en) * | 2019-07-18 | 2022-07-22 | 中国银行股份有限公司 | Batch operation and maintenance method and system |
| CN112732417B (en) * | 2019-10-28 | 2024-09-20 | 北京京东振世信息技术有限公司 | Method and device for processing application request |
| CN111104218B (en) * | 2019-11-29 | 2022-04-22 | 北京浪潮数据技术有限公司 | Storage system data synchronization method, device, equipment and readable storage medium |
| CN111563084A (en) * | 2020-05-06 | 2020-08-21 | 中国银行股份有限公司 | Batch fee deduction data processing method and device |
| CN113190418A (en) * | 2021-07-01 | 2021-07-30 | 奇安信科技集团股份有限公司 | Log receiving method and device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1577278A (en) * | 2003-07-22 | 2005-02-09 | 株式会社东芝 | Method and system for scheduling real-time periodic tasks |
| CN1787000A (en) * | 2005-11-15 | 2006-06-14 | 中国工商银行股份有限公司 | System and method for centralized managing and dispatching of batch work |
| CN101001174A (en) * | 2006-12-31 | 2007-07-18 | 华为技术有限公司 | Method and system for implementing integral dispatching in network element management system |
-
2011
- 2011-11-30 CN CN201110391457.6A patent/CN102393822B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1577278A (en) * | 2003-07-22 | 2005-02-09 | 株式会社东芝 | Method and system for scheduling real-time periodic tasks |
| CN1787000A (en) * | 2005-11-15 | 2006-06-14 | 中国工商银行股份有限公司 | System and method for centralized managing and dispatching of batch work |
| CN101001174A (en) * | 2006-12-31 | 2007-07-18 | 华为技术有限公司 | Method and system for implementing integral dispatching in network element management system |
Non-Patent Citations (2)
| Title |
|---|
| 一种多用户MapReduce集群的作业调度算法的设计与实现;王凯,吴泉源,杨树强;《计算机与现代化》;20101030(第10期);23-27页 * |
| 王凯,吴泉源,杨树强.一种多用户MapReduce集群的作业调度算法的设计与实现.《计算机与现代化》.2010,(第10期),第23-28页. |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102393822A (en) | 2012-03-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102393822B (en) | Batch scheduling system and method | |
| US7873563B2 (en) | Systems and methods for automatically backing out buy orders | |
| US8996612B1 (en) | System and method for transferring data between a user space and a kernel space in a server associated with a distributed network environment | |
| US20080162326A1 (en) | Systems and Methods for Automatically Backing Out Buy Orders | |
| US20170293980A1 (en) | System and method for managing processing resources of a computing system | |
| US8838495B2 (en) | Method and system for monitoring market data to identify user defined market conditions | |
| CN111881221B (en) | Method, device and equipment for customer portrayal in logistics service | |
| CN102609875A (en) | Method and system for concurrently and dynamically processing bank data | |
| Borgs et al. | The optimal admission threshold in observable queues with state dependent pricing | |
| CN106327053B (en) | Construction method of weaving process recommendation model based on multi-mode set | |
| CN202584217U (en) | A parallel batch processing system of bank data | |
| CN101246578A (en) | Three-party deposit processing system and method | |
| CN113723881A (en) | Freight note separate-moistening settlement method, device, equipment and storage medium | |
| EP2174283A1 (en) | System, method and program for agency cost estimation | |
| CN116452351A (en) | Method and device for processing fund data, computer equipment and storage medium | |
| JP2004164030A (en) | Point management system and management method for financial institution dealings | |
| CN201514656U (en) | Third-party depository management system | |
| US20210304228A1 (en) | Method of information processing, electronic device, and computer program product | |
| CN110599111A (en) | Labor reward dispensing method and labor reward dispensing system | |
| CN110264334A (en) | Odd corpus distribution method and system | |
| KR100650348B1 (en) | Optimization for Distribution of Sending in Cash of ATM | |
| CN117172842B (en) | Lottery configuration method, lottery configuration device, lottery configuration equipment and storage medium | |
| CN113469686B (en) | Method, device, equipment and storage medium for adjusting business transaction amount | |
| RU2599951C2 (en) | System for organizing electronic trade process using financial instruments | |
| Konana et al. | Pricing of Information Services Using Real-Time Databases: A Framework for Integrating User Preferences and Real-Time Workload (Best Paper Runner Up) |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |