CN110457159A

CN110457159A - A kind of method, apparatus, calculating equipment and the storage medium of processing batch tasks

Info

Publication number: CN110457159A
Application number: CN201910775666.7A
Authority: CN
Inventors: 王磊; 江旻; 李斌; 黄俏龙; 席俊杰
Original assignee: WeBank Co Ltd
Current assignee: WeBank Co Ltd
Priority date: 2019-08-21
Filing date: 2019-08-21
Publication date: 2019-11-15
Also published as: WO2021032056A1

Abstract

This application discloses a kind of method, apparatus for handling batch tasks, equipment and storage medium are calculated, belongs to financial technology technical field, for carrying out effective monitoring to the abnormal of batch tasks.The described method includes: determining the actual amount of data of batch tasks when batch tasks meet batch processing trigger condition；According to the corresponding history batch execution data of batch tasks, the data volume term of reference that batch tasks are carried out with batch processing is determined；If actual amount of data not within data volume term of reference, blocks this batch processing for batch tasks；If actual amount of data carries out batch processing to target batch task within data volume term of reference, according to actual amount of data.The program can promote the accuracy of batch forecast based on history batch execution data, and then make accurate batch processing decision.

Description

A kind of method, apparatus, calculating equipment and the storage medium of processing batch tasks

Technical field

This application involves the field of computer technology more particularly to a kind of processing batch tasks of financial (Fintech) science and technology Method, apparatus, calculate equipment and storage medium.

Background technique

With the development of computer technology, more and more technical applications are in financial field, and traditional financial industry is gradually Change to financial technology (Finteh), batch processing technology is no exception, but since the safety of financial industry, real-time are wanted It asks, the higher requirement that also batch processing technology is proposed.As the quantity of financial industry batch tasks is more and more, batch is The data volume handled of uniting is increasing, and it is also increasingly wider to influence face.If exception occurs in batch system, number on a large scale will occur According to mistake, and these wrong data are all irreversible mostly.

So the abnormal progress effective monitoring for batch tasks is current problem to be solved.

Summary of the invention

The embodiment of the present application provides a kind of method, apparatus for handling batch tasks, calculates equipment and storage medium, for pair The abnormal of batch tasks carries out effective monitoring.

In a first aspect, providing a kind of method for handling batch tasks, which comprises

Determine whether batch tasks meet preset batch processing trigger condition；

When meeting the batch processing trigger condition, the actual amount of data of the batch tasks is determined；

According to the corresponding history batch execution data of the batch tasks, determines and batch processing is carried out to the batch tasks Data volume term of reference；

If the actual amount of data not within the data volume term of reference, blocks the sheet for the batch tasks Secondary batch processing；

If the actual amount of data is within the data volume term of reference, according to the actual amount of data to the mesh It marks batch tasks and carries out batch processing.

In a kind of possible design, according to the corresponding history batch data of the batch tasks, determine to the batch The data volume term of reference of task progress batch processing, comprising:

Parse the pending data type in the batch tasks；

In the history batch data, the corresponding fixed reference feature object of the pending data type is determined, and transfer The corresponding target data values of each fixed reference feature object, the target data values exist for characterizing the fixed reference feature object Corresponding data value in preset duration；

According to the batch tasks corresponding trained in advance batch tasks prediction model and each fixed reference feature pair As corresponding target data values, the corresponding reference data amount of each fixed reference feature object is determined, and according to each ginseng The corresponding reference data amount of feature object is examined, determines the data volume reference threshold of the batch tasks；Wherein, the batch tasks Prediction model is obtained according to the corresponding data value training of the fixed reference feature object in the history batch data；

According to the data volume reference threshold of the batch tasks, the data volume term of reference is determined.

In a kind of possible design, training obtains the batch tasks prediction model in the following way:

From all feature objects that the history batch data includes, determined according to default selection strategy described with reference to special Levy object；Wherein, each feature object data volume corresponding with pending data type progress batch processing has association Relationship；

Multiple historical time sections are determined from the history batch data, and are extracted respectively each in each historical time section The corresponding data value of a fixed reference feature object；

According to the corresponding data value of each fixed reference feature object in each historical time section, initial batch is appointed Business prediction model is trained, with the batch tasks prediction model after being trained.

In a kind of possible design, from all feature objects that the history batch data includes, according to default choosing It selects strategy and determines the fixed reference feature object, comprising:

Determine the phase between each feature object data volume corresponding with pending data type progress batch processing Guan Du；

The feature object that the degree of correlation meets default screening conditions is determined as the fixed reference feature object.

In a kind of possible design, the feature object that the degree of correlation meets default screening conditions is determined as described with reference to special Levy object, comprising:

The feature object that the degree of correlation is greater than predetermined relevance threshold is determined as the fixed reference feature object；Alternatively,

According to the sequence that the degree of correlation is descending, the feature object for being located at the predetermined quantity of front is determined as the reference Feature object.

From all feature objects, select predetermined characteristic object as the fixed reference feature object.

In a kind of possible design, according to the corresponding reference data amount of each fixed reference feature object, comprising:

Determine object increment of each fixed reference feature object in the first scheduled duration；

According to the batch tasks prediction model and each fixed reference feature object in first scheduled duration Object increment determines the corresponding data increment of each fixed reference feature object；

According to data volume of each fixed reference feature object in the second scheduled duration and in first scheduled duration Interior data increment determines the corresponding reference data amount of each fixed reference feature object.

In a kind of possible design, according to the corresponding reference data amount of each fixed reference feature object, determine described in The data volume reference threshold of batch tasks, comprising:

Determine balanced growth amplitude of the data volume of the batch tasks in third scheduled duration；

According to the balanced growth amplitude and the corresponding reference data amount of each fixed reference feature object, described batch is determined The data volume reference threshold of amount task.

In a kind of possible design, the actual amount of data and the data volume term of reference include that the batch is appointed The corresponding total amount of the task of the quantity of the batch processing of business and all quantity.

Second aspect, provides a kind of device for handling batch tasks, and described device includes:

First determining module, for determining whether batch tasks meet preset batch processing trigger condition；

Second determining module, for determining the reality of the batch tasks when meeting the batch processing trigger condition Data volume；

Third determining module determines the batch tasks for the history batch execution data according to the batch tasks Carry out the data volume term of reference of batch processing；

Batch blocks module, if for the actual amount of data not within the data volume term of reference, blocking pair In this batch processing of the batch tasks；

Module is executed in batches, if for the actual amount of data within the data volume term of reference, according to Actual amount of data carries out batch processing to the target batch task.

In a kind of possible design, the third determining module is used for:

Parse the pending data type in the batch tasks；

In a kind of possible design, described device further includes model training module, is used for:

In a kind of possible design, the model training module is used for:

Amplitude and the corresponding reference data amount of each fixed reference feature object are serviced according to the balanced growth, determines institute State the data volume reference threshold of batch tasks.

The third aspect, provide it is a kind of handle batch tasks device, including at least one processor and at least one deposit Reservoir, wherein the memory is stored with computer program, when described program is executed by the processor, so that the place Reason device executes the step of method of any processing batch tasks in above-mentioned first aspect.

Fourth aspect provides a kind of storage medium, and the storage medium is stored with computer instruction, when the computer refers to When order is run on computers, so that computer executes the method for any processing batch tasks in above-mentioned first aspect Step.

In the embodiment of the present application, when satisfaction needs to carry out the batch processing trigger condition of batch processing to batch tasks, It can determine the actual amount of data of the batch tasks, and can be determined according to the history batch execution data of the batch tasks The data volume term of reference of this batch processing, and then criticized compared with data volume term of reference by actual amount of data Decision is measured, specifically, when actual amount of data is within data volume term of reference, then it is assumed that this batch tasks and usual Batch tasks processing is similar, it is believed that and no exceptions, so can directly be criticized at this time according to actual amount of data Amount processing then may be used to ensure the timeliness of batch tasks processing, and when actual amount of data is not within data volume term of reference To think that this batch tasks is not consistent with history disposition, at this time it may be considered that batch process be likely to occur it is different Often, then can then block the processing of batch tasks at this time.It, can be in this way, in conjunction with the history batch execution data of batch tasks The accuracy of batch decision is promoted, real time monitoring batch is run, if it find that abnormal block batch in time, guarantees the correct of data Property, so as to avoid the abnormal bring loss of batch, enhance the usage experience of user.

It should be understood that above general description and following detailed description be only it is exemplary and explanatory, not The disclosure can be limited.

Detailed description of the invention

In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this Inventive embodiments for those of ordinary skill in the art without creative efforts, can also be according to mentioning The attached drawing of confession obtains other attached drawings.

Fig. 1 is the process schematic of the method for the processing batch tasks in the embodiment of the present application；

Fig. 2 is the flow chart of the method for the processing batch tasks in the embodiment of the present application；

Fig. 3 is the flow chart that data volume term of reference is determined using batch tasks prediction model in the embodiment of the present application；

The structural block diagram of the device of processing batch tasks in Fig. 4 the embodiment of the present application；

Fig. 5 is the structural schematic diagram of the calculating equipment in the embodiment of the present application；

Fig. 6 is another structural schematic diagram of the calculating equipment in the embodiment of the present application.

Specific embodiment

To make the objectives, technical solutions, and advantages of the present invention clearer, below in conjunction in the embodiment of the present invention Attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is only It is a part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, ordinary skill people Member's every other embodiment obtained without making creative work, shall fall within the protection scope of the present invention.In In the case where not conflicting, the feature in embodiment and embodiment in the present invention can mutual any combination.Although also, flowing Logical order is shown in journey figure, but in some cases, it can be to be different from shown or described by sequence execution herein The step of.

Term " first " and " second " in description and claims of this specification and above-mentioned attached drawing are for distinguishing Different objects, not for description particular order.In addition, term " includes " and their any deformations, it is intended that covering is not Exclusive protection.Such as it contains the process, method, system, product or equipment of a series of steps or units and is not limited to The step of listing or unit, but optionally further comprising the step of not listing or unit, or optionally further comprising for these The intrinsic other step or units of process, method, product or equipment.

In the embodiment of the present invention, " multiple " can indicate at least two, for example, can be two, three or more, The embodiment of the present invention is with no restrictions.

In addition, the terms "and/or", only a kind of incidence relation for describing affiliated partner, indicates may exist Three kinds of relationships, for example, A and/or B, can indicate: individualism A exists simultaneously A and B, these three situations of individualism B.Separately Outside, character "/" herein typicallys represent the relationship that forward-backward correlation object is a kind of "or" in the case where not illustrating.

The design philosophy of the application introduced below.

It as previously described, is urgently to be resolved at present for the abnormal progress effective monitoring of the various batch tasks of financial industry The technical issues of.In order to capture the exception occurred in batch processing, batch monitoring system is essential.With bank machine For structure, it is all lag mostly that traditional bank, which runs and criticizes system exception monitoring, that is, is usually abnormal caused in appearance batch The abnormal problem of batch could be found after bad result, in order to solve the problems, such as monitoring lag, many bank's batch systems are also adopted Simple batch blocking mechanism is taken.For example, by using the mode of setting threshold value, the same day pre- place is run out of by program before data processing The data volume of reason, if it find that preprocessed data amount is more than that the threshold value being arranged in advance then blocks batch.The side of this set threshold value Although formula can also guarantee the correct operation of batch by look-ahead, it is a difficult point that threshold value, which how is accurately arranged, and It is at present usually that a threshold value is empirically roughly arranged in staff, due to the limited experience of different staff, and And the cognitive Bias that different staff is subjective there is likely to be some individuals, if threshold value setting is too high, possible pocket is not Firmly lead to error in data extremely, if setting is too low, and will lead to abnormal wrong report, batch just finds not different after blocking Often, batch operational efficiency is influenced.

In consideration of it, the application implements to provide a kind of method for handling batch tasks, can use batch by this method is System effectively monitors the abnormality processing of batch tasks.As shown in Figure 1, it can first be based on BDP (Beagledata Platform, a enterprise-level big data middleware platform based on Hadoop ecosystem) to the historical data of batch tasks It is processed, can be used as the prediction data of same day batch processing.Further, operation system generates batch withholding and easily ties Fruit carries out the actual amount of data of this batch processing, and judgement is compared with prediction data for obtained actual amount of data, To obtain final batch processing decision.That is, the embodiment of the present application can be by history when carrying out this batch processing Including the case where batch processing fully considers, i.e., based on historical data, data are excavated from the mass data of history Feature, by carrying out analysis to these data characteristicses to export batch decision.By combining history batch execution data to carry out The mode of this batch forecast can be promoted as best one can so as best one can using the batch processing situation of history as referring to foundation The accuracy of batch forecast, real time monitoring batch is run, if it find that abnormal block batch in time, guarantees batch data processing Correctness and timeliness, so as to avoid the abnormal bring loss of batch.

To further illustrate technical solution provided by the embodiments of the present application, with reference to the accompanying drawing and specific embodiment pair This is described in detail.Although the embodiment of the present application provides as the following examples or method operating procedure shown in the drawings, It but based on routine or in the method may include more or less operating procedure without creative labor.It is patrolling It collected in upper the step of there is no necessary causalities, the execution sequence of these steps is not limited to execution provided by the embodiments of the present application Sequentially.The method can be according to embodiment or method shown in the drawings when perhaps device executes in actual treatment process Sequence executes or parallel execution.

Based on above content, the embodiment of the present application provides a kind of method for handling batch tasks, and this method can be deployed in Such as bank, lending platforms etc. need to carry out in the system of batch tasks processing.It is shown in Figure 2, in the embodiment of the present application The processes of method of processing batch tasks be described as follows.

Step 201: judging whether the batch processing trigger condition for meeting batch tasks.

It as previously described, include a plurality of types of batch tasks in financial industry, such as wage batch transfers accounts, provides a loan and criticize The business such as deduction are measured, the batch tasks in the embodiment of the present application can be the batch tasks of any possible type.For difference The batch tasks of type, bank can carry out batch processing in different time nodes, such as online transaction business processing is usually wanted Banking system is asked to be enable to respond quickly and return in real time, so in order to avoid being impacted to online business, batch Task can be carried out in the online business probability of happening lower period, such as be executed in batches at night or morning, so In a kind of possible embodiment, segmentum intercalaris when batch processing trigger condition can be the processing for reaching preset batch tasks Point in other embodiments, such as can also be triggered manually the batch processing for carrying out batch tasks by bank clerk, Be processed in batches trigger condition can also be that batch processing system receives the processing request, etc. for batch tasks.

When determining that batch tasks meet batch processing trigger condition, that is, demonstrate the need for carrying out at batch the batch tasks Reason, can further execute step 202, however, it is determined that be unsatisfactory for batch processing trigger condition, then show temporarily to be further without at this time Batch processing is carried out to the batch tasks, can further continue whether meet sentencing for batch processing trigger condition It is disconnected.

Step 202: determining the actual amount of data of batch tasks.

By taking the batch in credit operation withholds business as an example, when needing to carry out batch to withhold, at the batch of financial institution Reason system can go out every reimbursement business according to element factor calculations such as the borrowing balances, refund issue, interest-bearing rule of each user This repayment amount, and then all users for needing batch to withhold according to this, calculate this pen of always withholing withholdd in batches Several and total amount of withholing, and calculated this always withhold stroke count and total amount of withholing for withholing in batches for example can be understood as For the actual amount of data for this batch tasks of business of withholing in batches in the embodiment of the present application, that is to say, that determining needs pair When batch tasks carry out batch processing, it can first calculate and batch processing actual needs processing locally is carried out to the batch tasks Data volume, the actual amount of data in the embodiment of the present application are batch processing system according to the calculated reality batch of existing system rule Amount processing foundation.

Step 203: according to the history batch execution data of batch tasks, determining the number that batch tasks are carried out with batch processing According to amount term of reference.

For batch processing system during carrying out batch tasks processing, the calculation amount being related to is larger and time-consuming general It is longer, there is mistake if may cause the related data that calculated batch is withholdd if period occurs if exception, such as count More than user, this should go back the amount of money to the deducted amount of calculating, in this case would potentially result in customer complaint or calculated button Less than user, this should go back the amount of money to the money amount of money, in this case then may cause bank and occur losing, etc..Also, due to being batch Amount processing, if if mistake occurs in the calculating for a user, then for other each users in this batch processing Identical problem may also generally occur, when the stroke count of withholing of batch processing is more, then the error occurred is also bigger.

In consideration of it, also ensuring at batch tasks as far as possible to be monitored to the exception that batch processing system is likely to occur The correctness and validity of reason are come pair using using history batch execution data as reference frame in the embodiment of the present application The processing of this batch tasks carries out the mode of control treatment, in this way, by the big data of history as reference, it can be in certain journey Show the disposed of in its entirety situation and variation tendency of the batch tasks of recent a period of time on degree, so as to relatively accurately to this The processing of secondary batch tasks is predicted, is resistance in order to which batch processing system finally makes accurate batch processing decision Disconnected batch processing still executes batch processing, to improve the validity of batch processing.

For this purpose, same type of batch tasks can be directed in the embodiment of the present application, with the history batch of the batch tasks Data are handled, determine the data volume term of reference that the batch tasks are carried out with batch processing, and then model is referred to the data volume Enclose as foundation is compared, to judge whether this batch processing is abnormal.For example, can be obtained most by taking business of withholing in batches as an example The history batch execution data of nearly one month all loan users, or can obtain nearest 100 time points of withholing and include All loan users history batch execution data, and then predict that according to these data, this carries out the data of batch processing Term of reference is measured, because in general the batch tasks correctly executed that succeeded before are exactly normal in batch processing system In the case of execute, so by the processing data of the batch tasks correctly executed that largely succeeded come to batch next time Amount processing, which carries out prediction, has certain guidance meaning, and in other words, data volume term of reference in the embodiment of the present application can be with Regard substantially data volume range when batch processing system is normally carried out batch processing as.

In the specific implementation process, the execution sequence of step 202 and step 203 can be arbitrary, such as can first be held Row step 202 executes step 203 again, can perhaps first carry out that step 203 executes step 202 again or two steps can be same Shi Zhihang, the embodiment of the present application is with no restrictions.

Step 204: actual amount of data being compared with data volume term of reference, to judge actual amount of data whether in number Within amount term of reference.

It, then can be using the data volume term of reference as foundation is compared, to sentence after obtaining data volume term of reference Whether disconnected actual amount of data is within normal range (NR), to be carried out by batch processing of the batch processing historical data to the later period pre- It surveys and instructs.

In the embodiment of the present application, actual amount of data and data volume term of reference may each comprise the business batch of batch tasks The corresponding total amount of the business of the quantity of processing and all quantity continues by taking business of withholing in batches as an example, then real data Amount and data volume term of reference may each comprise the business of withholing of withhold stroke count and all stroke counts that the business of withholing needs to be implemented Corresponding total deducted amount, in this way, can generally reflect whether every business of withholing goes wrong by way of total amount.Cause For in general, batch processing system is the same for the calculation processing mode of every business of withholing, if wherein one withhold If business calculates mistake, then this other business of withholing withholdd in batches also just will appear similar error, and one The error of withholing of a user may be less obvious, such as the interest of some user is calculated 5 yuan more, and if at this batch The stroke count of withholing of reason has 1000, then total error of this 1000 business of withholing then may be thousands of or even up to ten thousand, so Calculating error more obvious can be effectively detected by way of total amount, and then determines system exception, to promote exception The validity of monitoring.

Step 205: when actual amount of data is within data volume term of reference, then according to actual amount of data, batch being appointed Business carries out batch processing.

If the actual amount of data of batch tasks within data volume term of reference, illustrates according to existing batch processing system Calculated batch execution data of uniting is to may indicate that normally, i.e., not batch processing system is within normal range (NR) with this It is abnormal, so at this time batch processing can be carried out to batch tasks with the actual amount of data determined, such as deducts simultaneously The summation of the current period repayment amount of 300 loan users.

Step 206: when actual amount of data is not within data volume term of reference, then block for batch tasks this Batch processing.

If actual amount of data is not within data volume term of reference, then showing this batch processing and previous history For treatment process compared to there are larger differences, this is likely to be the mutation that a large amount of larger traffic has occurred in the business of withholing itself, this It generally may less be consistent with normal smooth business feature, illustrate that batch processing system may largely go out Exception is showed, then can then block this batch tasks in time at this time in order to ensure the accuracy that batch tasks execute It executes in batches.After blocking batch tasks, blocking warning information can be further exported, to carry out business to staff It is abnormal and different in appearance can to confirm whether batch system occurs really convenient for staff in time for the effective alarm blocked System maintenance and reparation can be carried out in time when often, to eliminate exception as early as possible, to restore the regular traffic energy of batch system Power carries out secondary batch processing calculating, it is also possible to be calculated again handled in time batch tasks.

In the embodiment of the present application, using history batch execution data as reference, it can reflect this batch to a certain extent There is abnormal possibility in amount task, and then the exception that batch processing system can occur is effectively predicted, to ensure batch Task accurate and effective can execute.

In the specific implementation process, true according to the history batch execution data of batch tasks in above-mentioned steps 203 Determine the mode of data volume term of reference, the embodiment of the present application provides following two embodiment.

The first method of determination

It is by machine learning techniques to the corresponding history batch execution data of batch tasks in the first method of determination It is analyzed, batch tasks prediction model is trained by the quantitative analysis of mass data, and then appoint by trained batch Business prediction model is predicted come the data volume term of reference of each batch process to batch tasks.Below to based on batch The process that amount task prediction model is predicted is illustrated.

The training process of training batch tasks prediction model is first introduced based on history batch execution data below.

It may include Feature Selection, model selection, with several portions of feature training pattern of selection during model training Point, detailed description below.

1) Feature Selection.Feature selecting is vital for building for machine learning model.Good feature can mention The performance of rising mould type, the characteristics of capable of more helping us understand data and fabric, this to it is further improve model, algorithm has Important function.However using too many variable to may result in model as model training feature becomes inaccurately, especially When in the presence of not influencing on output result or have the model training feature of larger impact to other variables.With business of withholing in batches For, on the independent variable feature of the relevant influence of self-clinching data for example including account number (quantity of all loan accounts), receipt Number (stroke counts of all loans), overdue receipt number (there are the loan stroke counts of overdue refund), loan types, installment reimbursement mode, How user credit grade, installment reimbursement issue, loan balance, etc. choose some spies in so many independent variable feature Fixed feature is as model training feature in order to avoid shadow of the variable quantity too much to the accuracy of model training as far as possible It rings, the feature for model training is selected as screening foundation using the degree of correlation in the embodiment of the present application, such as can be according to pre- If selection strategy selects references object feature, and the default selection strategy is using the degree of correlation as foundation.

In a kind of possible embodiment, the pending data type in batch tasks can be first parsed, with batch For business of withholing, pending data type for example can be understood as self-clinching data, and carry out batch to data type to be processed The total amount that automatic batch is withholdd can be not understood as by handling corresponding data volume, then determination has with pending data type All feature objects of incidence relation (such as positive influences or negative effect), this feature object be it is mentioned above for example The independents variable features such as account number, receipt number.And then it is directed to each feature object again, calculating each feature object (such as can see Work is independent variable) and pending data type carry out batch processing data volume (such as can be regarded as dependent variable) between phase Guan Du, such as the degree of correlation between each independent variable and dependent variable can be calculated using Pearson correlation coefficients, or can also To determine the degree of correlation between each independent variable and dependent variable using other methods for calculating the degree of correlation.Obtaining each feature After the corresponding degree of correlation of object, it can choose with the biggish feature object of the degree of correlation as the input eventually for model training Feature, such as the final feature object as model training can be referred to as fixed reference feature object.Obtaining each feature object After the degree of correlation between (i.e. independent variable feature) and dependent variable, the degree of correlation can be selected to meet from all feature objects default The feature object of screening conditions is as the fixed reference feature object for carrying out model training.For example, the degree of correlation can be greater than or Equal to predetermined relevance threshold feature object as fixed reference feature object, for the fixed reference feature object that ensures to select with because becoming There is strong correlation, which can be set somewhat larger, such as be set as 80% between amount；In another example Sequence that can be descending according to the degree of correlation, is determined as fixed reference feature pair for the feature object for being arranged in the predetermined quantity of front As, it can the quantity of fixed reference feature object, such as 4 are first set, and then can choose maximum 4 feature objects of the degree of correlation As final fixed reference feature object.

In alternatively possible embodiment, predetermined characteristic object rule of thumb can be set as joining by user in advance Feature object is examined, for example, account number, receipt number, overdue receipt number these three predetermined characteristic objects are to self-clinching known to rule of thumb The influence of data is strong correlation, it is possible to directly select these three predetermined characteristic objects as most from all feature objects Whole fixed reference feature object.

Because the degree of correlation is higher, illustrate that the positive correlation between corresponding independent variable feature and dependent variable is stronger, then then Illustrate that the influence of the independent variable feature to dependent variable is also larger, so carrying out using the high some fixed reference feature objects of the degree of correlation Model training can also reduce the quantity of training characteristics to the greatest extent, under the premise of guaranteeing that model is accurate to further improve mould The accuracy of type training.

2) model is chosen.In model selection, the embodiment of the present application uses the thought of regression fit, by observing dependent variable Appropriate regression function is selected with the data distribution of independent variable.By analysis, self-clinching data have strong correlation with what is selected Property fixed reference feature object between be in apparent linear relationship, i.e., the growth of self-clinching data volume with the growth of fixed reference feature object and Increase, and is presentation linear increase, so, such as the such business of business of withholing in batches, should be based on industry The stabilization of business and linear increase development.That is, the growth of self-clinching data and each fixed reference feature object is in apparent aobvious Sexual intercourse, therefore data can be carried out using multiple linear regression (Multiple Linear Regression, MLR) model Prediction, it can select multiple linear regression model as initial model to carry out model training.Multiple linear regression is by Primary data find a linear equation describe two or more features (independent variable) and export (dependent variable) between relationship, And with this linear equation come prediction result.

The mathematical form of multiple linear regression is as follows:

Y=b0+b1x1+b2x2+b3x3+ ...+bnxn.

In above-mentioned formula, y indicates dependent variable；X1, x2, x3 indicate independent variable；B1, b2, b3 are corresponding to x1, x2, x3 B1, b2, b3, can also be understood to correspond to the independent variable weight of these independents variable of x1, x2, x3 by independent variable coefficient, and weight It can reflect influence size of the independent variable to dependent variable, such as the weighted value of the higher independent variable of the degree of correlation is bigger, shows that its is right The influence of dependent variable is bigger；B0 can be understood as a customized constant, not had to according to the type of dependent variable, b0 can be set For different values, in the specific implementation process, b0 may be set to be 0.

3) training pattern.It, can be according to the aforementioned fixed reference feature object selected after choosing initial training pattern Model training is carried out to initial training pattern, to obtain trained batch tasks prediction model.

It is possible, firstly, to determine multiple historical time sections from history batch data, such as history batch data is 1 month Data, then then 6 isometric historical time sections can be divided into according to same time interval (such as 5 days) by 1 month, i.e., - the 5 day 1 day is first historical time section, and-the 10 day the 6th day is second historical time section, and-the 15 day the 11st day is the Three historical time sections ,-the 10 day the 6th day is second historical time section, and-the 15 day the 11st day is third historical time Section ,-the 20 day the 16th day be the 4th historical time section ,-the 25 day the 21st day be the 5th historical time section, the 26th day-the 30 days are the 6th historical time section.

Then, the corresponding data value of each references object in each historical time section then is respectively extracted, such as can be mentioned Account number, receipt number, the overdue receipt number in each historical time section are taken, then available 6 groups of account numbers, receipt number, overdue Receipt number.

It is possible to further according to the corresponding data value of each fixed reference feature object in each historical time section, to first The batch tasks prediction model (multiple linear regression model i.e. above-mentioned) of beginning is trained, with the batch tasks after being trained Prediction model.Specifically, this above-mentioned 6 groups of data can be substituted into respectively in above-mentioned multiple linear regression equations, i.e., respectively will Independent variable x1, x2, x3 in above-mentioned multiple linear regression equations are substituted into above-mentioned 6 groups of data, so as to calculate b1, b2, B3, it can obtain the corresponding independent variable weight of these independents variable, such as calculated b1, b2, b3 are 0.4 respectively, 0.6, 0.5, then the batch tasks prediction model that training obtains are as follows: y=b0+0.4x1+0.6x2+0.5x3.

It should be noted that above-mentioned be only illustrated model training process with relatively simple understanding mode, having During the model training of body, generally may include take turns repetitive exercises, such as can be calculated by least square method b1, B2, b3 are just not explained in detail herein.

In addition, for the batch tasks prediction model (i.e. y=b0+0.4x1+0.6x2+0.5x3) that training obtains, it is therein X1, x2, x3 can indicate the actual quantity of each fixed reference feature object, can also indicate the object increment of each references object, In the specific implementation process, customized understanding can be carried out to x1, x2, x3 according to practical business demand.

After obtaining the batch tasks prediction model for batch tasks by above description, then it can be appointed based on the batch Business prediction model carries out the prediction of data volume term of reference to this batch processing of batch tasks, below in conjunction with shown in Fig. 3 Process is illustrated the prediction process.

Step 301: parsing the pending data type in batch tasks.

As previously described, according to the difference of the type of batch tasks, corresponding pending data type may not also Together, by taking automatic batch withholds business as an example, pending data type is, for example, self-clinching data.

Step 302: in history batch data, determining the corresponding fixed reference feature object of pending data type.

Continue above-mentioned batch to withhold for business, fixed reference feature object corresponding with self-clinching data is, for example, previous embodiment In account number, receipt number, the overdue receipt number, loan balance, etc. mentioned.

Step 303: transferring the corresponding target data values of each fixed reference feature object.

Wherein, the corresponding target data values of fixed reference feature object are right in preset duration for characterizing the fixed reference feature object The data value answered, such as take the history batch execution data in 1 month, the corresponding target data values of each fixed reference feature object are For the corresponding data value of each fixed reference feature object in this 1 month.

Step 304: according to preparatory trained batch tasks prediction model and the corresponding target of each fixed reference feature object Data value determines the corresponding reference data amount of each fixed reference feature object.

Based on the batch tasks prediction model that above-mentioned training obtains, i.e. y=b0+0.4x1+0.6x2+0.5x3 can will be each The corresponding reference data amount of a fixed reference feature object substitutes x1, x2, x3 in the formula respectively, it can obtain 0.4x1, The corresponding value of 0.6x2,0.5x3, the corresponding value of 0.4x1,0.6x2,0.5x3 are the corresponding reference number of each fixed reference feature object According to amount.

Step 305: according to the corresponding reference data amount of each fixed reference feature object, determining the data volume reference of batch tasks Threshold value, to obtain carrying out the data volume term of reference of this batch tasks processing.

Again since b0 is customized constant, according to the above-mentioned corresponding reference number of each fixed reference feature object being calculated According to amount, it is possible to calculate the value of y accordingly to get the data volume reference threshold of batch tasks is arrived, and then according to some thresholds It is worth range and condition is set, can correspondingly determines the corresponding data volume term of reference of batch tasks.

As previously described, the independents variable such as x1, x2, x3 in trained batch tasks prediction model can indicate corresponding The object increment of fixed reference feature object, it can determine object increment of each fixed reference feature object in the first scheduled duration, Such as the object increment within 1 month, it is predetermined first further according to batch tasks prediction model and each fixed reference feature object Object increment in duration determines the corresponding data increment of each fixed reference feature object, finally according to each fixed reference feature object In the second scheduled duration data volume (such as before the processing of this batch tasks upper primary data volume or on several times Average amount) and data increment in the first scheduled duration, determine the corresponding reference data amount of each fixed reference feature object. Further, then balanced growth amplitude of the data volume within third scheduled duration (such as June) of batch tasks is determined, and according to The balanced growth amplitude and the corresponding reference data amount of each fixed reference feature object, determine batch tasks finally corresponding data volume Reference threshold.

According to the thought of above-mentioned incremental computations, such as obtained predictor formula are as follows: the same day batch button data increment=6 month Balanced growth amplification+last month increases account number * account weight+last month newly-increased receipt number * receipt weight+last month newly-increased loan newly Money remaining sum * loan balance weight.Correspond to above-mentioned batch tasks prediction model, i.e., corresponding y=b0+0.4x1+0.6x2+ 0.5x3, b0 are 6 months balanced growth amplification, are a known constants, and x1, x2, x3 respectively indicate last month newly-increased account Number, last month increase receipt number, last month newly-gained loan remaining sum newly, and account weight, receipt weight, loan balance weight correspond to It is 0.4,0.6,0.5.Pass through the formula, it can predicting the same day batch button data increment, (i.e. the data of this batch tasks increase Amount), and then (the data of actually withholing of batch tasks several times on or of the data volume of actually withholing by it with last batch tasks again The average value of amount) it is added, then the data volume reference threshold of this available batch tasks, to realize to batch auto deduction The Accurate Prediction of the self-clinching total amount of business.

In the first method of determination, the analysis to historical data is realized in the way of machine learning, concludes and answers With, and then the trained machine mould of machine learning can be used and be effectively predicted, improve entire batch processing system Intelligence, the efficiency predicted by machine mould is also higher, so as to improve the efficiency of prediction, and then improves batch and appoints The treatment effeciency and timeliness of business.

Second of method of determination

(such as 1 month or 10 days or 15 days) can be obtained in scheduled duration and carried out multiple batch tasks processing History batch processing statistical data, and then recycle predetermined process mode to handle batch tasks statistical data, such as adopt The data volume term of reference for this batch tasks is dynamically calculated with the Computation schema that aforementioned machines learn.

That is, can provisionally be moved to data volume term of reference by algorithm in second of method of determination The prediction in real time of state ground, in this way, corresponding scheduled duration can be flexibly set when carrying out this prediction, such as can be with In the way of time inverted order of withholing in batches, history service processing data that 500 times nearest batches are withholdd as this into The calculation basis of row prediction, it is all newest historical data that each history service can be made to handle data in this way, so as to Including the considerations of being handled cmpletely with the batch tasks for closing on the time, according to the principle bigger more the correlation closed on, institute Can ensure the accuracy predicted to a certain extent by this way.

In the embodiment of the present application, including the case where history batch processing, being fully considered, i.e., using historical data as Data characteristics is excavated from the mass data of history in basis, by carrying out analysis to these data characteristicses to export batch Decision.It is able to ascend the accuracy of batch forecast by this way, real time monitoring batch is run, if it find that abnormal block in time In batches, guarantee the correctness of batch data processing, so as to avoid the abnormal bring loss of batch.

Based on the same inventive concept, the embodiment of the present application provides a kind of device for handling batch tasks.Processing batch is appointed The method that the processing batch tasks in previous embodiment may be implemented in the device of business.It is shown in Figure 4, in the embodiment of the present application Processing batch tasks device include the first determining module 401, the second determining module 402, third determining module 403, batch It blocks module 404 and executes in batches module 405, in which:

First determining module 401, for determining whether batch tasks meet preset batch processing trigger condition；

Second determining module 402, for determining the real data of batch tasks when meeting batch processing trigger condition Amount；

Third determining module 403 determines that batch tasks are criticized for the history batch execution data according to batch tasks Measure the data volume term of reference of processing；

Batch blocks module 404, if blocking for actual amount of data not within data volume term of reference for batch This batch processing of task；

Module 405 is executed in batches, if for actual amount of data within data volume term of reference, according to actual amount of data Batch processing is carried out to batch tasks.

In a kind of possible embodiment, third determining module 403 is used for:

Parse the pending data type in batch tasks；

In history batch data, the corresponding fixed reference feature object of pending data type is determined, and transfer each reference The corresponding target data values of feature object, target data values are for characterizing fixed reference feature object corresponding data in preset duration Value；

It is corresponding according to the corresponding each fixed reference feature object of batch tasks prediction model core trained in advance of batch tasks Target data values determine the corresponding reference data amount of each fixed reference feature object, and corresponding according to each fixed reference feature object Reference data amount determines the data volume reference threshold of batch tasks；Wherein, batch tasks prediction model is according to history lot number What the corresponding data value training of fixed reference feature object in obtained；

According to the data volume reference threshold of batch tasks, data volume term of reference is determined.

In a kind of possible design, the device of the processing batch tasks in the embodiment of the present application further includes model training mould Block 406, is used for:

From all feature objects that history batch data includes, fixed reference feature object is determined according to default selection strategy； Wherein, each feature object data volume corresponding with pending data type progress batch processing has incidence relation；

Multiple historical time sections are determined from history batch data, and extract each ginseng in each historical time section respectively Examine the corresponding data value of feature object；

It is pre- to initial batch tasks according to the corresponding data value of each fixed reference feature object in each historical time section It surveys model to be trained, with the batch tasks prediction model after being trained.

In a kind of possible design, model training module 406 is used for:

Determine the degree of correlation between each feature object data volume corresponding with pending data type progress batch processing；

The feature object that the degree of correlation meets default screening conditions is determined as fixed reference feature object.

In a kind of possible design, model training module 406 is used for:

The feature object that the degree of correlation is greater than predetermined relevance threshold is determined as fixed reference feature object；Alternatively,

According to the sequence that the degree of correlation is descending, the feature object for being located at the predetermined quantity of front is determined as fixed reference feature Object.

In a kind of possible design, model training module 406 is used for:

From all feature objects, select predetermined characteristic object as fixed reference feature object.

In a kind of possible design, model training module 406 is used for:

According to the object increment of batch tasks prediction model and each fixed reference feature object in the first scheduled duration, determine The corresponding data increment of each fixed reference feature object；

According to each fixed reference feature object in the data volume in the second scheduled duration and the data in the first scheduled duration Increment determines the corresponding reference data amount of each fixed reference feature object.

In a kind of possible design, model training module 406 is used for:

Determine balanced growth amplitude of the data volume of batch tasks in third scheduled duration；

Amplitude and the corresponding reference data amount of each fixed reference feature object are serviced according to balanced growth, determines batch tasks Data volume reference threshold.

In a kind of possible design, actual amount of data and data volume term of reference include the batch processing of batch tasks The corresponding total amount of the task of quantity and all quantity.

All related contents for each step that the embodiment of the method for processing batch tasks above-mentioned is related to can be quoted The application applies the function description of functional module corresponding to the device of the processing batch tasks in example, and details are not described herein.

It is schematical, only a kind of logical function partition to the division of module in the embodiment of the present application, it is practical to realize When there may be another division manner, in addition, each functional module in each embodiment of the application can integrate at one It manages in device, is also possible to individualism, can also be integrated in two or more modules in a module.Above-mentioned integrated mould Block both can take the form of hardware realization, can also be realized in the form of software function module.

Based on the same inventive concept, the embodiment of the present application also provides a kind of calculating equipment, as shown in figure 5, the application is implemented Calculating equipment in example includes at least one processor 501, and the memory 502 that is connect at least one processor 501 and Communication interface 503, does not limit the specific connection medium between processor 501 and memory 502 in the embodiment of the present application, in Fig. 5 It is for passing through bus 500 between processor 501 and memory 502 and connect, bus 500 is indicated in Fig. 5 with thick line, other Connection type between component is only to be schematically illustrated, does not regard it as and be limited.Bus 500 can be divided into address bus, number According to bus, control bus etc., only to be indicated with a thick line in Fig. 5, it is not intended that an only bus or one convenient for indicating The bus of seed type.

In the embodiment of the present application, memory 502 is stored with the instruction that can be executed by least one processor 501, at least The instruction that one processor 501 is stored by executing memory 502, can execute institute in full link performance test method above-mentioned Include the steps that.

Wherein, processor 501 is the control centre for calculating equipment, can use various interfaces and connection entirely calculates The various pieces of equipment are stored in memory 502 by running or executing the instruction being stored in memory 502 and calling Data, calculate equipment various functions and processing data, thus to calculate equipment carry out integral monitoring.Optionally, processor 501 may include one or more processing modules, and processor 501 can integrate application processor and modem processor, wherein place The main processing operation system of device 501, user interface and application program etc. are managed, modem processor mainly handles wireless communication. It is understood that above-mentioned modem processor can not also be integrated into processor 501.In some embodiments, it handles Device 501 and memory 502 can realize that in some embodiments, they can also be on independent chip on the same chip It realizes respectively.

Processor 501 can be general processor, such as central processing unit (CPU), digital signal processor, dedicated integrated Circuit, field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware Component may be implemented or execute each method, step disclosed in the embodiment of the present application and logic diagram.General processor can be with It is microprocessor or any conventional processor etc..The step of method in conjunction with disclosed in the embodiment of the present application, can direct body Now executes completion for hardware processor, or in processor hardware and software module combine and execute completion.

Memory 502 is used as a kind of non-volatile computer readable storage medium storing program for executing, can be used for storing non-volatile software journey Sequence, non-volatile computer executable program and module.Memory 502 may include the storage medium of at least one type, It such as may include flash memory, hard disk, multimedia card, card-type memory, random access storage device (Random Access Memory, RAM), static random-access memory (Static Random Access Memory, SRAM), may be programmed read-only deposit Reservoir (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), band Electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic storage, disk, CD etc..Memory 502 can be used for carrying or storing have instruction or data The desired program code of structure type and can by any other medium of computer access, but not limited to this.The application is real Applying the memory 502 in example can also be circuit or other devices that arbitrarily can be realized store function, for storing program Instruction and/or data.

Communication interface 503 can be used for the coffret communicated, can by communication interface 503 receive data or Person sends data, and then is communicated in other equipment.

The structural schematic diagram further of calculating equipment shown in Figure 6, the calculating equipment further include helping to calculate The basic input/output (I/O system) 601 of information is transmitted between each device in equipment, is used for storage program area 602, the mass-memory unit 605 of application program 603 and other program modules 604.

Basic input/output 601 includes display 608 for showing information and inputs information for user The input equipment 607 of such as mouse, keyboard etc.Wherein display 608 and input equipment 607 are all by being connected to system bus 500 basic input/output 601 is connected to processor 501.The basic input/output 601 can also include defeated Enter o controller for receiving and handling the input from multiple other equipment such as keyboard, mouse or electronic touch pen.Class As, input and output controller also provides output to display screen, printer or other kinds of output equipment.

The mass-memory unit 605 is by being connected to the bulk memory controller (not shown) of system bus 500 It is connected to processor 501.The mass-memory unit 605 and its associated computer-readable medium are that the server packet mentions For non-volatile memories.That is, mass-memory unit 605 may include such as hard disk or CD-ROM drive etc Computer-readable medium (not shown).

According to various embodiments of the present invention, which can also pass through the network connections such as internet to net Remote computer operation on network.Namely the calculating equipment can be by the communication interface that is connected on the system bus 500 503 are connected to network 606, in other words, communication interface 503 can be used also to be connected to other kinds of network or remote computation Machine system (not shown).

Based on the same inventive concept, the embodiment of the present application also provides a kind of storage medium, which is, for example, to calculate Machine readable storage medium storing program for executing, the computer-readable recording medium storage have computer instruction, when the computer instruction on computers When operation, so that the step of computer executes the method for processing batch tasks as the aforementioned.

In some possible embodiments, the various aspects of the method for processing batch tasks provided by the embodiments of the present application It is also implemented as a kind of form of program product comprising program code, when described program product is run on computers, Said program code is used to that the computer to be made to execute the place for stating the illustrative embodiments various according to the present invention of description above Manage the step in the method for batch tasks.

It should be understood by those skilled in the art that, the embodiment of the present invention can provide as method, system or computer program Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the present invention Apply the form of example.Moreover, it wherein includes the computer of computer usable program code that the present invention, which can be used in one or more, The shape for the computer program product implemented in usable storage medium (including but not limited to magnetic disk storage and optical memory etc.) Formula.

The present invention is referring to method, the process of equipment (system) and computer program product according to the embodiment of the present application Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real The device for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates, Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or The function of being specified in multiple boxes.

These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one The step of function of being specified in a box or multiple boxes.

Obviously, various changes and modifications can be made to the invention without departing from essence of the invention by those skilled in the art Mind and range.In this way, if these modifications and changes of the present invention belongs to the range of the claims in the present invention and its equivalent technologies Within, then the present invention is also intended to include these modifications and variations.

Claims

1. a kind of method for handling batch tasks, which is characterized in that the described method includes:

Determine whether batch tasks meet preset batch processing trigger condition；

According to the corresponding history batch execution data of the batch tasks, the number that the batch tasks are carried out with batch processing is determined According to amount term of reference；

If the actual amount of data not within the data volume term of reference, blocks this batch for the batch tasks Amount processing；

If the actual amount of data is within the data volume term of reference, according to the actual amount of data to the target batch Amount task carries out batch processing.

2. the method as described in claim 1, which is characterized in that according to the corresponding history batch data of the batch tasks, really The fixed data volume term of reference that the batch tasks are carried out with batch processing, comprising:

Parse the pending data type in the batch tasks；

In the history batch data, the corresponding fixed reference feature object of the pending data type is determined, and transfer each The corresponding target data values of the fixed reference feature object, the target data values are for characterizing the fixed reference feature object default Corresponding data value in duration；

According to the batch tasks corresponding trained in advance batch tasks prediction model and each fixed reference feature object pair The target data values answered determine the corresponding reference data amount of each fixed reference feature object, and according to each described with reference to special The corresponding reference data amount of object is levied, determines the data volume reference threshold of the batch tasks；Wherein, the batch tasks prediction Model is obtained according to the corresponding data value training of the fixed reference feature object in the history batch data；

3. method according to claim 2, which is characterized in that the batch tasks prediction model is trained in the following way It arrives:

From all feature objects that the history batch data includes, the fixed reference feature pair is determined according to default selection strategy As；Wherein, each feature object data volume corresponding with pending data type progress batch processing has incidence relation；

Multiple historical time sections are determined from the history batch data, and extract each institute in each historical time section respectively State the corresponding data value of fixed reference feature object；

4. method as claimed in claim 3, which is characterized in that all feature objects for including from the history batch data In, the fixed reference feature object is determined according to default selection strategy, comprising:

5. method as claimed in claim 3, which is characterized in that all feature objects for including from the history batch data In, the fixed reference feature object is determined according to default selection strategy, comprising:

6. method according to claim 2, which is characterized in that determine the corresponding reference data of each fixed reference feature object Amount, comprising:

According to the object of the batch tasks prediction model and each fixed reference feature object in first scheduled duration Increment determines the corresponding data increment of each fixed reference feature object；

According to each fixed reference feature object in the data volume in the second scheduled duration and in first scheduled duration Data increment determines the corresponding reference data amount of each fixed reference feature object.

7. method as claimed in claim 6, which is characterized in that according to the corresponding reference data of each fixed reference feature object Amount, determines the data volume reference threshold of the batch tasks, comprising:

According to the balanced growth amplitude and the corresponding reference data amount of each fixed reference feature object, determine that the batch is appointed The data volume reference threshold of business.

8. a kind of device for handling batch tasks, which is characterized in that described device includes:

Second determining module, for determining the real data of the batch tasks when meeting the batch processing trigger condition Amount；

Third determining module, for the history batch execution data according to the batch tasks, determine to the batch tasks into The data volume term of reference of row batch processing；

Batch blocks module, if blocking for the actual amount of data not within the data volume term of reference for institute State this batch processing of batch tasks；

Module is executed in batches, if for the actual amount of data within the data volume term of reference, according to the reality Data volume carries out batch processing to the target batch task.

9. a kind of calculating equipment, which is characterized in that including at least one processor and at least one processor, wherein described Memory is stored with computer program, when described program is executed by the processor, so that the processor perform claim is wanted The step of seeking any one of 1-7 the method.

10. a kind of storage medium, which is characterized in that the storage medium is stored with computer instruction, when the computer instruction When running on computers, so that computer is executed such as the step of any one of claim 1-7 the method.