CN106095554B - The method and device of batch data processing is carried out in the online stage in the daytime - Google Patents
The method and device of batch data processing is carried out in the online stage in the daytime Download PDFInfo
- Publication number
- CN106095554B CN106095554B CN201610439380.8A CN201610439380A CN106095554B CN 106095554 B CN106095554 B CN 106095554B CN 201610439380 A CN201610439380 A CN 201610439380A CN 106095554 B CN106095554 B CN 106095554B
- Authority
- CN
- China
- Prior art keywords
- processing
- batch
- batch service
- request
- processing request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/02—Banking, e.g. interest calculation or account maintenance
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Development Economics (AREA)
- Technology Law (AREA)
- General Business, Economics & Management (AREA)
- Strategic Management (AREA)
- Marketing (AREA)
- Economics (AREA)
- General Engineering & Computer Science (AREA)
- Financial Or Insurance-Related Operations Such As Payment And Settlement (AREA)
Abstract
The present invention provides a kind of method and devices that batch data processing is carried out in the online stage in the daytime, wherein method includes: that batch service processing request is received in online business interaction；Batch service processing request is divided into the different queue of processing priority according to the mark that the batch service handles request；The batch service processing request is sequentially being handled from the background during online business processing according to the priority of the queue.The present invention can allow operator that can view the processing result of batch data business in current trading day, improve the satisfaction of customer.
Description
Technical field
The present invention relates to field of computer technology, more particularly to a kind of big data processing method quasi real time, specifically
It is exactly a kind of method and device that batch data processing is carried out in the online stage in the daytime.
Background technique
It is well known that banking is divided into online transaction business, (online transaction business refers to reception and the place of service request
Reason is all real-time, once processing result will be returned into requesting party, i.e. requirement of real-time in first time after the completion of processing
High business, for example, withdrawal transaction, money transfer transactions etc.) and batch processing business (batch processing business refers to the business of user
Request is not set about being handled immediately after being received, but accumulates latter same submission processing to a certain extent, then computer
These service requests are handled one by one according to rule ordering, all request be disposed after again by processing result together
Output, the i.e. lower business of requirement of real-time, for example, wage keeps accounts, report generation, account balance historical record etc.).In the daytime
Online phase process online transaction business carries out batch processing business in the stage at night.Batch processing business usually requires to carry out
It is a large amount of to calculate, it takes a long time and consume duration and be not fixed, it is general to be handled in batches by backstage at night, and by processing result
Operator is supplied in T+1 days (i.e. next day of current trading day)；And online transaction business processing usually requires that banking
Business processing system is enable to respond quickly and returns in real time, allows the operator to obtain processing result as early as possible.
But under a little situations, operator needs to obtain the processing knot of batch processing business at current trading day (i.e. T days)
At this moment fruit has to allow the technical staff for having system maintenance right rear since operator can not directly initiate on foreground
Platform is directly submitted, poor operability.Moreover, batch processing business meeting long-time occupying system resources, due to system resource
It cannot discharge for a long time, the processing of online transaction business certainly will be influenced, there are the on-line transaction production safeties risk such as unsuccessfully, shadows
Ring the progress of bank's routine work.
Therefore, how meet online transaction business processing concurrency it is big, the response time is short under the premise of, steadily carry out
Data processing of large quantities, and processing result can be returned on T, it is the long-term urgent problem to be solved of those skilled in the art.
Summary of the invention
In view of this, the technical problem to be solved in the present invention is that providing a kind of in the progress of online stage in the daytime batch data
The method and device of processing, solving the problems, such as in the prior art can not be in the processing result of transaction same day acquisition batch service.
In order to solve the above-mentioned technical problem, a specific embodiment of the invention provides one kind and is criticized in the online stage in the daytime
The method for measuring data processing, comprising: batch service processing request is received in online business interaction；At the batch service
Batch service processing request is divided into the different queue of processing priority by the mark for managing request；According to the preferential of the queue
Grade is sequentially handling the batch service processing request during online business processing from the background.
A specific embodiment of the invention also provides a kind of device that batch data processing is carried out in the online stage in the daytime, packet
It includes: receiving unit, for receiving batch service processing request in online business interaction；Queue assignment unit, for according to institute
Batch service processing request is divided into the different queue of processing priority by the mark for stating batch service processing request；Processing is single
Member is asked during online business processing sequentially handling batch service processing from the background for the priority according to the queue
It asks.
Above-mentioned specific embodiment according to the present invention is it is found that the method for carrying out batch data processing in the online stage in the daytime
And device at least has the advantages that the processing request that batch service is received in online transaction (online business) interaction,
Data mart modeling is carried out to batch processing business datum on backstage, is not take up the time window of online transaction, and operator is in T
Day (current trading day) can view the processing result of batch service；According to the priority of batch data business in background process
Batch data business, and using the dynamic priority algorithm based on batch data business throughput and processing time in background process
The other batch data business of lowest priority, can be rationally using online window in the daytime, so that each batch data service request
It has the opportunity to be handled, operator can view the processing result of batch data business in current trading day, improve and care for
The satisfaction of visitor.
It is to be understood that above-mentioned general description and following specific embodiments are merely illustrative and illustrative, not
The range of the invention to be advocated can be limited.
Detailed description of the invention
Following appended attached drawing is part of specification of the invention, depicts example embodiments of the present invention, institute
Attached drawing is used to illustrate the principle of the present invention together with the description of specification.
Fig. 1 is a kind of method for carrying out batch data processing in the online stage in the daytime that the specific embodiment of the invention provides
Specific embodiment one flow chart；
Fig. 2 is a kind of method for carrying out batch data processing in the online stage in the daytime that the specific embodiment of the invention provides
Specific embodiment two flow chart；
Fig. 3 is a kind of method for carrying out batch data processing in the online stage in the daytime that the specific embodiment of the invention provides
Specific embodiment three flow chart；
Fig. 4 is a kind of method for carrying out batch data processing in the online stage in the daytime that the specific embodiment of the invention provides
Specific embodiment four flow chart；
Fig. 5 is a kind of device that batch data processing is carried out in the online stage in the daytime that the specific embodiment of the invention provides
Specific embodiment one schematic block diagram；
Fig. 6 is a kind of device that batch data processing is carried out in the online stage in the daytime that the specific embodiment of the invention provides
Specific embodiment two schematic block diagram；
Fig. 7 is a kind of device that batch data processing is carried out in the online stage in the daytime that the specific embodiment of the invention provides
Specific embodiment three schematic block diagram；
Fig. 8 is a kind of device that batch data processing is carried out in the online stage in the daytime that the specific embodiment of the invention provides
Specific embodiment four schematic block diagram.
Specific embodiment
Understand in order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below will with attached drawing and in detail
Narration clearly illustrates the spirit of disclosed content, and any skilled artisan is understanding the content of present invention
After embodiment, when the technology that can be taught by the content of present invention, it is changed and modifies, without departing from the essence of the content of present invention
Mind and range.
The illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but not as a limitation of the invention.
In addition, in the drawings and embodiments the use of element/component of same or like label is for representing same or like portion
Point.
About " first " used herein, " second " ... etc., not especially censure the meaning of order or cis-position,
It is non-to limit the present invention, only for distinguish with same technique term description element or operation.
About direction term used herein, such as: upper and lower, left and right, front or rear etc. are only the sides with reference to attached drawing
To.Therefore, the direction term used is intended to be illustrative and not intended to limit this creation.
It is open term, i.e., about "comprising" used herein, " comprising ", " having ", " containing " etc.
Mean including but not limited to.
About it is used herein " and/or ", including any of the things or all combination.
About term used herein " substantially ", " about " etc., to modify it is any can be with the quantity or mistake of microvariations
Difference, but this slight variations or error can't change its essence.In general, microvariations that such term is modified or error
Range in some embodiments can be 20%, in some embodiments can be 10%, can be in some embodiments 5% or its
His numerical value.It will be understood by those skilled in the art that the aforementioned numerical value referred to can be adjusted according to actual demand, it is not limited thereto.
It is certain to describe the word of the application by lower or discuss in the other places of this specification, to provide art technology
Personnel's guidance additional in relation to the description of the present application.
Fig. 1 is a kind of method for carrying out batch data processing in the online stage in the daytime that the specific embodiment of the invention provides
Specific embodiment one flow chart, as shown in Figure 1, online transaction (online business) interaction in receive batch service processing ask
It asks, data mart modeling is carried out to batch processing business datum on backstage, is not take up the time window of online transaction.
The specific embodiment shown in the drawings includes:
Step 101: batch service processing request is received in online business interaction.Online business (on-line transaction) refers to connection
The reception and processing of machine service request are all in real time, to tie once will will handle in first time after the completion of online business processing
Fruit is transmitted to requesting party.Batch service processing request is received in online business interaction, is not take up the time window of online business.
Step 102: batch service processing request is divided by processing according to the mark that the batch service handles request
The different queue of priority.In specific embodiments of the present invention, queue includes very urgent queue, urgent queue and common team
Column, the processing priority of the very urgent queue, the urgent queue and the common queue successively reduce.Processing is completed non-
Often after the batch service processing request in urgent queue, the batch service processing request in urgent queue can be just handled；It has handled
After the batch service processing request in urgent queue, the batch service processing request in common queue can be just handled.It is very tight
Batch service processing request urgency level highest in anxious queue, can obtain the processing of highest priority, highest priority
It can maintain always to complete until its processing, each batch service processing request in the very urgent queue of entrance can obtain immediately
Processing, in very urgent queue multiple batch services handle request, then according to request initiate the time time order and function
Sequence is successively handled.Batch service processing in urgent queue requests urgent journey to be lower than at the batch service in very urgent queue
Reason request；And if batch service processing request in urgent queue can preset processing time window threshold value (such as
Processing time window threshold value can be set to 5 minutes) in completed, then the batch service processing request can normally handle knot
Beam；If it exceeds preset processing time window threshold value still without having handled, then stops to handle batch service processing request, and
And the priority of batch service processing request can decline, batch service processing request can be redistributed into common queue
It goes, is handled by the processing rule of common queue, the mode of this priority decline is to prevent from growing based on the considerations of fairness
Phase occupies process resource, and is also to prevent the very urgent queue of abuse, encourages to initiate short operation processing request.Common queue
The urgency level of interior batch service processing request is minimum, only at the batch service in very urgent queue and urgent queue
After reason request is processed, the batch service processing request in common queue, most of batch service are just started to process
Processing request belongs to common queue.
Step 103: sequentially handling the batch from the background during online business processing according to the priority of the queue
Business processing request.In a specific embodiment of the invention, step 103 is specifically included: step 1031 is in online business process phase
Between handle in the very urgent queue the batch service processing request when, according to the batch service handle request hair
Playing the time sequentially handles batch service processing request；Step 1032 handles the urgent queue during online business processing
When the interior batch service handles request, the batch is sequentially handled according to the initiation time that the batch service handles request
Business processing request；Since the processing rule of each batch service processing request is fixed, each batch service processing request
The input data object and output data object being related to are also fixation, and difference is the of different sizes of processing data volume, therefore
In the specific embodiment of the invention, step 1033 handles the batch service in the common queue during online business processing
When processing request, the handling capacity of request is handled according to the batch service and handling duration successively handles the batch service and handles
Request.
Referring to Fig. 1, batch service processing request is received in online transaction (online business) interaction, on backstage to batch industry
Business processing request is handled, and is not take up the time window of online transaction, and operator was in T days (current trading day) energy
View the processing result of batch service processing request；The priority of request is handled in background process batch industry according to batch service
Business processing request, and the other in background process lowest priority of the handling capacity and handling duration requested is handled according to batch service
Batch service processing request, can be rationally using online window in the daytime, so that the processing request of each batch service has the opportunity to
To processing, operator can view the processing result of batch service processing request in current trading day, improve expiring for customer
Meaning degree.
Fig. 2 is a kind of method for carrying out batch data processing in the online stage in the daytime that the specific embodiment of the invention provides
Specific embodiment two flow chart, as shown in Fig. 2, setting processing time window threshold value, if batch service in urgent queue
The handling duration of processing request is more than processing time window threshold value, then stops the processing of batch service processing request, and team
Batch service processing request is included into common queue and is handled by column allocation unit, and the batch service is handled in common queue
Request the processing for continuing remaining data from suspension place.
In the specific embodiment shown in the drawings, step 1032 is specifically included:
Step 10321: successively handling the batch service processing request according to the sequencing that the time is initiated in the request.
In urgent inner queue, the processing sequence of batch service processing request is carried out according to the rule of prerequisite variable.
Step 10322: judging whether the processing time span of the batch service processing request was greater than for the first processing time
Window threshold value.In specific embodiments of the present invention, the first processing time window threshold value is 5 minutes.
Step 10323: if it is greater, then stopping to handle batch service processing request, and batch service processing being asked
It asks and is included into common queue.If the handling duration of batch service processing request is more than processing time window threshold in urgent queue
Value, then stop the batch service processing request processing, and queue assignment unit by the batch service processing request be included into it is general
It is handled in logical queue, batch service processing request (is utilized from the processing that suspension place continues remaining data in common queue
Breakpoint mechanism supports subsequent reentry to handle, and records breakpoint information when stopping, when reentrying processing next time at breakpoint weight
It newly starts to process).
Referring to fig. 2, the mode for the priority decline that the specific embodiment of the invention provides is prevented based on the considerations of fairness
The long-term occupancy process resource of batch service processing request in urgent queue, and abuse of emergency priority can be prevented, it encourages
Operator initiates short job patch business processing request.
Fig. 3 is a kind of method for carrying out batch data processing in the online stage in the daytime that the specific embodiment of the invention provides
Specific embodiment three flow chart, as shown in figure 3, being calculated in common queue at batch service according to handling capacity and handling duration
The changeable weight value for managing request successively handles the processing request of the batch service in common queue according to changeable weight value；At setting
Time window threshold value is managed, if the handling duration of batch service processing request is more than processing time window threshold value in common queue,
Then stop the processing of batch service processing request, and batch service processing request is discharged to the tail of the queue of common queue, commonly
Batch service processing request continues the processing of remaining data from suspension place in queue.
In the specific embodiment shown in the drawings, step 1033 is specifically included:
Step 10331: the dynamic of the batch service processing request is calculated according to the handling capacity and the handling duration
Weighted value.
Step 10332: the batch service processing request is successively handled according to the changeable weight value.
Step 10333: judging whether the handling duration is greater than second processing time window threshold value.
Step 10334: if it is greater, then stopping to handle batch service processing request, and batch service processing being asked
Seek the tail of the queue for being discharged to common queue.
Referring to Fig. 3, the specific embodiment of the invention is according to the handling capacity and processing that batch service processing is requested in common queue
The changeable weight value of duration calculation batch service processing request, the batch in common queue is sequentially handled according to changeable weight value
Business processing request, and stop the batch service processing request that handling duration is more than processing time window threshold value, and by this batch
Amount business processing request is discharged to the tail of the queue of common queue, prevents the batch service in common queue from handling and requests long-term occupancy processing
Resource encourages operator to initiate short job patch business processing request.
In specific embodiments of the present invention, the calculation formula of the changeable weight value Priority are as follows:
Wherein, Priority indicates the priority calculated, when value is the batch service processing request of n times processing in the past
The average value of handling capacity and handling duration ratio；ProcessTime (i) indicates to ask in the processing batch service processing of past i-th
The time used is asked, is calculated as unit of millisecond；InputSize (i) indicates to ask in the processing batch service processing of past i-th
The size of input data when asking, is calculated as unit of byte；OutputSize (i) indicates to handle batch industry in past i-th
The size of output data, is calculated as unit of byte when business processing request.
Wherein, the calculation formula of the InputSize (i) and the OutputSize (i) is respectively as follows:
Wherein, InputRecordLength (k) is the input data when past i-th handles batch service processing request
The record length of object；M indicates the number of input data object, such as i-th processing batch service processing request input data
Comprising m report, each report is an input data object；InputRecordCount (k) is in the processing batch of past i-th
The record strip number of the accessed processing of input data object when measuring business processing request, for example, each report has 10000 rows again,
Every behavior one record；K indicates the order of input data object, i.e. k-th in m input data object；
OutputRecordLength (k) is that the record of the output data object when past i-th handles batch service processing request is long
Degree；P indicates the number of output data object, for example, i-th processing batch service processing request output data includes p report,
Each report is an output data object；OutputRecordCount (k) is to handle batch service processing in past i-th
The record strip number of the accessed processing of output data object when request, for example, each report has 10000 rows, every behavior one again
Record；J indicates the order of output data object, i.e. j-th in p output data object.
Fig. 4 is a kind of method for carrying out batch data processing in the online stage in the daytime that the specific embodiment of the invention provides
Specific embodiment four flow chart, as shown in figure 4, in order to further prevent batch service in common queue to handle request length
Phase occupies process resource, and embodiment has equal opportunities and fairness, at the batch service using round policy control processing overlong time
Reason request, round i.e. batch service processing request are handled in which wheel, and every to carry out primary reentry processing, wheel sub-value is just
From increasing by 1, wheel sub-value is bigger, and processing priority is lower.
In the specific embodiment shown in the drawings, after step 10334, step 1033 further include:
Step 10335: when stopping to handle batch service processing request, corresponding round is requested into batch service processing
Value plus 1, wherein the priority of the wheel sub-value is higher than the changeable weight value, and the priority of the wheel sub-value is lower than institute
Mark is stated, the bigger priority of the wheel sub-value is lower.
Referring to fig. 4, the introducing of round strategy further prevents the processing request of the batch service in common queue is long-term to occupy
Process resource, embodiment has equal opportunities and fairness, handles request using the batch service of round policy control processing overlong time,
Operator is encouraged to initiate long business in short-term.
Fig. 5 is a kind of device that batch data processing is carried out in the online stage in the daytime that the specific embodiment of the invention provides
Specific embodiment one schematic block diagram, device as shown in Figure 5 can be applied in method as Figure 1-Figure 4, joining
Batch service processing request is received in machine affairs (online business) interaction, and data are carried out to batch processing business datum on backstage and are added
Work is not take up the time window of online transaction.
It include connecing in the device that the online stage in the daytime carries out batch data processing in the specific embodiment shown in the drawings
Receive unit 10, queue assignment unit 20 and processing unit 30, wherein receiving unit 10 is used in online business interaction receive to criticize
Measure business processing request；Queue assignment unit 20 is for handling the mark requested for the batch service according to the batch service
Processing request is divided into the different queue of processing priority；Processing unit 30 is used for the priority according to the queue in online business
The batch service processing request is sequentially being handled from the background during processing.In specific embodiments of the present invention, queue includes very
Urgent queue, urgent queue and common queue, the very urgent queue, the urgent queue and the common queue processing
Priority successively reduces.Processing unit 30 further comprises: first processing module 301, Second processing module 302 and third processing
Module 303.Wherein, first processing module 301 is used to handle the institute in the very urgent queue during online business processing
When stating batch service processing request, sequentially handled at the batch service according to the initiation time that the batch service handles request
Reason request.Second processing module 302 is used to handle the batch service in the urgent queue during online business processing
When processing request, the batch service processing request is sequentially handled according to the initiation time that the batch service handles request.The
Three processing modules 303 are used to handle the batch service processing request in the common queue during online business processing
When, the handling capacity of request is handled according to the batch service and handling duration successively handles the batch service processing request.
Referring to Fig. 5, batch service processing request is received in online transaction (online business) interaction, on backstage to batch industry
Business processing request is handled, and is not take up the time window of online transaction, and operator was in T days (current trading day) energy
View the processing result of batch service processing request；The priority of request is handled in background process batch industry according to batch service
Business processing request, and the other in background process lowest priority of the handling capacity and handling duration requested is handled according to batch service
Batch service processing request, can be rationally using online window in the daytime, so that the processing request of each batch service has the opportunity to
To processing, operator can view the processing result of batch service processing request in current trading day, improve expiring for customer
Meaning degree.
Fig. 6 is a kind of device that batch data processing is carried out in the online stage in the daytime that the specific embodiment of the invention provides
Specific embodiment two schematic block diagram, as shown in fig. 6, setting processing time window threshold value, if batch industry in urgent queue
The handling duration of business processing request is more than processing time window threshold value, then stops the processing of batch service processing request, and
Batch service processing request is included into common queue and is handled by queue assignment unit, in common queue at the batch service
Reason request continues the processing of remaining data from suspension place.
In the specific embodiment shown in the drawings, the Second processing module 302 further comprises the first processing
Module 3021, the first judging submodule 3022 and first stop submodule 3023, wherein first processing module 301 be used for according to
The sequencing that the time is initiated in request successively handles the batch service processing request；First judgment module 302 is for judging institute
Whether the processing time span for stating batch service processing request is greater than the first processing time window threshold value；First stops module 303
Stop to handle the batch service processing request when being greater than the first processing time window threshold value for the processing time span,
And batch service processing request is included into common queue by the queue assignment unit 20.
Referring to Fig. 6, the mode for the priority decline that the specific embodiment of the invention provides is prevented based on the considerations of fairness
The long-term occupancy process resource of batch service processing request in urgent queue, and abuse of emergency priority can be prevented, it encourages
Operator initiates short job patch business processing request.
Fig. 7 is a kind of device that batch data processing is carried out in the online stage in the daytime that the specific embodiment of the invention provides
Specific embodiment three schematic block diagram, as shown in fig. 7, being calculated in common queue according to the handling capacity and the handling duration
The changeable weight value of batch service processing request successively handles the processing of the batch service in common queue according to changeable weight value and asks
It asks；Setting processing time window threshold value, if the handling duration of batch service processing request is more than the processing time in common queue
Window threshold value then stops the processing of batch service processing request, and batch service processing request is discharged to common queue
Tail of the queue, batch service processing request continues the processing of remaining data from suspension place in common queue.
In the specific embodiment shown in the drawings, the third processing module 303 further comprises computational submodule
3031, second processing submodule 3032, second judgment submodule 3033 and second stop submodule 3034, wherein calculate submodule
Block 3031 is used to calculate the changeable weight value of the batch service processing request according to the handling capacity and the handling duration；The
Two processing submodules 3032 are used to successively handle the batch service processing request according to the changeable weight value；Second judgement
Module 3033 is for judging whether the handling duration is greater than second processing time window threshold value；Second, which stops submodule 3034, uses
Stop to handle the batch service processing request when handling duration is greater than the second processing time window threshold value, and should
Batch service processing request is discharged to the tail of the queue of common queue.
Referring to Fig. 7, the specific embodiment of the invention is according to the handling capacity and processing that batch service processing is requested in common queue
The changeable weight value of duration calculation batch service processing request, the batch in common queue is sequentially handled according to changeable weight value
Business processing request, and stop the batch service processing request that handling duration is more than processing time window threshold value, and by this batch
Amount business processing request is discharged to the tail of the queue of common queue, prevents the batch service in common queue from handling and requests long-term occupancy processing
Resource encourages operator to initiate short job patch business processing request.
In specific embodiments of the present invention, the calculation formula of the changeable weight value Priority are as follows:
Wherein, Priority indicates the priority calculated, when value is the batch service processing request of n times processing in the past
The average value of handling capacity and handling duration ratio；ProcessTime (i) indicates to ask in the processing batch service processing of past i-th
The time used is asked, is calculated as unit of millisecond；InputSize (i) indicates to ask in the processing batch service processing of past i-th
The size of input data when asking, is calculated as unit of byte；OutputSize (i) indicates to handle batch industry in past i-th
The size of output data, is calculated as unit of byte when business processing request.
Wherein, the calculation formula of the InputSize (i) and the OutputSize (i) is respectively as follows:
Wherein, InputRecordLength (k) is the input data when past i-th handles batch service processing request
The record length of object；The number of m expression input data object；InputRecordCount (k) is in the processing batch of past i-th
The record strip number of the accessed processing of input data object when measuring business processing request；OutputRecordLength (k) is in mistake
The record length of output data object when i-th processing batch service processing being gone to request；The number of p expression output data object；
OutputRecordCount (k) is the accessed processing of output data object when past i-th handles batch service processing request
Record strip number.
Fig. 8 is a kind of device that batch data processing is carried out in the online stage in the daytime that the specific embodiment of the invention provides
Specific embodiment four schematic block diagram, as shown in figure 8, in order to further prevent batch service in common queue to handle request
Long-term to occupy process resource, embodiment has equal opportunities and fairness, utilizes the batch service of round policy control processing overlong time
Processing request, round i.e. batch service processing request is handled in which wheel, every to carry out primary reentry processing, takes turns sub-value
With regard to from increasing by 1, wheel sub-value is bigger, and processing priority is lower.
In the specific embodiment shown in the drawings, the third processing module 303 further includes setting submodule 3035,
Wherein, corresponding wheel is requested into the batch service processing when setting submodule 3035 is for stopping processing batch service processing request
Sub-value adds 1, wherein the priority of the wheel sub-value is higher than the changeable weight value, and the priority of the wheel sub-value is lower than
The mark, the bigger priority of the wheel sub-value are lower.
Referring to Fig. 8, present invention introduces round rule, the batch service in common queue can be further prevented to handle request
Long-term to occupy process resource, embodiment has equal opportunities and fairness, utilizes the batch service of round policy control processing overlong time
Processing request, encourages operator to initiate long business in short-term.
In a specific embodiment of the invention, processing unit 30 further includes register module, inquires and have access to module, bursting tube
It controls module, load balancing module, split merging module and info push module etc..
The each batch service processing request of register module registers, register information mainly includes the data type of processing,
Data processing state registers time, the information such as request teller (operator)；Teller is initiating batch service processing request
When can be according to actual business requirement designated treatment priority, the i.e. urgency level of the processing: it is very urgent, it is urgent and common；It steps on
Note book can save the processing priority that teller specifies, and different processing queues is sent into rear extended meeting according to these priority；Register
It is a very main table, is the carrier of entire process data.
Security management and control module: for banking system, different human roles have different batch service processing request permissions.
It, can access safety control when carrying out registration batch service processing request or consulting the processing result of batch service processing request
Module verifies the identity legitimacy of claimant, the reasonability of batch service processing request, for authentication failure or batch
When the data area of amount business processing request is illegal, carries out prompt and report an error.
Load balancing module: occurring overload in order to prevent influences the processing of on-line transaction in the daytime, and it is equal to devise a load
Weigh module, the maximum number of concurrent that load balancing module can be handled by parameter setting, can be temporary when reaching maximum number of concurrent
Time delay lags the processing of continuous batch service processing request, waits the batch service an of present to handle request and is disposed
Afterwards, then the processing of new batch service processing request is carried out；The introducing of load balancing module is to the stability of banking system and can
Control property is very important, and to provide operability in terms of Properties Control, avoids the occurrence of the nothing in a helpless situation in bad performance
The difficult situation of plan.
Info push module: embodying to enhance user, and user is allow to understand the processed last state of request in time,
Devise info push module.In each round after treatment, info push module obtains data processing from register and asks
The relevant information asked for help, one data process state information of tissue are sent to the claimant.Specific implementation can pass through message team
The mode of column (MQ) sends message.
Split merging module: the module provides customizability of the user about report size.For certain biggish reports,
Number of pages is longer, and user not have access to by aspect, and more points of small reports can be split by the mechanism and successively carry out having access to processing；It is right simultaneously
A report can also be merged by the mechanism in the data result of the dispersion of more rounds processing.
Module is had access in inquiry: data processing request people can have access to the result that module has access to request processing by the inquiry.
As register module, the permission of confidentiality and user identity based on bank data, module is had access in inquiry also must be through
The verifying for crossing security management and control module can just be normally carried out having access to for data after being verified.When data are had access to, Gui Yuanke
To specify in the form of the showing of report, such as every page of how many row information, if need to split the latter and merge and show, closed from the background by splitting
And the fractionation of module merges mechanism and is specifically handled, that is, provides a kind of inquiry side of having access to that can be customized according to client's needs
Formula improves the flexibility of user's interaction, enhances user service experience satisfaction.
The present invention provides a kind of method and device that batch data processing is carried out in the online stage in the daytime, in online transaction
The processing request that batch service is received in (online business) interaction carries out data mart modeling to batch processing business datum on backstage,
It is not take up the time window of online transaction, and operator can view the place of batch service at T days (current trading day)
Manage result；According to the priority of batch data business in background process batch data business, and using based on batch data business
The dynamic priority algorithm of handling capacity and processing time, can be reasonable in the other batch data business of background process lowest priority
Using online window in the daytime, so that each batch data service request has the opportunity to be handled, operator trades currently
Day can view the processing result of batch data business, improve the satisfaction of customer.
The above-mentioned embodiment of the present invention can be implemented in various hardware, Software Coding or both combination.For example, this hair
Bright embodiment can also be the execution above method in data signal processor (Digital Signal Processor, DSP)
Program code.The present invention can also refer to computer processor, digital signal processor, microprocessor or field-programmable gate array
Arrange the multiple functions that (Field Programmable Gate Array, FPGA) is executed.Above-mentioned processing can be configured according to the present invention
Device executes particular task, and machine-readable software code or the firmware generation of the ad hoc approach that the present invention discloses are defined by executing
Code is completed.Software code or firmware code can be developed as different program languages and different formats or form.Can also be
Different target platform composing software codes.However, executing software code and the other types configuration generation of task according to the present invention
Different code pattern, type and the language of code do not depart from spirit and scope of the invention.
The foregoing is merely the schematical specific embodiments of the present invention, before not departing from conceptions and principles of the invention
It puts, the equivalent changes and modifications that any those skilled in the art is made should belong to the scope of protection of the invention.
Claims (11)
1. a kind of method for carrying out batch data processing in the online stage in the daytime, which is characterized in that this method comprises:
Batch service processing request is received in online business interaction；
Handling the mark of request according to the batch service, to be divided into processing priority different by batch service processing request
Queue；And
The batch service processing request is sequentially being handled from the background during online business processing according to the priority of the queue；
The queue includes very urgent queue, urgent queue and common queue, the very urgent queue, the urgent queue
It is successively reduced with the processing priority of the common queue；
The batch service processing request is sequentially being handled from the background during online business processing according to the priority of the queue
The step of, it specifically includes:
When handling the batch service processing request in the common queue during online business processing, according to the batch
The handling capacity and handling duration of business processing request successively handle the batch service processing request；
When handling the batch service processing request in the common queue during online business processing, according to the batch
The step of handling capacity and handling duration of business processing request successively handle the batch service processing request, specifically includes:
The changeable weight value of the batch service processing request is calculated according to the handling capacity and the handling duration；
The batch service processing request is successively handled according to the changeable weight value；
Judge whether the handling duration is greater than second processing time window threshold value；And
If it is greater, then stopping to handle batch service processing request, and batch service processing request is discharged to common queue
Tail of the queue；
The calculation formula of the changeable weight value Priority are as follows:
Wherein, Priority indicates that the priority calculated, value are handled up when being the batch service processing request of n times processing in the past
The average value of amount and handling duration ratio；ProcessTime (i) is indicated in past i-th processing batch service processing request institute
Time, as unit of millisecond；InputSize (i) indicates to input number in the processing batch service processing request of past i-th
According to size, as unit of byte；OutputSize (i) indicates to export in the processing batch service processing request of past i-th
The size of data, as unit of byte.
2. the method for carrying out batch data processing in the online stage in the daytime as described in claim 1, which is characterized in that according to institute
The priority for stating queue is sequentially handling the step of batch service processing is requested during online business processing from the background, specifically
Include:
When handling the batch service processing request in the very urgent queue during online business processing, according to described
The initiation time of batch service processing request sequentially handles the batch service processing request.
3. the method for carrying out batch data processing in the online stage in the daytime as described in claim 1, which is characterized in that according to institute
The priority for stating queue is sequentially handling the step of batch service processing is requested during online business processing from the background, specifically
Include:
When handling the batch service processing request in the urgent queue during online business processing, according to the batch
The initiation time of business processing request sequentially handles the batch service processing request.
4. the method for carrying out batch data processing in the online stage in the daytime as claimed in claim 3, which is characterized in that according to institute
The initiation time for stating batch service processing request sequentially handles the step of batch service processing is requested, and specifically includes:
The batch service processing request is successively handled according to the sequencing that the time is initiated in request；
Judge whether the processing time span of the batch service processing request is greater than the first processing time window threshold value；And
If it is greater, then stopping to handle batch service processing request, and common queue is included into batch service processing request
In.
5. the method for carrying out batch data processing in the online stage in the daytime as described in claim 1, which is characterized in that by this batch
After the step of amount business processing request is discharged to the tail of the queue of common queue, further includes:
When stopping to handle batch service processing request, corresponding wheel sub-value is requested to add 1 batch service processing, wherein institute
The priority for stating wheel sub-value is higher than the changeable weight value, and the priority of the wheel sub-value is lower than the mark, the wheel
Sub-value is bigger, and priority is lower.
6. the method for carrying out batch data processing in the online stage in the daytime as described in claim 1, which is characterized in that described
The calculation formula of InputSize (i) and the OutputSize (i) is respectively as follows:
Wherein, InputRecordLength (k) is the input data object when past i-th handles batch service processing request
Record length；The number of m expression input data object；InputRecordCount (k) is to handle batch industry in past i-th
The record strip number of the accessed processing of input data object when business processing request；OutputRecordLength (k) is in the past i-th
The record length of output data object when secondary processing batch service processing request；The number of p expression output data object；
OutputRecordCount (k) is the accessed processing of output data object when past i-th handles batch service processing request
Record strip number.
7. a kind of device for carrying out batch data processing in the online stage in the daytime, which is characterized in that the device includes:
Receiving unit, for receiving batch service processing request in online business interaction；
Batch service processing request is divided by queue assignment unit, the mark for handling request according to the batch service
The different queue of processing priority；And
Processing unit is sequentially handling the batch during online business processing for the priority according to the queue from the background
Business processing request；
The queue includes very urgent queue, urgent queue and common queue, the very urgent queue, the urgent queue
It is successively reduced with the processing priority of the common queue；
The processing unit further comprises:
Third processing module is asked for handling the processing of the batch service in the common queue during online business is handled
When asking, the handling capacity of request is handled according to the batch service and handling duration successively handles the batch service processing request；
The third processing module further comprises:
Computational submodule, for calculating the dynamic of the batch service processing request according to the handling capacity and the handling duration
Weighted value；
Second processing submodule, for successively handling the batch service processing request according to the changeable weight value；
Second judgment submodule, for judging whether the handling duration is greater than second processing time window threshold value；And
Second stops submodule, stops to handle this batch when being greater than the second processing time window threshold value for the handling duration
Business processing request is measured, and batch service processing request is discharged to the tail of the queue of common queue；
The calculation formula of the changeable weight value Priority are as follows:
Wherein, Priority indicates that the priority calculated, value are handled up when being the batch service processing request of n times processing in the past
The average value of amount and handling duration ratio；ProcessTime (i) is indicated in past i-th processing batch service processing request institute
Time, as unit of millisecond；InputSize (i) indicates to input number in the processing batch service processing request of past i-th
According to size, as unit of byte；OutputSize (i) indicates to export in the processing batch service processing request of past i-th
The size of data, as unit of byte.
8. the device of batch data processing is carried out in the online stage in the daytime as claimed in claim 7, which is characterized in that the place
Reason unit further comprises:
First processing module, for being handled during online business is handled at the batch service in the very urgent queue
When reason request, the batch service processing request is sequentially handled according to the initiation time that the batch service handles request.
9. the device of batch data processing is carried out in the online stage in the daytime as claimed in claim 7, which is characterized in that the place
Reason unit further comprises:
Second processing module is asked for handling the processing of the batch service in the urgent queue during online business is handled
When asking, the batch service processing request is sequentially handled according to the initiation time that the batch service handles request.
10. the device of batch data processing is carried out in the online stage in the daytime as claimed in claim 9, which is characterized in that described
Second processing module further comprises:
First processing submodule, the sequencing for initiating the time according to request successively handle the batch service processing and ask
It asks；
First judging submodule, for judging whether the processing time span of the batch service processing request is greater than the first processing
Time window threshold value；And
First stops submodule, and suspension is handled when being greater than the first processing time window threshold value for the processing time span
Batch service processing request, and batch service processing request is included into common queue by the queue assignment unit.
11. the device of batch data processing is carried out in the online stage in the daytime as claimed in claim 7, which is characterized in that described
Third processing module further include:
Submodule being set, corresponding wheel sub-value is requested into the batch service processing when for stopping processing batch service processing request
Add 1, wherein the priority of the wheel sub-value is higher than the changeable weight value, and the priority of the wheel sub-value is lower than described
Mark, the bigger priority of the wheel sub-value are lower.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610439380.8A CN106095554B (en) | 2016-06-17 | 2016-06-17 | The method and device of batch data processing is carried out in the online stage in the daytime |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610439380.8A CN106095554B (en) | 2016-06-17 | 2016-06-17 | The method and device of batch data processing is carried out in the online stage in the daytime |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106095554A CN106095554A (en) | 2016-11-09 |
| CN106095554B true CN106095554B (en) | 2019-08-23 |
Family
ID=57236868
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610439380.8A Active CN106095554B (en) | 2016-06-17 | 2016-06-17 | The method and device of batch data processing is carried out in the online stage in the daytime |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106095554B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106685853B (en) * | 2016-11-23 | 2020-05-12 | 泰康保险集团股份有限公司 | Method and device for processing data |
| CN106648874B (en) * | 2016-12-30 | 2020-05-22 | 中国建设银行股份有限公司 | Processing method and device for batch tasks |
| CN107819696B (en) * | 2017-11-22 | 2021-09-28 | 中国银行股份有限公司 | Transaction flow control method and system |
| CN107832134B (en) * | 2017-11-24 | 2021-07-20 | 平安科技（深圳）有限公司 | Multitasking method, application server and storage medium |
| CN109064311A (en) * | 2018-07-17 | 2018-12-21 | 中国银行股份有限公司 | Batch based on online mode is withholdd processing method and processing device |
| CN110333942A (en) * | 2019-07-05 | 2019-10-15 | 中国工商银行股份有限公司 | Electronic equipment and the task processing method executed by electronic equipment |
| CN110381150B (en) * | 2019-07-24 | 2021-12-17 | 中国工商银行股份有限公司 | Data processing method and device on block chain, electronic equipment and storage medium |
| CN110610414A (en) * | 2019-09-19 | 2019-12-24 | 中国银行股份有限公司 | Data processing method and system |
| CN111510396B (en) * | 2020-06-30 | 2020-10-27 | 腾讯科技（深圳）有限公司 | Message sending method and related device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2002046887A2 (en) * | 2000-10-23 | 2002-06-13 | Xyron Corporation | Concurrent-multitasking processor |
| CN101246437A (en) * | 2008-01-28 | 2008-08-20 | 中兴通讯股份有限公司 | Built-in real-time system course equalization scheduling method |
| CN101261592A (en) * | 2007-03-07 | 2008-09-10 | 国际商业机器公司 | Method and apparatus for scheduling task request |
| CN101510164A (en) * | 2009-03-25 | 2009-08-19 | 北京中星微电子有限公司 | Multi-service scheduling method, apparatus and system |
| CN102521055A (en) * | 2011-12-15 | 2012-06-27 | 广州杰赛科技股份有限公司 | Virtual machine resource allocating method and virtual machine resource allocating system |
| CN104156264A (en) * | 2014-08-01 | 2014-11-19 | 西北工业大学 | Baseband signal processing task parallelism real-time scheduling method based on multiple GPUs |
| CN105389209A (en) * | 2015-12-25 | 2016-03-09 | 中国建设银行股份有限公司 | Asynchronous batch task processing method and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8863136B2 (en) * | 2011-05-19 | 2014-10-14 | International Business Machines Corporation | Lock control in multiple processor systems |
-
2016
- 2016-06-17 CN CN201610439380.8A patent/CN106095554B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2002046887A2 (en) * | 2000-10-23 | 2002-06-13 | Xyron Corporation | Concurrent-multitasking processor |
| CN101261592A (en) * | 2007-03-07 | 2008-09-10 | 国际商业机器公司 | Method and apparatus for scheduling task request |
| CN101246437A (en) * | 2008-01-28 | 2008-08-20 | 中兴通讯股份有限公司 | Built-in real-time system course equalization scheduling method |
| CN101510164A (en) * | 2009-03-25 | 2009-08-19 | 北京中星微电子有限公司 | Multi-service scheduling method, apparatus and system |
| CN102521055A (en) * | 2011-12-15 | 2012-06-27 | 广州杰赛科技股份有限公司 | Virtual machine resource allocating method and virtual machine resource allocating system |
| CN104156264A (en) * | 2014-08-01 | 2014-11-19 | 西北工业大学 | Baseband signal processing task parallelism real-time scheduling method based on multiple GPUs |
| CN105389209A (en) * | 2015-12-25 | 2016-03-09 | 中国建设银行股份有限公司 | Asynchronous batch task processing method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106095554A (en) | 2016-11-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106095554B (en) | The method and device of batch data processing is carried out in the online stage in the daytime | |
| US7957999B2 (en) | Electronic acquisition system and method | |
| US8522241B1 (en) | Systems and methods for auto-balancing of throughput in a real-time event-driven system | |
| CN107784424A (en) | Task management method, device, computer equipment and storage medium | |
| US11475448B2 (en) | Maintenance plant management method, system and data management server | |
| CN101483601A (en) | Scheduling method and apparatus based on service priority | |
| US20200250598A1 (en) | Systems and methods for risk management in a geographically distributed trading system | |
| US20220358590A1 (en) | Fractional shares order execution methods | |
| CN107862615A (en) | Claims Resolution information processing method, device, computer equipment and storage medium | |
| CN106817424A (en) | Method and system for controlling flowing of access | |
| US11669806B2 (en) | Retirement score calculator | |
| CN112017042A (en) | Resource quota determining method and device based on tweed distribution and electronic equipment | |
| WO2019075972A1 (en) | Method and apparatus for generating no claims based discount data | |
| Tang et al. | Pricing model for dynamic resource overbooking in edge computing | |
| CN114221908A (en) | Dynamic current-limiting fusing processing method and device, computer equipment and storage medium | |
| CN114640700B (en) | Calling frequency control method and device | |
| US10855617B1 (en) | System and method for controlling access to resources in a multicomputer network | |
| CN106453557A (en) | Two-time scale dynamic bidding and resource management algorithm for user in IaaS service | |
| US11086675B2 (en) | Processing future-dated resource reservation requests | |
| US20120179594A1 (en) | Credit allocation in an open order manager | |
| CN116662387A (en) | Service data processing method, device, equipment and storage medium | |
| US20200286163A1 (en) | Intraday resource management system | |
| CN115577983A (en) | Enterprise task matching method based on block chain, server and storage medium | |
| US20180018735A1 (en) | Systems and methods for allocating trades among trading accounts | |
| CN113159751A (en) | Bank transfer amount sharing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |