CN106375241B - Batch data processing method, front-end system, host and batch data processing system - Google Patents
Batch data processing method, front-end system, host and batch data processing system Download PDFInfo
- Publication number
- CN106375241B CN106375241B CN201610754742.2A CN201610754742A CN106375241B CN 106375241 B CN106375241 B CN 106375241B CN 201610754742 A CN201610754742 A CN 201610754742A CN 106375241 B CN106375241 B CN 106375241B
- Authority
- CN
- China
- Prior art keywords
- transaction
- data
- host
- transaction data
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
- H04L47/62—Queue scheduling characterised by scheduling criteria
- H04L47/625—Queue scheduling characterised by scheduling criteria for service slots or service orders
- H04L47/6275—Queue scheduling characterised by scheduling criteria for service slots or service orders based on priority
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
- H04L47/62—Queue scheduling characterised by scheduling criteria
- H04L47/6245—Modifications to standard FIFO or LIFO
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Financial Or Insurance-Related Operations Such As Payment And Settlement (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a batch data processing method, a front-end system, a host and a batch data processing system, wherein the method comprises the following steps: acquiring to-be-processed transaction data submitted by a user; sorting the transaction data to be processed according to the processing type of the transaction data to be processed, and extracting the transaction data needing to be processed by a host; temporarily storing transaction data needing to be processed by a host computer, and simultaneously generating an unprocessed transaction record taking a main key as a user transaction ID; the temporarily stored transaction data are listed according to a first-in first-out mode, and the listed transaction data are summarized and then are sent to a host in batches; and acquiring a transaction processing result generated by the host according to the transaction data, updating the state of the corresponding unprocessed transaction record according to the transaction processing result, and simultaneously feeding back the state to the user. The invention can solve the system bottleneck problem caused by high transaction concurrency, reduce the pressure of network bandwidth, improve the data processing efficiency, improve the stability of the system when data are simultaneously burst and ensure the production safety.
Description
Technical Field
The invention relates to the technical field of computer data interaction, in particular to a batch data processing method, a front-end system, a host and a batch data processing system.
Background
With the rapid development of the internet, information systems of large-scale enterprises are more and more extensive, interaction among different application systems is more and more frequent, the data concurrency is more and more large, and load balancing among the application systems is always a problem that the enterprises pay attention to continuously. Meanwhile, the centralized outbreak of the transaction amount in certain periods may cause data transmission blockage among systems, and the production stability and the experience of customers are greatly influenced. For example, a stock fund buying and selling peak appears at the closing time of a stock exchange, ticket robbing and treasure washing network 11 robbing appears during the spring transportation of a railway ticketing system, the instantaneous data concurrency is huge often, so that the system is overloaded to operate, the phenomena of large transmission data blockage, network delay and data processing failure due to resource preemption appear, certain economic loss is caused to users, and meanwhile, a large negative effect is caused to enterprises. Therefore, how to effectively solve the efficiency of data transmission and data processing when the data volume is intensively exploded is becoming an urgent problem to be solved in the large-scale enterprise information system.
Disclosure of Invention
In order to solve the technical problems, the invention provides a batch data processing method, a front-end system, a host and a batch data processing system.
One aspect of the present invention provides a batch data processing method, including:
acquiring to-be-processed transaction data submitted by a user;
sorting the transaction data to be processed according to the processing type of the transaction data to be processed, and extracting the transaction data needing to be processed by a host;
temporarily storing the transaction data needing to be processed by the host computer, and simultaneously generating an unprocessed transaction record taking a main key as a user transaction ID;
the temporarily stored transaction data are listed according to a first-in first-out mode, and the listed transaction data are gathered and then sent to a host in batches;
and acquiring a transaction processing result generated by the host according to the transaction data, updating the state of the corresponding unprocessed transaction record according to the transaction processing result, and feeding back the state to the user.
In one embodiment, temporarily storing the transaction data to be processed by the host includes: and queuing the transaction data needing to be processed by the host according to a preset transaction priority, and for the transaction data with the queuing waiting time exceeding the preset waiting time, queuing the transaction data according to the sequence of the queuing waiting time from long to short.
In an embodiment, the batch data processing method further includes:
when the transaction processing result generated by the host cannot be acquired, generating a to-be-confirmed command containing a user transaction ID according to-be-processed transaction data re-submitted by a user and sending the to-be-confirmed command to the host;
and receiving a transaction processing result returned by the host based on the command to be confirmed, updating the state of the corresponding unprocessed transaction record according to the transaction processing result and feeding back the state to the user.
In an embodiment, the batch data processing method further includes:
periodically acquiring a transaction result file generated by the host according to the transaction data, wherein the transaction result file comprises at least one transaction processing result;
and checking and updating the unprocessed transaction records according to the transaction result file, and feeding back the suspicious transaction result caused by the communication failure with the host to the user.
In an embodiment, the batch data processing method further includes:
acquiring a transaction result file generated by the host according to the transaction data on a T +1 th day, wherein the transaction result file comprises at least one transaction processing result, and T is the transaction date of the transaction data;
inquiring whether a corresponding transaction processing result exists in the transaction result file according to the user transaction ID in the unprocessed transaction record;
if not, the state of the unprocessed transaction record is modified into the state of failure of the sending host and is fed back to the user;
if yes, the state of the unprocessed transaction record is modified into processed state, and the corresponding transaction processing result is sent to the user after the completion of the check.
The invention also provides a batch data processing method, which comprises the following steps:
receiving transaction data needing local processing, and temporarily storing the transaction data into a queue to be processed in batches;
classifying the transaction data in the queue to be batched according to the data characteristics of the transaction data;
and carrying out multi-batch concurrent processing and logic processing on the classified transaction data to obtain and output a transaction processing result, and marking the state of the transaction processing result as being returned.
In an embodiment, the batch data processing method further includes: and regularly scanning the transaction processing results which are not returned, summarizing and outputting the transaction processing results which are not returned when the accumulated number of the transaction processing results which are not returned reaches a preset number, and marking the state of the transaction processing results as returned.
In an embodiment, the batch data processing method further includes: and regularly scanning the transaction processing results which are not returned, summarizing and outputting the transaction processing results which are not returned after a preset waiting time upper limit is reached, and marking the state of the transaction processing results as returned.
In an embodiment, the batch data processing method further includes: and after the business hours of the current transaction day are finished, scanning transaction processing results corresponding to the processed transaction data of the current transaction day, generating a transaction result file and outputting the transaction result file.
In an embodiment, the batch data processing method further includes:
receiving a command to be confirmed containing a user transaction ID;
and inquiring a transaction processing file containing at least one transaction processing result according to the user transaction ID in the command to be confirmed, and outputting the transaction processing result related to the user transaction ID.
In another aspect, the present invention further provides a front-end system, including:
the transaction data acquisition unit is used for acquiring to-be-processed transaction data submitted by a user;
the transaction data sorting unit is used for sorting the transaction data to be processed according to the processing type of the transaction data to be processed and extracting the transaction data needing to be processed by the host;
the cache unit is used for temporarily storing the transaction data needing to be processed by the host computer and generating an unprocessed transaction record with a main key as a user transaction ID;
the transaction data transmission unit is used for dequeuing the temporarily stored transaction data in a first-in first-out mode, summarizing the dequeued transaction data and then transmitting the summarized transaction data to the host in batches;
and the processing result receiving unit is used for receiving a transaction processing result generated by the host according to the transaction data, updating the state of the corresponding unprocessed transaction record according to the transaction processing result and feeding back the state to the user.
In an embodiment, the cache unit includes a monitor process module, configured to queue the transaction data that needs to be processed by the host according to a preset transaction priority, and queue the transaction data in an order from long to short queue wait time for the transaction data with queue wait time exceeding the preset queue time.
In one embodiment, the front-end system further comprises:
the to-be-confirmed command generating unit is used for generating a to-be-confirmed command containing a user transaction ID according to-be-processed transaction data re-submitted by a user and sending the to-be-confirmed command to the host when a transaction processing result generated by the host cannot be acquired;
and the first feedback unit is used for receiving a transaction processing result returned by the host computer based on the command to be confirmed, updating the state of the corresponding unprocessed transaction record according to the transaction processing result and feeding back the state to the user.
In one embodiment, the front-end system further comprises:
the synchronization unit is used for periodically acquiring a transaction result file generated by the host according to the transaction data, wherein the transaction result file comprises at least one transaction processing result;
and the second feedback unit is used for checking and updating the unprocessed transaction record according to the transaction result file and feeding back the suspicious transaction result caused by the communication failure with the host to the user.
In one embodiment, the front-end system further comprises:
the file acquisition unit is used for acquiring a transaction result file generated by the host according to the transaction data on the T +1 th day, wherein the transaction result file comprises at least one transaction processing result, and T is the transaction date of the transaction data;
the query unit is used for querying whether the corresponding transaction processing result exists in the transaction result file according to the user transaction ID in the unprocessed transaction record;
and the third feedback unit is used for modifying the state of the unprocessed transaction record into failure in sending the host when the query result of the query unit is negative, modifying the state of the unprocessed transaction record into processed state when the query result of the query unit is positive, and sending the corresponding transaction processing result to the user after the completion of pairing.
One aspect of the present invention further provides a batch data processing host, including:
the data receiving unit is used for receiving transaction data needing local processing and temporarily storing the transaction data into a queue to be processed in batches;
the classification unit is used for classifying the transaction data in the queue to be batched according to the data characteristics of the transaction data;
and the data processing unit is used for carrying out multi-batch concurrent processing and logic processing on the classified transaction data to obtain and output a transaction processing result, and marking the state of the transaction processing result as returned.
In one embodiment, the host further comprises: and the processing result retransmission unit is used for periodically scanning the transaction processing results which are not retransmitted, summarizing and outputting the transaction processing results which are not retransmitted when the accumulated number of the transaction processing results which are not retransmitted reaches a preset number, and marking the state of the transaction processing results as retransmitted.
In an embodiment, the processing result returning unit is further configured to: and regularly scanning the transaction processing results which are not returned, summarizing and outputting the transaction processing results which are not returned after a preset waiting time upper limit is reached, and marking the state of the transaction processing results as returned.
In one embodiment, the host further comprises: and the file generating unit is used for scanning the transaction data processed on the same transaction day after the business hours on the same transaction day are finished, generating a transaction result file and outputting the transaction result file.
In one embodiment, the host further comprises:
a command to be confirmed receiving unit for receiving a command to be confirmed containing a user transaction ID;
and the searching unit is used for inquiring a transaction processing file containing at least one transaction processing result according to the transaction ID in the command to be confirmed and outputting the transaction processing result related to the user transaction ID.
The present invention additionally provides a batch data processing system, comprising: the front-end system and the host.
By utilizing the batch data processing method, the front-end system, the host and the batch data processing system provided by the invention, the system structure is reasonably organized and designed on the computer system, the convenience of the front-end system for receiving data is effectively combined with the high efficiency of the host for processing data, and the problem of system bottleneck caused by high transaction concurrency is solved. The front-end system receives the user data, sorts and extracts the user data, a queue storage and pipeline scheduling mechanism is adopted, the summarized data are sent to the host in batches for processing, and meanwhile, the host performs concurrent processing according to multiple batches, so that the interaction times of the front-end system and the host are reduced, the pressure of network bandwidth is reduced, and the data processing efficiency is improved; meanwhile, compared with a large amount of single-pen concurrent processing, the small-batch concurrent processing mode adopted by the host computer has higher efficiency and is not easy to cause data blockage, thereby greatly improving the stability of the system when data are concurrently burst and ensuring the production safety.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of a batch data processing method according to an embodiment of the invention;
FIG. 2 is a flow chart of a batch data processing method according to another embodiment of the present invention;
FIG. 3 is a flow chart of an interactive batch data processing method according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a front-end system according to an embodiment of the present invention;
FIG. 5 is a block diagram of a batch data processing host according to an embodiment of the present invention;
FIG. 6 is a block diagram of a batch data processing system according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
FIG. 1 is a flowchart illustrating a batch data processing method according to an embodiment of the present invention. As shown in fig. 1, the batch data processing method mainly includes the following steps:
and step S10, acquiring the transaction data to be processed submitted by the user.
And step S11, sorting the transaction data to be processed according to the processing type of the acquired transaction data to be processed, and extracting the transaction data needing to be processed by the host.
Step S12, temporarily storing the transaction data extracted in step S11 and needing host processing, and generating an unprocessed transaction record with the primary key as the user transaction ID.
And step S13, the temporarily stored transaction data are listed according to a first-in first-out mode, and the listed transaction data are gathered and then sent to the host in batches.
When the dequeued transaction data is summarized and then sent to the host in batches, the dequeued data can be packaged by using a general gateway protocol and sent to the host for processing. The method of packaging and sending a plurality of transaction data reduces the number of times of interaction with the host, reduces the pressure of network bandwidth and improves the processing efficiency of the transaction data.
And step S14, acquiring a transaction processing result generated by the host according to the transaction data, updating the state of the corresponding unprocessed transaction record according to the transaction processing result, and feeding back the state to the user.
Generally, after a transaction processing result generated by a host is obtained, data needs to be analyzed into locally recognizable types according to a gateway protocol, and the state of an unprocessed transaction record is updated to ensure the unification with data of the host.
The batch data processing method provided by the embodiment of the invention receives the transaction data to be processed submitted by the user, sorts and extracts the transaction data, and sends the transaction data to the host in batches for processing after the data is summarized by adopting a queue storage and pipeline scheduling mechanism, so that the interaction times with the host are reduced, the pressure of network bandwidth is reduced, and the data processing efficiency is improved.
In an embodiment, after the transaction data that needs to be processed by the host is extracted, when the transaction data that needs to be processed by the host is temporarily stored in step S12, the transaction data that needs to be processed by the host is usually queued according to a preset transaction priority, and for the transaction data whose queuing time exceeds the preset waiting time, the transaction data are queued according to the sequence from long queuing time to short queuing time. For example, transaction data with high priority is first queued, transaction data with low priority is queued for waiting, and for transaction data exceeding the preset maximum waiting time, the transaction data with the longest waiting time is first queued, and when dequeued, the queuing is performed according to the sequence of queue construction, i.e. the principle of first-in first-out.
By adopting the strategy of setting the transaction processing priority, more space is provided for processing batch transaction data, and the user experience is improved.
Generally, when a network is delayed or interrupted or even hardware fails to generate a transaction processing result which cannot be successfully acquired from a host end, a user is required to resubmit a transaction application containing transaction data to be processed, and by using the batch data processing method provided by the embodiment of the invention, a command to be confirmed containing a user transaction ID is generated according to the transaction data to be processed resubmitted by the user, and the command to be confirmed is sent to the host to actively initiate confirmation of the transaction processing result; and then, receiving a transaction processing result returned by the host computer based on the command to be confirmed, updating the state of the corresponding unprocessed transaction record according to the transaction processing result and feeding back the state to the user.
Besides receiving the transaction processing result generated by the host, the method provided by the embodiment of the invention can also be used for periodically acquiring the transaction result file (the transaction result file comprises at least one transaction processing result) generated by the host according to the transaction data so as to reduce the number of times of interaction with the host. And after the transaction result file generated by the host is obtained, checking and updating the unprocessed transaction record according to the transaction result file, and if an account suspected transaction result caused by communication failure with the host exists, feeding the account suspected transaction result back to the user.
In addition to the above active acquisition of the transaction result file from the host, the method of the embodiment of the invention may also be utilized to acquire the transaction result file generated by the host according to the transaction data on the T +1 th day, where T is the transaction date of the transaction data. Since the host computer generally processes a relatively large amount of transaction data on the day of the transaction day, a plurality of transaction processing results are correspondingly contained in the transaction result file. And then, inquiring whether a transaction processing result corresponding to the transaction data exists in the transaction result file or not according to the user transaction ID in the unprocessed transaction record. If the transaction data does not exist, the transaction data cannot be sent to the host side for processing, the state of the unprocessed transaction record corresponding to the transaction data is modified to be failed to send to the host, and the colleague feeds back the transaction data to the user. And if the transaction processing result corresponding to the transaction data exists in the transaction result file, modifying the state of the unprocessed transaction record corresponding to the transaction data into processed state, and sending the corresponding transaction processing result to the user after the completion of the check. The message may be sent to the user by a mail or a short message, which is not limited in the present invention.
The batch data processing method provided by the embodiment of the invention is used for checking the condition of the transaction doubtful account, and effectively solves the problems of data inconsistency between systems, data synchronization untimely and the like caused by interactive data loss.
FIG. 2 is a flowchart illustrating a batch data processing method according to another embodiment of the invention. As shown in fig. 2, the batch data processing is mainly completed at the host, and mainly includes the following steps:
and step S20, receiving transaction data needing local processing, and temporarily storing the transaction data to a pending queue.
Generally, after transaction data which needs to be processed locally is received, the transaction data needs to be analyzed into an identifiable data type according to a gateway protocol, and the data type is stored into a to-be-batched processing queue.
And step S21, classifying the transaction data in the pending processing queue according to the data characteristics of the transaction data. For example, the transaction data can be classified according to the transaction region and the transaction type, so that the processing time of the transaction data is shortened, and the processing efficiency is improved.
And step S22, performing multi-batch concurrent processing and logic processing on the classified transaction data to obtain and output a transaction processing result, and marking the state of the transaction processing result as being returned. The return transmission here means that the transaction processing result is output to one end of the sent transaction data.
After the transaction processing result is obtained, the transaction processing result is usually packaged and output according to a gateway protocol.
By using the batch data processing method provided by the embodiment of the invention, the transaction data is classified and then processed concurrently, so that the processing efficiency of the transaction data is effectively improved, and the system bottleneck of high concurrency is solved.
In an embodiment, in order to ensure that all transaction processing results are output, the batch data processing method provided in the embodiment of the present invention further includes periodically scanning transaction processing results that are not returned, summarizing and outputting the transaction processing results that are not returned when the accumulated number of the transaction processing results that are not returned reaches a preset number, and marking the state of the corresponding transaction processing result as returned.
In addition to the manner of outputting the transaction processing results according to the accumulated number of the transaction processing results, the embodiment of the invention can also output the transaction processing results at regular time according to a time manner, namely periodically scanning the transaction processing results which are not returned, summarizing and outputting the transaction processing results which are not returned after a preset waiting time upper limit is reached, and marking the state of the corresponding transaction processing results as returned.
Generally, the transaction data volume on the same day of the transaction is large, and in order to avoid reducing the efficiency of processing the transaction data, in the embodiment of the present invention, after the business on the same day of the transaction day is finished, the processed transaction data on the same day of the transaction day is scanned, and the transaction processing results corresponding to the transaction data are scanned to generate the transaction result file and then the transaction result file is output.
In an embodiment, the batch data processing method provided by the present invention further includes: receiving a command to be confirmed containing a user transaction ID; inquiring a transaction processing file according to the user transaction ID in the command to be confirmed, and outputting a transaction processing result if a transaction processing record corresponding to the user transaction ID exists in the transaction result file.
Compared with a large number of single-stroke concurrent processing, the small-batch concurrent processing mode adopted by the embodiment of the invention has higher efficiency and is not easy to cause data blockage, thereby greatly improving the stability of the system when data are concurrently burst and ensuring the production safety.
FIG. 3 is a flowchart of an interactive batch data processing method according to an embodiment of the present invention. As shown in fig. 3, the method is completed at both ends, after the transaction data to be processed submitted by the user is obtained (step S10), the transaction data to be processed is sorted according to the processing type of the transaction data to be processed, the transaction data to be processed by the host is extracted and temporarily stored (step S11), and meanwhile, an unprocessed transaction record with the main key as the user transaction ID is generated (step S12); and then, the temporarily stored transaction data is listed in a first-in first-out manner, and the listed transaction data is summarized and then sent to the host in batches (step S13). After receiving the transaction data to be processed locally, the host temporarily stores the transaction data into a pending queue (step S20); sorting according to the data characteristic stack of the temporarily stored transaction data (step S21); the classified transaction data is processed in multiple batches and logically, so as to obtain and output the transaction processing result, and the status of the transaction processing result is marked as being returned (step S22). And acquiring a transaction processing result generated by the host, updating the state of the corresponding unprocessed transaction record according to the transaction processing result, and feeding back the state to the user in time (step S14).
By utilizing the embodiment of the invention, the problem of system bottleneck caused by high concurrency of transaction data can be solved, the pressure of network bandwidth is reduced, and the data processing efficiency is improved.
Based on the same inventive concept as the batch data processing method shown in fig. 1, the embodiment of the present application further provides a front-end system, as described in the following embodiments. Because the principle of solving the problem by the front-end system is similar to that of the batch data processing method in fig. 1, the implementation of the front-end system can refer to the implementation of the batch data processing method in fig. 1, and repeated details are not repeated.
In another embodiment, the present invention further provides a front-end system, a schematic structural diagram of which is shown in fig. 4, the front-end system mainly includes: a transaction data acquisition unit 11, a transaction data sorting unit 12, a buffer unit 13, a transaction data transmission unit 14, and a processing result receiving unit 15.
The transaction data obtaining unit 11 is used for obtaining the transaction data to be processed submitted by the user. The transaction data sorting unit 12 is configured to sort the transaction data to be processed according to the processing type of the transaction data to be processed, and extract the transaction data that needs to be processed by the host. The buffer unit 13 is used for temporarily storing transaction data to be processed by the host computer and generating an unprocessed transaction record with the main key as the user transaction ID. The transaction data transmission unit 14 is configured to dequeue the temporarily stored transaction data in a first-in first-out manner, and send the dequeued transaction data to the host in batches after summarizing the dequeued transaction data. The processing result receiving unit 15 is configured to receive a transaction processing result generated by the host according to the transaction data, update a state of a corresponding unprocessed transaction record according to the transaction processing result, and feed back the state to the user.
In an embodiment, the cache unit 13 in the embodiment of the present invention includes a monitor process module, configured to queue the transaction data sorted by the transaction data sorting unit 12 and needing to be processed by the host according to a preset transaction priority, and for the transaction data with a queuing waiting time exceeding a preset waiting time, queue the transaction data according to a sequence from long to short queuing waiting time.
The front-end system in the embodiment of the present invention further typically includes a command to be confirmed generating unit 16 and a feedback unit 17. The to-be-confirmed command generating unit 16 is configured to generate a to-be-confirmed command including a user transaction ID according to the to-be-processed transaction data re-submitted by the user and send the to-be-confirmed command to the host when the transaction processing result generated by the host is not obtained. The feedback unit 17 is configured to receive a transaction processing result returned by the host computer based on the command to be confirmed, update a state of a corresponding unprocessed transaction record according to the transaction processing result, and simultaneously feed back the state to the user.
In an embodiment, the front-end system in the embodiment of the present invention further includes a synchronization unit 18 and a feedback unit 19. The synchronization unit 18 is configured to periodically obtain a transaction result file generated by the host according to the transaction data, where the transaction result file includes at least one transaction processing result. The feedback unit 19 is used for checking and updating the unprocessed transaction records according to the transaction result file, and feeding back the suspicious transaction result caused by the communication failure with the host to the user.
The front-end system in the embodiment of the invention further comprises: file acquisition unit 20, query unit 21, and feedback unit 22. The file acquiring unit 20 is configured to acquire a transaction result file generated by the host computer according to the transaction data on a T +1 th day, where T is a transaction date of the transaction data. The query unit 21 is configured to query whether a corresponding transaction processing result exists in the transaction result file according to the user transaction ID in the unprocessed transaction record. The feedback unit 22 is used for modifying the state of the unprocessed transaction record into failure of sending the host when the query result of the query unit 21 is negative; and when the query result of the query unit 21 is yes, modifying the state of the corresponding unprocessed transaction record into processed state, and sending the corresponding transaction processing result to the user after the completion of the check.
The front-end system provided by the embodiment of the invention is used for receiving the transaction data to be processed submitted by the user, sorting and extracting the transaction data, and sending the transaction data to the host in batches for processing after the data is summarized by adopting a queue storage and pipeline scheduling mechanism, so that the interaction times with the host are reduced, the pressure of network bandwidth is reduced, and the processing efficiency of the data is improved.
Based on the same inventive concept as the batch data processing method shown in fig. 2, the embodiment of the present application further provides a batch data processing host, as described in the following embodiments. Since the principle of solving the problem of the batch data processing host is similar to that of the batch data processing method in fig. 2, the batch data processing host can be implemented by referring to the implementation of the batch data processing method in fig. 2, and repeated parts are not described again.
The mechanism diagram of the batch data processing host provided by the embodiment of the invention is shown in fig. 5, and the batch data processing host mainly comprises: a data receiving unit 31, a classifying unit 32 and a data processing unit 33. The data receiving unit 31 is configured to receive transaction data that needs to be processed locally, and temporarily store the transaction data into a queue to be processed. The classifying unit 32 is configured to classify the transaction data in the pending queue according to the data characteristics of the transaction data. The data processing unit 33 is configured to perform multi-batch concurrent processing and logic processing on the classified transaction data to obtain and output a transaction processing result, that is, return the transaction processing result to the front-end system, and mark the state of the transaction processing result as returned.
By utilizing the method for classifying and then concurrently processing the transaction data by the host provided by the embodiment of the invention, the processing efficiency of the transaction data is effectively improved, and the system bottleneck of high concurrency is solved.
In an embodiment, the batch data processing host in the embodiment of the present invention further includes a processing result returning unit 34, configured to periodically scan transaction processing results that are not returned, and when an accumulated number of the transaction processing results that are not returned reaches a preset number, sum up the transaction processing results that are not returned and output the summarized transaction processing results, and mark a state of a corresponding transaction processing result as returned.
In addition to the manner of outputting the transaction processing result according to the accumulated amount of the transaction processing result, the processing result returning unit 34 may output the transaction processing result in a time manner at regular time, that is, periodically scan the transaction processing result that is not returned, when a preset waiting time upper limit is reached, summarize and output the transaction processing result that is not returned, and mark the status of the transaction processing result as returned.
In an embodiment, the batch data processing host further includes a file generating unit 35, configured to scan a transaction processing result of the transaction data processed on the same day of the transaction day after business hours on the same day of the transaction day are ended, and generate a transaction result file for output.
Generally, the batch data processing host provided by the embodiment of the present invention further includes a command receiving unit 36 to be confirmed and a lookup unit 37. The command to be confirmed receiving unit 36 is configured to receive a command to be confirmed including a user transaction ID. The searching unit 37 is configured to query a transaction processing file containing a plurality of transaction processing results according to the transaction ID in the command to be confirmed, and output a transaction processing result related to the user transaction ID in the command to be confirmed.
Compared with a large number of single-stroke concurrent processing, the small-batch concurrent processing mode adopted by the embodiment of the invention has higher efficiency and is not easy to cause data blockage, thereby greatly improving the stability of the system when data are concurrently burst and ensuring the production safety.
FIG. 6 is a block diagram of a batch data processing system according to an embodiment of the present invention. As shown in FIG. 6, the batch data processing system includes the front-end system shown in FIG. 4 and the batch data processing host shown in FIG. 5. The transaction data acquisition unit 11 is responsible for acquiring transaction data input by a user at a terminal, the front-end system starts the transaction data sorting unit 12 to sort the acquired transaction data, extracts the transaction data required to be processed by the host computer, stores the transaction data into the cache unit 13, and is responsible for being listed from the cache unit 13 and sending the transaction data to the host computer by the transaction data transmission unit 14. The data receiving unit 31 in the host receives the transaction data to be processed transmitted by the transaction data transmitting unit 14, completes the enqueue processing, and stores the enqueue processing into the queue to be batched. The classifying unit 32 classifies the transaction data according to the data type of the transaction data in the queue to be processed in batches, and the data processing unit 33 performs multi-batch concurrent processing and logic processing on the classified transaction data to obtain a transaction processing result and outputs the transaction processing result. The processing result returning unit 34 periodically scans the transaction processing results that are not returned, and sends the transaction processing results to the front-end system after the accumulated number of records to be sent or the waiting time reaches the upper limit, and the processing result receiving unit 15 completes the synchronization of the transaction data in the front-end system. The file generating unit 35 scans the transaction processing result of the transaction data after the business hours of the transaction day are over, generates a transaction result file according to the transaction processing result and outputs the transaction result file to the front-end system, and the file acquiring unit 20 synchronizes and ensures the unification of the data of the front-end system and the host computer again.
The monitoring process module in the cache unit 13 temporarily stores the transaction data to be processed by the host according to the transaction priority preset in the transaction listing policy parameter table, and simultaneously generates an unprocessed transaction record, wherein the main key is the user transaction ID. By adopting the strategy that the transaction processing priority passes through the parameter setting, more space is provided for flexible customization of the system, and the user experience is improved.
Before sending the transaction data temporarily stored in the buffer unit 13 to the host, the transaction data transmission unit 14 encapsulates and packages the listed transaction data by using a general gateway protocol, and sends the encapsulated and packaged transaction data to the host in batches, so as to reduce the number of interactions between the front-end system and the host, reduce the pressure of network bandwidth, and improve the processing efficiency of the batch data processing system.
After receiving the transaction data transmitted by the front-end system, the data receiving unit 31 analyzes the transaction data into a type that can be identified by the host according to a gateway protocol, and stores the type into a queue to be batched.
The sorting unit 32 analyzes the transaction data in the queue to be batched, sorts the transaction data according to data characteristics (such as transaction area and transaction type), and the host starts the data processing unit 33 to perform multi-batch concurrent processing on the sorted transaction data, and meanwhile, the processing result is packaged by a gateway protocol and then returned to the front-end system. By means of classification processing of transaction data, efficient processing capacity of the host is fully utilized, and overall processing efficiency of the batch data processing system is improved.
After receiving the transaction processing result returned by the host, the processing result receiving unit 15 analyzes the data into a type which can be identified by the front-end system according to the gateway protocol, updates the unprocessed transaction record in the front-end system according to the transaction processing result, and feeds back the unprocessed transaction record to the user.
When the network is delayed or interrupted or even the hardware is failed, the front-end system cannot normally receive the transaction processing result returned by the host, the user resubmits the transaction application containing the transaction data to the command to be confirmed generating unit 16, and actively initiates confirmation of the transaction processing result. The to-be-confirmed command generating unit 16 acquires the user transaction ID and sends the user transaction ID to the host, after the to-be-confirmed command receiving unit 36 receives the user transaction ID, the searching unit 37 queries the transaction processing result file and returns the transaction processing result related to the user transaction ID, and the feedback unit 17 updates the state of the corresponding unprocessed transaction record of the front-end system after confirmation is completed and simultaneously feeds back the state to the user.
The processing result returning unit 34 periodically scans the transaction data processed by the host, or the processing result of the transaction data, and performs timing or counting when the transaction processing result that is not returned is found, for example, the transaction processing result that is not returned reaches a certain number or a certain time (which may be adjusted and optimized according to a preset setting), and then automatically collects and sends the transaction processing result to the front-end system, and at the same time, identifies the status of the transaction processing result as returned.
The synchronization unit 18 is configured to periodically obtain a transaction combing result generated by the host, complete reconciliation and update of transaction data in the front-end system, and when a suspicious account transaction occurs due to a communication failure between the front-end system and the host, the feedback unit 19 feeds the result back to the user. The timeliness of data updating in the front-end system can be effectively improved through the real-time synchronization mode, and the customer experience is also improved.
After the business time of the transaction day is over, the file generating unit 35 at the host end scans the transaction data processed by the host and the transaction processing result of each transaction data on the same day, and generates a transaction result file for the front-end system to perform data synchronization processing.
The front-end system starts the file acquisition unit 20 on the T +1 th day to acquire the transaction result file generated by the host end, and the unprocessed transaction records in the front-end system are checked. The inquiring unit 21 inquires whether a corresponding transaction processing result exists in the transaction result file through the user transaction ID of the unprocessed transaction record, and if the corresponding transaction processing result does not exist, the feedback unit 22 is started to modify the state of the unprocessed transaction record into failure of sending the host; if the corresponding transaction processing result exists, the state of the unprocessed transaction record in the front-end system is modified into processed state, and the transaction processing result is fed back to the user in the form of short message or mail after the completion of the check.
In an embodiment, the file obtaining unit 20 may further generate transaction flow monitoring data according to the host file, so as to provide a reference for tuning of the batch data processing system in the embodiment of the present invention, and a processing mode of full-volume check ensures unification of data of the front-end system and the host through full-volume downloading of transaction results, so as to further reduce a risk of interaction between the front-end system and the host.
The batch data processing system provided by the embodiment of the invention receives user transaction data by using the front-end system, sorts and extracts the user transaction data, summarizes the transaction data by using the queue storage and flow scheduling mechanism and then sends the transaction data to the host, and the host completes the concurrent processing of multiple batches, thereby reducing the interaction times of the front-end system and the host, reducing the pressure of network bandwidth and improving the processing efficiency of the data.
The invention utilizes the flexibility of data access of a front-end system to sort and extract the transaction data, places the core processing of the data at the host end, utilizes the high-efficiency data processing speed of the host to improve the data processing efficiency and reduce the data processing time, and is provided with various synchronizing devices (including an active synchronizing device and a passive synchronizing device), thereby ensuring the timeliness and effectiveness of data synchronization and solving the problem of transaction loss of a high-concurrency data processing system. Furthermore, the invention greatly improves the overall performance capacity and the data processing efficiency of the system, reduces the resource consumption of the system, lowers the operation and maintenance cost of enterprises, improves the user experience and improves the overall economic benefit.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principle and the implementation mode of the invention are explained by applying specific embodiments in the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (8)
1. A batch data processing method, comprising:
acquiring to-be-processed transaction data submitted by a user;
sorting the transaction data to be processed according to the processing type of the transaction data to be processed, and extracting the transaction data needing to be processed by a host;
temporarily storing the transaction data needing to be processed by the host computer, and simultaneously generating an unprocessed transaction record taking a main key as a user transaction ID;
the temporarily stored transaction data are listed according to a first-in first-out mode, and the listed transaction data are gathered and then sent to a host in batches;
acquiring a transaction processing result generated by the host according to the transaction data, updating the state of a corresponding unprocessed transaction record according to the transaction processing result, and feeding back the state to a user;
the temporary storage of the transaction data to be processed by the host comprises the following steps: and queuing the transaction data needing to be processed by the host according to a preset transaction priority, and for the transaction data with the queuing waiting time exceeding the preset waiting time, queuing the transaction data according to the sequence of the queuing waiting time from long to short.
2. The batch data processing method of claim 1, further comprising:
when the transaction processing result generated by the host cannot be acquired, generating a to-be-confirmed command containing a user transaction ID according to-be-processed transaction data re-submitted by a user and sending the to-be-confirmed command to the host;
and receiving a transaction processing result returned by the host based on the command to be confirmed, updating the state of the corresponding unprocessed transaction record according to the transaction processing result and feeding back the state to the user.
3. The batch data processing method of claim 1, further comprising:
periodically acquiring a transaction result file generated by the host according to the transaction data, wherein the transaction result file comprises at least one transaction processing result;
and checking and updating the unprocessed transaction records according to the transaction result file, and feeding back the suspicious transaction result caused by the communication failure with the host to the user.
4. The batch data processing method of claim 1, further comprising:
acquiring a transaction result file generated by the host according to the transaction data on a T +1 th day, wherein the transaction result file comprises at least one transaction processing result, and T is the transaction date of the transaction data;
inquiring whether a corresponding transaction processing result exists in the transaction result file according to the user transaction ID in the unprocessed transaction record;
if not, the state of the unprocessed transaction record is modified into the state of failure of the sending host and is fed back to the user;
if yes, the state of the unprocessed transaction record is modified into processed state, and the corresponding transaction processing result is sent to the user after the completion of the check.
5. A front-end system, comprising:
the transaction data acquisition unit is used for acquiring to-be-processed transaction data submitted by a user;
the transaction data sorting unit is used for sorting the transaction data to be processed according to the processing type of the transaction data to be processed and extracting the transaction data needing to be processed by the host;
the cache unit is used for temporarily storing the transaction data needing to be processed by the host computer and generating an unprocessed transaction record with a main key as a user transaction ID;
the transaction data transmission unit is used for dequeuing the temporarily stored transaction data in a first-in first-out mode, summarizing the dequeued transaction data and then transmitting the summarized transaction data to the host in batches;
the processing result receiving unit is used for receiving a transaction processing result generated by the host according to the transaction data, updating the state of a corresponding unprocessed transaction record according to the transaction processing result and feeding back the state to a user;
the cache unit comprises a monitoring process module which is used for queuing the transaction data needing to be processed by the host according to a preset transaction priority, and for the transaction data with queuing waiting time exceeding the preset waiting time, the transaction data are listed according to the sequence of the queuing waiting time from long to short.
6. The front-end system of claim 5, further comprising:
the to-be-confirmed command generating unit is used for generating a to-be-confirmed command containing a user transaction ID according to-be-processed transaction data re-submitted by a user and sending the to-be-confirmed command to the host when a transaction processing result generated by the host cannot be acquired;
and the first feedback unit is used for receiving a transaction processing result returned by the host computer based on the command to be confirmed, updating the state of the corresponding unprocessed transaction record according to the transaction processing result and feeding back the state to the user.
7. The front-end system of claim 5, further comprising:
the synchronization unit is used for periodically acquiring a transaction result file generated by the host according to the transaction data, wherein the transaction result file comprises at least one transaction processing result;
and the second feedback unit is used for checking and updating the unprocessed transaction record according to the transaction result file and feeding back the suspicious transaction result caused by the communication failure with the host to the user.
8. The front-end system of claim 7, further comprising:
the file acquisition unit is used for acquiring a transaction result file generated by the host according to the transaction data on the T +1 th day, wherein the transaction result file comprises at least one transaction processing result, and T is the transaction date of the transaction data;
the query unit is used for querying whether the corresponding transaction processing result exists in the transaction result file according to the user transaction ID in the unprocessed transaction record;
and the third feedback unit is used for modifying the state of the unprocessed transaction record into failure in sending the host when the query result of the query unit is negative, modifying the state of the unprocessed transaction record into processed state when the query result of the query unit is positive, and sending the corresponding transaction processing result to the user after the completion of pairing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610754742.2A CN106375241B (en) | 2016-08-29 | 2016-08-29 | Batch data processing method, front-end system, host and batch data processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610754742.2A CN106375241B (en) | 2016-08-29 | 2016-08-29 | Batch data processing method, front-end system, host and batch data processing system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106375241A CN106375241A (en) | 2017-02-01 |
| CN106375241B true CN106375241B (en) | 2020-03-13 |
Family
ID=57900576
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610754742.2A Active CN106375241B (en) | 2016-08-29 | 2016-08-29 | Batch data processing method, front-end system, host and batch data processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106375241B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107273225A (en) * | 2017-06-16 | 2017-10-20 | 深圳易嘉恩科技有限公司 | High concurrent competition for orders function realizing method based on message queue |
| CN107809480A (en) * | 2017-10-25 | 2018-03-16 | 上海瀚银信息技术有限公司 | One kind transaction commutation system |
| CN107819696B (en) * | 2017-11-22 | 2021-09-28 | 中国银行股份有限公司 | Transaction flow control method and system |
| CN108830715B (en) * | 2018-05-30 | 2023-04-25 | 平安科技（深圳）有限公司 | Batch file part disc returning processing method and system |
| CN108876521A (en) * | 2018-06-04 | 2018-11-23 | 贵州数据宝网络科技有限公司 | A kind of high concurrent data transacting system and method |
| CN109102402A (en) * | 2018-07-26 | 2018-12-28 | 阿里巴巴集团控股有限公司 | A kind of monitoring method of transaction risk, device and system |
| CN109246077B (en) * | 2018-08-01 | 2021-06-29 | 广州唯品会信息科技有限公司 | Distributed concurrent transaction verification method, device and computer storage medium |
| CN109583859A (en) * | 2018-10-22 | 2019-04-05 | 中国平安人寿保险股份有限公司 | It trades in batches between multisystem task processing method and system |
| CN110866828A (en) * | 2019-10-15 | 2020-03-06 | 北京海益同展信息科技有限公司 | Method, apparatus, computer device and storage medium for processing blockchain data |
| CN112260952A (en) * | 2020-10-20 | 2021-01-22 | 四川天邑康和通信股份有限公司 | Wifi6 router rapid data access protection method |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH0446746A (en) * | 1990-06-11 | 1992-02-17 | Oki Electric Ind Co Ltd | Progress control system for batch processing |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7222223B2 (en) * | 2004-10-29 | 2007-05-22 | Pillar Data Systems, Inc. | Management of I/O operations in data storage systems |
| CN101017546A (en) * | 2007-03-12 | 2007-08-15 | 中国建设银行股份有限公司 | Method and device for categorical data batch processing |
| CN101697536B (en) * | 2009-10-16 | 2012-05-30 | 深圳市科陆电子科技股份有限公司 | Method for transmitting mass data in real time by penetrating positive physical isolation device |
| CN101719153A (en) * | 2009-12-09 | 2010-06-02 | 中国工商银行股份有限公司 | Device and method for carrying out transmission processing on massive data files |
| CN102929585B (en) * | 2012-09-25 | 2016-01-13 | 上海证券交易所 | A kind of batch processing method and system supporting the distributed data processing of many main frames |
| CN105760395A (en) * | 2014-12-18 | 2016-07-13 | 华为技术有限公司 | Data processing method, device and system |
-
2016
- 2016-08-29 CN CN201610754742.2A patent/CN106375241B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH0446746A (en) * | 1990-06-11 | 1992-02-17 | Oki Electric Ind Co Ltd | Progress control system for batch processing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106375241A (en) | 2017-02-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106375241B (en) | Batch data processing method, front-end system, host and batch data processing system | |
| CN108200544B (en) | Short message issuing method and short message platform | |
| CN106603598B (en) | Method and device for processing service request | |
| CN104601696B (en) | Service processing method, service calling system, device and system | |
| US10075549B2 (en) | Optimizer module in high load client/server systems | |
| US20160285812A1 (en) | Method of issuing messages of a message queue and a message issuing device | |
| US20130318178A1 (en) | Systems and Methods for Processing Emails | |
| CN106095585A (en) | Task requests processing method, device and enterprise information system | |
| US10673969B2 (en) | Scalable data feed system | |
| CN105610972A (en) | Clustered task dispatching system | |
| CN112162865A (en) | Server scheduling method and device and server | |
| WO2021057500A1 (en) | Message sending management method and device | |
| CN104834558A (en) | Method and system for processing data | |
| US20070094336A1 (en) | Asynchronous server synchronously storing persistent data batches | |
| WO2012037760A1 (en) | Method, server and system for improving alarm processing efficiency | |
| CN109039817B (en) | Information processing method, device, equipment and medium for flow monitoring | |
| CN102801737A (en) | Asynchronous network communication method and device | |
| CN105761039A (en) | Method for processing express delivery information big data | |
| CN109547162A (en) | Data communications method based on two sets of unidirectional boundaries | |
| CN107346270A (en) | Method and system based on the sets cardinal calculated in real time | |
| CN110928905A (en) | Data processing method and device | |
| CN115622906A (en) | Application log capturing system and method | |
| CN111131499A (en) | Concurrent and asynchronous task processing method and device thereof | |
| US7437733B2 (en) | System and method for using a mobile agent object to collect data | |
| CN113364857A (en) | Service data processing method and device and server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |