CN114745442A - On-line batch processing system - Google Patents

On-line batch processing system Download PDF

Info

Publication number
CN114745442A
CN114745442A CN202210383280.3A CN202210383280A CN114745442A CN 114745442 A CN114745442 A CN 114745442A CN 202210383280 A CN202210383280 A CN 202210383280A CN 114745442 A CN114745442 A CN 114745442A
Authority
CN
China
Prior art keywords
file
batch
service
unit
service module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210383280.3A
Other languages
Chinese (zh)
Inventor
郭赫
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Bank of China Ltd
Original Assignee
Bank of China Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Bank of China Ltd filed Critical Bank of China Ltd
Priority to CN202210383280.3A priority Critical patent/CN114745442A/en
Publication of CN114745442A publication Critical patent/CN114745442A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/06Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1097Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The embodiment of the application provides an online batch processing system, relates to the cloud computing field, includes: the operation and maintenance front-end module, the access gateway module and the batch transfer union service module; the operation and maintenance front-end module is used for receiving a service calling instruction sent by a user and generating a service calling request to the access gateway module; the access gateway module is used for receiving a service calling request sent by the operation and maintenance front-end module and forwarding the service calling request to a corresponding service layer component according to a set routing rule, and the access gateway module is also used for receiving a request file pushed by a data bus and informing the batch forwarding united service module, so that the batch forwarding united service module obtains the request file and carries out transaction processing one by one, generates a disk returning file and returns the disk returning file to the data bus. The method and the device can improve the batch service processing efficiency.

Description

On-line batch processing system
Technical Field
The application relates to the field of cloud computing, in particular to an online batch processing system.
Background
In current systems, online batch processing is mainly used for processing online batch processing services initiated by tellers and having file input, such as service contents in aspects of salary generation, remittance and posting, and the like. The online batch processing adopts a synchronization mechanism to realize the processing of a batch of requests, the waiting time for front-end operators is longer, and the user experience is poorer; for complex services, the existing online transaction processing can realize transaction-level multiplexing, avoid redevelopment of a batch processing program which is clumsy, and maintain a set of service code logic. In the online batch design of each system, a set of mature and uniform mechanism for processing batches is not provided.
The requirement of the report class is a very important and common requirement in the banking system, so it is necessary to design a report generation mechanism with high availability, flexibility and agility to support the efficient operation of business personnel.
The current situation is as follows: in the existing system, most report generation functions are realized by timing batch generation at night. The method has quite large limitation, firstly, the generation can be carried out only in fixed time, and if the batch fails in the same day, the batch can be generated again every other day after the problem is solved; secondly, the operability is poor, and the front-end operator can only passively wait for the report to be generated and then download the report, but cannot know the progress of report generation or the reason of failure of report generation.
Therefore, the inventor provides an online batch processing system by virtue of experience and practice of related industries for many years, so as to overcome the defects of the prior art.
Disclosure of Invention
Aiming at the problems in the prior art, the application provides an online batch processing system which can improve the batch service processing efficiency.
In order to solve at least one of the above technical problems, the present application provides the following technical solutions:
in a first aspect, the present application provides an online batch processing system, comprising: the operation and maintenance front-end module, the access gateway module and the batch transfer union service module;
the operation and maintenance front-end module is used for receiving a service calling instruction sent by a user and generating a service calling request to the access gateway module;
the access gateway module is used for receiving a service calling request sent by the operation and maintenance front-end module and forwarding the service calling request to a corresponding service layer component according to a set routing rule, and the access gateway module is also used for receiving a request file pushed by a data bus and informing the batch-to-batch service module, so that the batch-to-batch service module obtains the request file and carries out transaction processing one by one, generates a disk file and returns the disk file to the data bus.
Further, the batch forwarding service module includes a file arrival notification unit, and the file arrival notification unit is configured to monitor a file arrival notification sent by the access gateway module.
Furthermore, the batch forwarding service module further comprises a file downloading unit, and the file downloading unit is used for downloading the request files stored in the access gateway module in batch by calling the file access component.
Further, the batch-to-batch service module further comprises a file detail loading unit, and the file detail loading unit is used for performing file decompression and transcoding operation on the downloaded request file, and loading the obtained request file detail into the database.
Further, the baton-to-coupon service module further includes a detail scanning unit to be sent, and the detail scanning unit to be sent is configured to read the request file details from the database item by item according to the priority of the request file.
Further, the batch-and-forward service module further comprises a packet packaging unit, and the packet packaging unit is used for packaging the read request file details and sending the request file details to a service system for service processing.
Further, the batched and forwarded service module further comprises a flow control unit, and the flow control unit is used for controlling the flow of communication between the batched and forwarded service module and the service system.
Further, the batched and forwarded service module further comprises a message unpacking unit, and the message unpacking unit is used for unpacking the service processing result message returned by the service system.
Further, the batch-to-batch service module further comprises a detail state updating unit, and the detail state updating unit is used for performing state updating on locally stored request file details according to the service processing result.
Further, the batch-to-batch service module further comprises a task progress scanning unit, and the task progress scanning unit is used for monitoring whether all the request file details complete business processing.
Further, the batch-to-batch service module further comprises a disk-returning file generating unit, and the disk-returning file generating unit is used for generating a disk-returning file according to each service processing result after monitoring that all the request file details have completed service processing.
Further, the batch relay service module further comprises a disk return file pushing unit, and the disk return file pushing unit is used for pushing the generated disk return file to the data bus.
Further, the batch forwarding service module further comprises an unknown processing unit, and the unknown processing unit is used for carrying out retransmission or correction processing on the unknown transaction according to preset parameters.
Further, the operation and maintenance front-end module comprises a management interface unit, and the management interface unit comprises various services selected by a user.
Furthermore, the system also comprises an operation and maintenance service module, wherein the operation and maintenance service module is used for carrying out aggregation query and task management operation.
According to the technical scheme, the online batch processing system supports various new and old line request files and various service calls through flexible parameter setting, sends the new line service system or the old line service system to complete transaction processing through the enterprise-level registration center, generates a disk return file, returns the disk return file to a requester through a data bus, and is independently deployed in a multi-active mode to provide services for users in a micro-service mode, so that the batch service processing efficiency can be improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic diagram of an overall configuration of an online batch processing system according to the present application;
fig. 2 is a schematic structural diagram of a batch federation service module according to the present application;
FIG. 3 is a schematic diagram of an exemplary embodiment of an online batch processing system;
FIG. 4 is a second schematic diagram of an online batch processing system according to the second embodiment of the present invention;
FIG. 5 is a diagram illustrating an exemplary operating environment of the system of the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be used. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In this application, the terms "upper", "lower", "left", "right", "front", "rear", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal", and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings. These terms are used primarily to better describe the present application and its embodiments, and are not used to limit the indicated devices, elements or components to a particular orientation or to be constructed and operated in a particular orientation.
Moreover, some of the above terms may be used to indicate other meanings besides the orientation or positional relationship, for example, the term "on" may also be used to indicate some kind of attachment or connection relationship in some cases. The specific meaning of these terms in this application will be understood by those of ordinary skill in the art as appropriate.
Furthermore, the terms "mounted," "disposed," "provided," "connected," and "sleeved" are to be construed broadly. For example, it may be a fixed connection, a removable connection, or a unitary construction; can be a mechanical connection, or an electrical connection; may be directly connected, or indirectly connected through intervening media, or may be in internal communication between two devices, elements or components. The specific meaning of the above terms in the present application can be understood by those of ordinary skill in the art as appropriate.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
Consider that most report generation functions in the prior art are realized by timed batch generation at night. The method has great limitations, namely, the generation can be carried out only in fixed time, and if the batch fails in the same day, the batch can be generated again every other day after the problem is solved; secondly, the operability is poor, and the front-end operator can only passively wait for the report to be generated and then download the report, but cannot know the progress of report generation or the reason of failure of report generation.
In order to improve the batch service processing efficiency, the present application provides an embodiment of an online batch processing system, and referring to fig. 3, in the embodiment, the online batch processing system specifically includes: the operation and maintenance front-end module, the access gateway module and the batch forwarding service module.
The operation and maintenance front-end module is used for receiving a service calling instruction sent by a user and generating a service calling request to the access gateway module.
Optionally, the operation and maintenance front-end module includes a management interface unit, where the management interface unit includes various services for the user to select, for example, operations for task list query, task detail query, task suspension, and task recovery and corresponding display interfaces can be provided,
the access gateway module is used for receiving a service calling request sent by the operation and maintenance front-end module and forwarding the service calling request to a corresponding service layer component according to a set routing rule, and the access gateway module is also used for receiving a request file pushed by a data bus and informing the batch-to-batch service module, so that the batch-to-batch service module obtains the request file and carries out transaction processing one by one, generates a disk file and returns the disk file to the data bus.
Optionally, the access gateway module may be configured to receive a service call request from the operation and maintenance front-end module or an upstream system/component, or receive a request file pushed by the data bus. If a service calling request is received, the calling is forwarded to a proper service group according to a routing rule, if a request file is received, a batch-to-batch service module is informed to process according to the routing rule, so that the batch-to-batch service module acquires the request file and carries out transaction processing one by one, a disk file is generated and returned to the data bus.
Optionally, the batch forwarding service module is configured to implement a core function of batch file forwarding online transaction, and after receiving the file arrival notification, the service acquires the request file from the access gateway through the FTP, loads details in the file into the database, reads the details from the database one by one according to parameters such as file priority and processing window time, packages and sends the details to the service system, receives a transaction return, and records a transaction result. And after all transactions are completed, generating a disk file and pushing the disk file to a data bus. For unknown transactions, retransmission or correction can be carried out according to set parameters;
as can be seen from the above description, according to the online batch processing system provided in the embodiment of the present application, multiple request files and various service calls of a new line and an old line are supported through flexible parameter settings, and sent to a new line business service system or an old line system through an enterprise-level registry to complete transaction processing, a rollback file is generated and returned to a requester through a data bus, and meanwhile, the rollback file is independently deployed in a multi-active manner to provide services to users in a micro-service manner, so that the batch business processing efficiency can be improved.
Referring to fig. 1 and fig. 2, as an alternative implementation manner, the baton and antitransmission service module includes a file arrival notification unit, where the file arrival notification unit is configured to monitor a file arrival notification sent by the access gateway module.
As an optional implementation manner, the batched and forwarded service module further includes a file downloading unit, where the file downloading unit is configured to download the request files stored in the access gateway module in batch by calling a file access component.
As an optional implementation manner, the batch-to-subconjection service module further includes a file detail loading unit, where the file detail loading unit is configured to perform file decompression and transcoding operations on the downloaded request file, and load the obtained request file detail into the database.
As an optional implementation manner, the baton and antitransmission service module further includes a detail scanning unit to be sent, where the detail scanning unit to be sent is configured to read the request file details from the database one by one according to the priority of the request file.
As an optional implementation manner, the batch relay service module further includes a packet packaging unit, where the packet packaging unit is configured to package the read request file details and send the package to a service system for service processing.
As an optional implementation manner, the baton and antithetical couplet service module further includes a flow control unit, and the flow control unit is configured to perform flow control on communication between the baton and antithetical couplet service module and the service system.
As an optional implementation manner, the baton-to-antithetical couplet service module further includes a packet unpacking unit, where the packet unpacking unit is configured to unpack a service processing result packet returned by the service system.
As an optional implementation manner, the batch federation service module further includes a detail state updating unit, and the detail state updating unit is configured to perform state updating on locally stored request file details according to the service processing result.
As an optional implementation manner, the batch-to-batch service module further includes a task progress scanning unit, and the task progress scanning unit is configured to monitor whether all the request file details have completed business processing.
As an optional implementation manner, the batch-to-batch service module further includes a return file generating unit, where the return file generating unit is configured to generate a return file according to each service processing result after it is monitored that all the request file details have completed service processing.
As an optional implementation manner, the batch relay service module further includes a disk return file pushing unit, where the disk return file pushing unit is configured to push the generated disk return file to the data bus.
As an optional implementation manner, the batch federation service module further includes an unknown processing unit, and the unknown processing unit is configured to perform retransmission or correction processing on an unknown transaction according to preset parameters.
As an optional implementation manner, the operation and maintenance front-end module includes a management interface unit, and the management interface unit includes various services for the user to select.
Referring to fig. 4, as an optional implementation manner, the system further includes an operation and maintenance service module, where the operation and maintenance service module is configured to perform aggregation query and task management operations.
Optionally, the operation and maintenance service module may be an operation and maintenance background, and is configured to implement operation and maintenance management functions, including aggregation query, task management, full system monitoring based on an opentracking tool provided by a TSF platform, and the like.
Referring to fig. 5, as an alternative embodiment, the operating environment architecture for the system of the present application may be:
according to the application double-living AA-A deployment scheme, online batch components deploy double-living environments in the same city at two centers and deploy backup environments in different places at machine rooms in different places. And the database of the machine room in the same city is copied to the machine room in different places in an asynchronous mode.
The two machine rooms in the same city share the registration center, the same application is deployed, and TDSQL of the same service group of the two machine rooms form a cluster.
As can be seen from the above, the present application supports two different processing modes according to different service scenarios: a batch-to-batch mode and a batch mode; concurrent processing of batch service data is supported, and the batch service processing speed is accelerated; and supporting batch continuous running. Batch continuous running is supported no matter whether the first business processing or the subsequent whole batch billing is carried out. Convenience is provided for fault processing, and flexible support for business process design is also increased; a unified task scheduling mechanism is provided. To ensure the stability, performance and maintainability of the system; a unified asynchronous processing mechanism. Task acceptance is separated from batch processing, and after the task acceptance, the task processing is realized through an asynchronous mechanism, so that the occupation of system resources is reduced; and controlling the batch processing time period. The batch processing time period can be controlled through parametric configuration, the batch processing flow is limited, and the influence on normal online service is reduced; the system supports management information configuration and is responsible for acquiring and registering names, job numbers and user names of background jobs, and information such as abnormal dates, abnormal times, abnormal teller numbers, abnormal organization numbers, abnormal transaction codes and the like, so that the requirements of service development and management are met; and a unified monitoring management and exception handling mechanism for batch tasks is provided, and a perfect monitoring function is provided for background operation.
The above description is only an exemplary embodiment of the present invention, and is not intended to limit the scope of the present invention. Any equivalent changes and modifications of the invention without departing from the spirit and principles of the invention should be considered within the scope of the invention.

Claims (15)

1. An online batch processing system, comprising: the operation and maintenance front-end module, the access gateway module and the batch forwarding service module are connected;
the operation and maintenance front-end module is used for receiving a service calling instruction sent by a user and generating a service calling request to the access gateway module;
the access gateway module is used for receiving a service calling request sent by the operation and maintenance front-end module and forwarding the service calling request to a corresponding service layer component according to a set routing rule, and the access gateway module is also used for receiving a request file pushed by a data bus and informing the batch forwarding united service module, so that the batch forwarding united service module obtains the request file and carries out transaction processing one by one, generates a disk returning file and returns the disk returning file to the data bus.
2. The online batch processing system of claim 1, wherein the batch federation service module includes a file arrival notification unit configured to monitor for file arrival notifications sent by the access gateway module.
3. The online batch processing system of claim 2, wherein the wholesale union service module further comprises a file downloading unit, and the file downloading unit is configured to download the requested files stored in the access gateway module in batches by calling a file access component.
4. The online batch processing system of claim 3, wherein the batch transcoding service module further comprises a file detail loading unit, and the file detail loading unit is configured to perform file decompression and transcoding operations on the downloaded request file, and load the obtained request file detail into the database.
5. The online batch processing system according to claim 4, wherein the batch federation service module further comprises a detail scan unit to be sent, and the detail scan unit to be sent is configured to read request file details from the database item by item according to the request file priority.
6. The online batch processing system according to claim 5, wherein the batch relay service module further comprises a packet packaging unit, and the packet packaging unit is configured to package the read request file details and send the packaged request file details to a business system for business processing.
7. The online batch processing system of claim 6, wherein the referral service module further comprises a flow control unit for controlling flow of communications between the referral service module and the business system.
8. The online batch processing system of claim 7, wherein the wholesale federation service module further comprises a packet unpacking unit, and the packet unpacking unit is configured to unpack a service processing result packet returned by the service system.
9. The online batch processing system of claim 8, wherein the batch federation service module further comprises a detail state updating unit, and the detail state updating unit is configured to perform state updating on locally stored request file details according to the business processing result.
10. The online batch processing system of claim 9, wherein the batch federation service module further comprises a task progress scanning unit, the task progress scanning unit being configured to monitor whether all request file details have completed business processing.
11. The online batch processing system of claim 10, wherein the batch relay service module further comprises a backlog file generating unit, and the backlog file generating unit is configured to generate a backlog file according to each business processing result after it is monitored that all the request file details have completed business processing.
12. The online batch processing system of claim 11, wherein the wholesale federation service module further comprises a disk-back file pushing unit, and the disk-back file pushing unit is configured to push the generated disk-back file to the data bus.
13. The online batch processing system of claim 12, wherein the batch federation service module further comprises an unknown processing unit configured to perform retransmission or correction processing on an unknown transaction according to preset parameters.
14. The online batch processing system of claim 1, wherein the operation and maintenance front-end module comprises a management interface unit including services for selection by a user.
15. The online batch processing system of claim 1, further comprising an operation and maintenance service module, the operation and maintenance service module configured to perform aggregate query and task management operations.
CN202210383280.3A 2022-04-13 2022-04-13 On-line batch processing system Pending CN114745442A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210383280.3A CN114745442A (en) 2022-04-13 2022-04-13 On-line batch processing system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210383280.3A CN114745442A (en) 2022-04-13 2022-04-13 On-line batch processing system

Publications (1)

Publication Number Publication Date
CN114745442A true CN114745442A (en) 2022-07-12

Family

ID=82282116

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210383280.3A Pending CN114745442A (en) 2022-04-13 2022-04-13 On-line batch processing system

Country Status (1)

Country Link
CN (1) CN114745442A (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104077362A (en) * 2014-06-09 2014-10-01 中国建设银行股份有限公司 Online mass data processing system and method
CN111984303A (en) * 2020-08-20 2020-11-24 中国建设银行股份有限公司 Transaction data processing method, device, equipment and storage medium
CN112118184A (en) * 2020-08-06 2020-12-22 北京健康之家科技有限公司 Gateway automatic routing method and device, storage medium and computer equipment
CN114189525A (en) * 2021-12-02 2022-03-15 建信金融科技有限责任公司 Service request method and device and electronic equipment

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104077362A (en) * 2014-06-09 2014-10-01 中国建设银行股份有限公司 Online mass data processing system and method
CN112118184A (en) * 2020-08-06 2020-12-22 北京健康之家科技有限公司 Gateway automatic routing method and device, storage medium and computer equipment
CN111984303A (en) * 2020-08-20 2020-11-24 中国建设银行股份有限公司 Transaction data processing method, device, equipment and storage medium
CN114189525A (en) * 2021-12-02 2022-03-15 建信金融科技有限责任公司 Service request method and device and electronic equipment

Similar Documents

Publication Publication Date Title
US9842042B2 (en) Datacenter management computing system
CN110658794B (en) Manufacturing execution system
CN104272292B (en) Internet resources for the service based on cloud are disposed
CN108270855A (en) A kind of method of platform of internet of things access device
US9998323B2 (en) Datacenter configuration management tool
US5802293A (en) Integrated plant environment utilizing an advanced program-to-program server enabling communications between programs running in different computing environments
CN105979009A (en) Method for automatically balancing increased load of cloud application container
US20020194207A1 (en) System and method for data synronization between remote devices
CN101160563A (en) Method and system for hosting and executing a component application
CN107357571A (en) Maintaining method, system and the component server of apparatus assembly program
CN110248355A (en) Code number management method, equipment, platform and system based on internet of things equipment eUICC card
CN112199353A (en) Data processing method and electric power customer service platform
KR20030064779A (en) Systems and Methods for Providing Centralized Management of Heterogeneous Distributed Enterprise Application Integration Objects
CN111597033A (en) Task scheduling method and device
JP2014179837A (en) Information processing device and operator allocation method
CN110691131A (en) Micro-service framework system based on intelligent mobile terminal
JP2001265576A (en) Program replacing system, distributed processing system and program replacing method
CN102572954A (en) Roaming liquidation service scheduling method, middleware and system
US20200034765A1 (en) Systems and methods for contextual actions using a map interface
CN101236510B (en) EJB cluster failure recovery processing method and system
CN114745442A (en) On-line batch processing system
CN110262903B (en) Middleware framework and method based on aviation reservation system
CN111429127A (en) Service management method and device applied to payment
CN116828035A (en) Data integration system based on cloud computing
CN106302125A (en) A kind of solicited message is responded method, Apparatus and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination