CN109492034A - Batch processing system, method, apparatus and storage medium - Google Patents
Batch processing system, method, apparatus and storage medium Download PDFInfo
- Publication number
- CN109492034A CN109492034A CN201811079727.8A CN201811079727A CN109492034A CN 109492034 A CN109492034 A CN 109492034A CN 201811079727 A CN201811079727 A CN 201811079727A CN 109492034 A CN109492034 A CN 109492034A
- Authority
- CN
- China
- Prior art keywords
- file
- processing
- detail
- module
- data detail
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/02—Banking, e.g. interest calculation or account maintenance
Abstract
The present invention provides a kind of batch processing system, the system comprises: file reception module, for being received and being verified to file；File warehousing module, for importing database to the file after the reception and verification；Detail processing module, for reading the data detail of the file from the database and carrying out online outgoing call processing to the data detail；And result-generation module, for arranging the processing result of the online outgoing call processing, and is counted and recorded according to the processing result.The present invention passes through batch processing mode, greatly promotion data-handling efficiency.
Description
Technical field
The present invention relates to data processing fields, in particular to a kind of batch processing system, method, apparatus and storage
Medium.
Background technique
In banking system, due to in the third party systems interactive process such as card tissue, usually with file
Mode interacts, and in bank's built-in system, there is also carry out data biography by file with front-end interface or other assemblies
It is defeated, the demand of single processing is then carried out, therefore produce the demand that batch turns online, i.e., by the file received according to record
On-line transaction is changed into one by one to be handled, therefore produces the demand to batch agency.
In banking system, there are a large amount of batch documents to turn online demand, need a set of unified mechanism into
Row service logic and Row control, and process performance is promoted under the premise of guaranteeing effect.
In existing treatment mechanism, batch processing is mainly carried out eventually in day by batch jobs, in this way,
The timeliness of the great limit service processing of meeting, and should not have a large amount of service logic in storing process, so the business of influence
Realization effect.
Summary of the invention
In order to solve the above technical problems, the present invention provides a kind of batch processing system, method, apparatus and storage medium,
It solves the problems, such as present batch operation processing mode poor in timeliness and traffic handing capacity is low.
According to a first aspect of the embodiments of the present invention, a kind of batch processing system is provided, the system comprises:
File reception module, for being received and being verified to file；
File warehousing module, for importing database to the file after the reception and verification；
Detail processing module, for read from the database file data detail and to the data detail into
The online outgoing call processing of row；And
Result-generation module, for arranging the processing result of the online outgoing call processing, and according to the processing result into
Row statistics and record.
According to a second aspect of the embodiments of the present invention, a kind of batch processing method is provided, which comprises
File reception module is received and is verified to file；
File warehousing module imports database to the file after the reception and verification；
Detail processing module reads the data detail of the file from the database and joins to the data detail
Machine outgoing call processing；And
Result-generation module arranges the processing result of the online outgoing call processing, and is counted according to the processing result
And record.
According to a third aspect of the embodiments of the present invention, a kind of computer readable storage medium, the computer storage are provided
Medium includes computer program, wherein the computer program makes described one when being executed by one or more computers
A or multiple computers perform the following operations:
The operation include the steps that it is any one of as above described in batch processing method included.
According to a fourth aspect of the embodiments of the present invention, a kind of batch-processed devices are provided, described device includes:
Memory is stored with computer-readable instruction；
Processor executes the computer-readable instruction to execute the step that batch processing method as described above is included
Suddenly.
Implement a kind of batch processing system, method, apparatus and storage medium provided in an embodiment of the present invention, has following excellent
Point: by one kind in batch processing mode, greatly promotion data-handling efficiency.
Detailed description of the invention
Fig. 1 is a kind of structural schematic diagram of batch processing system 1 of the embodiment of the present invention；
Fig. 2 is the structural schematic diagram of file reception module 100 described in system 1 described in the embodiment of the present invention；
Fig. 3 is the structural schematic diagram of detail processing module 200 described in system 1 described in the embodiment of the present invention；
Fig. 4 is a kind of flow chart of batch jobs scheduling processing method of the embodiment of the present invention.
Specific embodiment
To keep the purposes, technical schemes and advantages of the embodiment of the present invention clearer, below in conjunction with attached drawing to this hair
It is bright to be described in further detail.
Fig. 1 is a kind of structural schematic diagram of batch processing system 1 of the embodiment of the present invention, and referring to Fig. 1, the system 1 is wrapped
It includes:
File reception module 100, for being received and being verified to file；This module is mainly responsible for the reception and conjunction of file
Method verification, wherein there are two types of modes for the reception of file, and one is front channels to be sent to finger for file by file transfer component
Determine catalogue, and the file arrival notice Service Notification document framework for calling batch agent framework to provide is sent to；Second of mode be
File is directly sent to agreement catalogue by third party's component, and batch agent framework at the appointed time whether under scanning directory if depositing by file
There is i.e. default document and is being sent to.
File warehousing module 200, for importing database to the file after the reception and verification；Reach service in file
In, question paper legitimacy verifies finish, and can trigger file warehousing service, and file detail can be imported into data by file warehousing service
Subsequent concurrent processing is convenient in library.In file warehousing module, file detail legitimacy verifies and detail pre-treatment are mainly responsible for,
Data detail is handled convenient for subsequent logic.
Detail processing module 300, for the data detail of the file is read from the database and to the data it is bright
Carefully carry out online outgoing call processing；Wherein file detail is read and processing uses Stream Processing technology, can pass through concurrent and streaming
It calculates and greatly promotes treatment effeciency.And
Result-generation module 400, for arranging the processing result of the online outgoing call processing, and according to the processing result
It is counted and is recorded.The module is mainly responsible for the processing result of integration combing respective file, records for failure and successfully
It is counted and is recorded respectively, return result to external module.
Fig. 2 is the structural schematic diagram of file reception module 100 described in system 1 described in the embodiment of the present invention, referring to fig. 2,
The file reception module 100 includes:
First receiving unit 110 for the file to be sent to specified directory by file transfer component, and calls file
First receiving unit described in arrival notice Service Notification receives the file；And
Second receiving unit 120, for the file to be sent to the specified directory by third party's component, when specified
Between scan under the specified directory with the presence or absence of the file, then default the file if it exists and be sent to.
Fig. 3 is the structural schematic diagram of detail processing module 200 described in system 1 described in the embodiment of the present invention, referring to Fig. 3,
The detail processing module 200 includes:
Pretreatment unit 210 reads the data detail for pressing page from the database according to paging schemes principle,
And the data detail is transferred to streaming computing engine；
Business logic processing unit 220, for receiving the data detail of the streaming computing engine transmission, according to business need
It asks and the data detail is handled, and outgoing call corresponds to component and record processing result；And
Post-processing unit 230, if being judged as YES, triggers result for judging whether the data detail is disposed
Generation module.
In embodiments of the present invention, it is described to file carry out verification include: to the format of the file, name and
Md5 value carries out checking treatment.After file is sent to, batch agent framework can name the format of file, and md5 etc. is verified,
Confirm the legitimacy of file.
Fig. 4 is a kind of flow chart of batch processing method of the embodiment of the present invention, referring to fig. 4, which comprises
Step S1, file reception module are received and are verified to file；
Step S2, file warehousing module import database to the file after the reception and verification；
Step S3, detail processing module from the database read the data detail of the file and to the data details
Carry out online outgoing call processing；And
Step S4, result-generation module arrange the processing result of the online outgoing call processing, and according to the processing result
It is counted and is recorded.
In embodiments of the present invention, in step sl, it includes: that the file reception module, which receive to file,
The file is sent to specified directory by file transfer component by one receiving unit, and calls file arrival notice Service Notification institute
It states the first receiving unit and receives the file；And second receiving unit the file is sent to by the finger by third party's component
Determine catalogue, at the appointed time scans under the specified directory with the presence or absence of the file, then default the file if it exists and be sent to.
In embodiments of the present invention, in step s 2, the detail processing module reads data from the database
Detail and to the detail carry out online outgoing call processing include: pretreatment unit according to paging schemes principle from the database
The data detail is read by page, and the data detail is transferred to streaming computing engine；Business logic processing unit receives
The data detail of the streaming computing engine transmission, is handled the data detail according to business demand, and outgoing call is corresponding
Component and record processing result；And post-processing unit judges whether the data detail is disposed, if being judged as YES, touches
Send out result-generation module.
In embodiments of the present invention, it is described to file carry out verification include: to the format of the file, name and
Md5 value carries out checking treatment.
It should be noted that the operation of the batch processing method include the steps that include as described above with above-mentioned batch
The mode of operation for measuring processing system is identical, and details are not described herein for particular content.
In addition, the computer storage medium includes to calculate the present invention also provides a kind of computer readable storage medium
Machine program, the computer program execute one or more of computers such as
Lower operation: the operation includes the steps that batch processing method as described above is included that details are not described herein.It needs to illustrate
It is that the computer readable storage medium may include the multi storage such as ROM/RAM, magnetic disk, CD, disk, flash memory,
To store the storage medium that above-mentioned computer program can be performed.
In addition, the present invention also provides a kind of batch-processed devices, described device includes:
Memory is stored with computer-readable instruction；
Processor executes the computer-readable instruction to execute the step that batch processing method as described above is included
Suddenly.It should be noted that the batch-processed devices include: for example, intelligent terminal, mobile device, laptop, desk-top meter
A variety of devices such as calculation machine, server or the network equipment.
Through the above description of the embodiments, those skilled in the art can be understood that the present invention can be by
The mode of software combination hardware platform is realized.Based on this understanding, technical solution of the present invention makes tribute to background technique
That offers can be embodied in the form of software products in whole or in part, which can store is situated between in storage
In matter, such as ROM/RAM, magnetic disk, CD, including some instructions use is so that a computer equipment (can be individual calculus
Machine, server or network equipment etc.) execute method described in certain parts of each embodiment of the present invention or embodiment.
The above disclosure is only a preferred embodiment of the invention, cannot limit protection of the invention certainly with this
Range, therefore is still fallen within by right of the present invention and is wanted for equivalent variations made by above-described embodiment according to the introduction of the claims in the present invention
It asks in the range of being covered.
Claims (10)
1. a kind of batch processing system, which is characterized in that the system comprises:
File reception module, for being received and being verified to file；
File warehousing module, for importing database to the file after the reception and verification；
Detail processing module, for reading the data detail of the file from the database and joining to the data detail
Machine outgoing call processing；And
Result-generation module for arranging the processing result of the online outgoing call processing, and is united according to the processing result
Meter and record.
2. system as described in claim 1, which is characterized in that the file reception module includes:
First receiving unit, for the file to be sent to specified directory by file transfer component, and it is logical to call file to reach
Know that the first receiving unit described in Service Notification receives the file；And
Second receiving unit is at the appointed time scanned for the file to be sent to the specified directory by third party's component
It whether there is the file under the specified directory, then default the file if it exists and be sent to.
3. system as described in claim 1, which is characterized in that the detail processing module includes:
Pretreatment unit reads the data detail for pressing page from the database according to paging schemes principle, and by institute
It states data detail and is transferred to streaming computing engine；
Business logic processing unit, for receiving the data detail of the streaming computing engine transmission, according to business demand to institute
It states data detail to be handled, and outgoing call corresponds to component and record processing result；And
Post-processing unit, if being judged as YES, triggers result and generates mould for judging whether the data detail is disposed
Block.
4. the system as described in claim any one of 1-3, which is characterized in that it is described to file carry out verification include:
Checking treatment is carried out to the format of the file, name and md5 value.
5. a kind of batch processing method, which is characterized in that the described method includes:
File reception module is received and is verified to file；
File warehousing module imports database to the file after the reception and verification；
Detail processing module reads the data detail of the file from the database and carries out to the data detail online outer
Exhale processing；And
Result-generation module arranges the processing result of the online outgoing call processing, and is counted and remembered according to the processing result
Record.
6. method as claimed in claim 5, which is characterized in that the file reception module, which receive to file, includes:
The file is sent to specified directory by file transfer component by the first receiving unit, and calls file arrival notice service
First receiving unit is notified to receive the file；And
The file is sent to the specified directory by third party's component by the second receiving unit, at the appointed time scans the finger
Determine then to default the file if it exists with the presence or absence of the file under catalogue and be sent to.
7. method as claimed in claim 6, which is characterized in that the detail processing module reads data detail from the database
And online outgoing call processing is carried out to the detail and includes:
Pretreatment unit presses page according to paging schemes principle from the database and reads the data detail, and by the data
Detail is transferred to streaming computing engine；
Business logic processing unit receives the data detail of the streaming computing engine transmission, according to business demand to the data
Detail is handled, and outgoing call corresponds to component and record processing result；And
Post-processing unit judges whether the data detail is disposed, if being judged as YES, triggers result-generation module.
8. such as any one of claim 5-7 the method, which is characterized in that described to carry out verification to file and include:
Checking treatment is carried out to the format of the file, name and md5 value.
9. a kind of computer readable storage medium, the computer storage medium includes computer program, which is characterized in that institute
Stating computer program performs the following operations one or more of computers when being executed by one or more computers:
The operation includes the steps that the batch processing method as described in any one of claim 5-8 includes.
10. a kind of batch-processed devices, which is characterized in that described device includes:
Memory is stored with computer-readable instruction；
Processor executes the computer-readable instruction to execute the batch processing side as described in any one of claim 5-8
The step of method is included.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811079727.8A CN109492034A (en) | 2018-09-17 | 2018-09-17 | Batch processing system, method, apparatus and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811079727.8A CN109492034A (en) | 2018-09-17 | 2018-09-17 | Batch processing system, method, apparatus and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109492034A true CN109492034A (en) | 2019-03-19 |
Family
ID=65690467
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811079727.8A Pending CN109492034A (en) | 2018-09-17 | 2018-09-17 | Batch processing system, method, apparatus and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109492034A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110780856A (en) * | 2019-10-16 | 2020-02-11 | 南方电网科学研究院有限责任公司 | Electricity consumption data publishing platform based on micro-service |
| CN112433994A (en) * | 2020-11-24 | 2021-03-02 | 北京神州数字科技有限公司 | File processing method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101000562A (en) * | 2006-12-30 | 2007-07-18 | 中国建设银行股份有限公司 | Method and device for executing batch processing job |
| CN101051373A (en) * | 2006-04-07 | 2007-10-10 | 中国银联股份有限公司 | Method for processing batch data and method for batch transfer cash, entering account and out-of-account |
| CN102637214A (en) * | 2012-04-28 | 2012-08-15 | 中国工商银行股份有限公司 | Method and system for synchronizing general data among database services |
| CN103578032A (en) * | 2013-11-14 | 2014-02-12 | 中国银行股份有限公司 | Data processing system |
| CN108537528A (en) * | 2018-04-10 | 2018-09-14 | 平安科技（深圳）有限公司 | Method and system is paid out in batch documents audit |
-
2018
- 2018-09-17 CN CN201811079727.8A patent/CN109492034A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101051373A (en) * | 2006-04-07 | 2007-10-10 | 中国银联股份有限公司 | Method for processing batch data and method for batch transfer cash, entering account and out-of-account |
| CN101000562A (en) * | 2006-12-30 | 2007-07-18 | 中国建设银行股份有限公司 | Method and device for executing batch processing job |
| CN102637214A (en) * | 2012-04-28 | 2012-08-15 | 中国工商银行股份有限公司 | Method and system for synchronizing general data among database services |
| CN103578032A (en) * | 2013-11-14 | 2014-02-12 | 中国银行股份有限公司 | Data processing system |
| CN108537528A (en) * | 2018-04-10 | 2018-09-14 | 平安科技（深圳）有限公司 | Method and system is paid out in batch documents audit |
Non-Patent Citations (2)
| Title |
|---|
| 于洪: "银行前置入账系统的分析与设计", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 高尚: "从金融科技创新到金融模式变革——流式计算在信用卡实时大数据应用领域的研究", 《国际金融》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110780856A (en) * | 2019-10-16 | 2020-02-11 | 南方电网科学研究院有限责任公司 | Electricity consumption data publishing platform based on micro-service |
| CN110780856B (en) * | 2019-10-16 | 2023-10-20 | 南方电网科学研究院有限责任公司 | Electricity data release platform based on micro-service |
| CN112433994A (en) * | 2020-11-24 | 2021-03-02 | 北京神州数字科技有限公司 | File processing method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111160845B (en) | Service processing method and device | |
| CN111857888B (en) | Transaction processing method and device | |
| CN111382252B (en) | Method, device, equipment and medium for determining problem category based on user request | |
| CN104965690B (en) | Data processing method and device | |
| CN111078436A (en) | Data processing method, device, equipment and storage medium | |
| CN110033280B (en) | Payment anti-shake method and device | |
| CN109492034A (en) | Batch processing system, method, apparatus and storage medium | |
| CN112288577A (en) | Transaction processing method and device for distributed service, electronic equipment and medium | |
| CN111415146A (en) | Resource data processing method, device and equipment | |
| CN108961034A (en) | System and method, storage medium based on user behavior certification | |
| CN107451301B (en) | Processing method, device, equipment and storage medium for real-time delivery bill mail | |
| CN109144992A (en) | A kind of method and apparatus of data storage | |
| CN110336791A (en) | A kind of breakpoint data transmission method, device, equipment and computer storage medium | |
| CN112181817B (en) | Test method and test device for SOA architecture platform | |
| CN116049099A (en) | Data processing method, device, electronic equipment and computer readable medium | |
| CN113627934A (en) | Transaction data acquisition method and related equipment | |
| CN111415245A (en) | Account opening method and device | |
| JP5691427B2 (en) | Communication cost settlement system, apparatus, method and program | |
| CN114428723A (en) | Test system, system test method, related device and storage medium | |
| CN112615965B (en) | Communication number verification method and system and computing device | |
| CN112258310B (en) | Resource processing method, device and system | |
| CN113055401B (en) | Enterprise business authorization processing method and device | |
| CN112950380B (en) | Block chain-based transaction consistency processing method and device | |
| CN111401840B (en) | Method, apparatus, device and computer readable medium for generating guarantor information | |
| CN111639936B (en) | Transaction information acquisition method and device, electronic equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190319 |