CN110134516A - Finance data processing method, device, equipment and computer readable storage medium - Google Patents
Finance data processing method, device, equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN110134516A CN110134516A CN201910411302.0A CN201910411302A CN110134516A CN 110134516 A CN110134516 A CN 110134516A CN 201910411302 A CN201910411302 A CN 201910411302A CN 110134516 A CN110134516 A CN 110134516A
- Authority
- CN
- China
- Prior art keywords
- data processing
- data
- finance
- processing request
- target account
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
Abstract
The invention discloses a kind of finance data processing method, device, equipment and computer readable storage mediums, this method comprises: obtaining the pending data of the target account if receiving the data processing request for target account；Based on the pending data, the quantity of calculate node is determined in distributed computing framework corresponding with the data processing request；The multiple tasks example generated according to the pending data is respectively allocated to multiple calculate nodes, and starts multiple calculate node parallel computations, processing result corresponding with the data processing request is obtained after calculating.The present invention improves accounting speed of the financial institution when handling magnanimity finance data, meets the high-timeliness demand that business end handles finance data.
Description
Technical field
The present invention relates to financial technology (Fintech) technical field more particularly to a kind of finance data processing methods, dress
It sets, equipment and computer readable storage medium.
Background technique
With the development of computer technology, more and more computer technology (such as artificial intelligence, block chain, cloud computing) quilts
It applies in financial field, traditional financial industry gradually changes to financial technology (Fintech)；In financial technology, exist big
The scene of the finance data processing of amount, for example, the purpose of financial institution is in order to realize calculation business fund cost or income, needs
It carries out internal finance transfer pricing (FTP, Funds Transfer Pricing), for assets operation, FTP price represents its money
Golden cost needs to pay FTP interest；For debt business, FTP represents its fund income, can therefrom obtain FTP interest receipts
Enter；Currently, financial institution is when handling magnanimity finance data, example in the case where finance data amount is more and more huger
When such as carrying out internal finance transfer pricing, exist calculate speed it is slow, be unable to satisfy the high timeliness that business end handles finance data
The problem of property demand.
Summary of the invention
The main purpose of the present invention is to provide a kind of finance data processing method, device, equipment and computer-readable deposit
Storage media, it is intended to solve in the case where finance data amount is more and more huger, financial institution is to magnanimity finance data
Accounting speed is slow when reason, is unable to satisfy the problem of high-timeliness demand that business end handles finance data.
To achieve the above object, the present invention provides a kind of finance data processing method, the finance data processing method packet
Include following steps:
If receiving the data processing request for target account, the pending data of the target account is obtained；
Based on the pending data, determines and calculate in distributed computing framework corresponding with the data processing request
The quantity of node；
The multiple tasks example generated according to the pending data is respectively allocated to multiple calculate nodes, and is opened
Multiple calculate node parallel computations are moved, processing result corresponding with the data processing request is obtained after calculating.
Optionally, if the data processing request received for target account, obtain the target account wait locate
Manage data the step of include:
If receiving the data processing request for target account, the corresponding Tool for Data Warehouse of the target account is obtained
Hive table；
Based on preset access mode, the corresponding Hadoop distributed file system HDFS file of the Hive table is accessed,
Obtain the pending data of the target account.
Optionally, described to be based on preset access mode, the corresponding HDFS file of the Hive table is accessed, the mesh is obtained
Mark account pending data the step of include:
According to the corresponding HDFS file of the Hive table, the quantity of access process corresponding with the HDFS file is determined；
Based on several determining described access processes, access the HDFS file with obtain the target account wait locate
Manage data.
Optionally, described to be based on the pending data, in distributed computing frame corresponding with the data processing request
The step of quantity of determining calculate node, includes: in frame
Obtain the target data amount of the pending data；
According to the type of the data processing request, in distributed computing framework corresponding with the data processing request
The calculating data-quantity threshold of each calculate node is set；
According to the target data amount and the calculating data-quantity threshold, determine corresponding with the pending data described
The quantity of calculate node.
Optionally, described that the multiple tasks example generated according to the pending data is respectively allocated to multiple meters
Operator node, and start multiple calculate node parallel computations, processing corresponding with the data processing request is obtained after calculating
As a result the step of includes:
Multiple tasks example is generated according to the pending data, and the multiple task instances are respectively allocated to multiple
The calculate node；
Based on the calculate node, rule configuration ginseng corresponding with the data processing request is transferred from memory database
Number；The rule configuration parameter is that the memory database is directed by disk database；
The task instances and the regular configuration parameter based on the calculate node, start multiple calculate nodes
Parallel computation obtains processing result corresponding with the data processing request after calculating.
Optionally, described that the multiple tasks example generated according to the pending data is respectively allocated to multiple meters
Operator node, and start multiple calculate node parallel computations, processing corresponding with the data processing request is obtained after calculating
As a result after the step of further include:
The processing result is saved into the HDFS file, and updates the Hive table, so that updated Hive
Table is corresponding with including the HDFS file of the processing result.
Optionally, described to save the processing result into the HDFS file, and the Hive table is updated, so that more
Hive table after new with including the corresponding step of the HDFS file of the processing result after further include:
Based on preset report tool and the updated Hive table, generate at data corresponding with the target account
Manage analytical statement.
In addition, to achieve the above object, the present invention also proposes a kind of finance data processing unit, the finance data processing
Device includes:
Obtain module, if for receiving the data processing request for target account, obtain the target account to
Handle data；
Determining module, for being based on the pending data, in distributed computing corresponding with the data processing request
The quantity of calculate node is determined in frame；
Processing module, it is multiple described for the multiple tasks example generated according to the pending data to be respectively allocated to
Calculate node, and start multiple calculate node parallel computations, place corresponding with the data processing request is obtained after calculating
Manage result.
Optionally, the acquisition module includes:
First acquisition unit, if obtaining the target account for receiving the data processing request for target account
Corresponding Tool for Data Warehouse Hive table；
It is distributed to access the corresponding Hadoop of the Hive table for being based on preset access mode for second acquisition unit
File system HDFS file, obtains the pending data of the target account.
Optionally, the second acquisition unit includes:
Subelement is determined, for according to the corresponding HDFS file of the Hive table, determination to be corresponding with the HDFS file
The quantity of access process；
Subelement is accessed, for accessing the HDFS file to obtain based on several determining described access processes
State the pending data of target account.
Optionally, the determining module includes:
Third acquiring unit, for obtaining the target data amount of the pending data；
Setting unit, for the type according to the data processing request, at corresponding with the data processing request point
The calculating data-quantity threshold of each calculate node is set in cloth Computational frame；
Determination unit, for according to the target data amount and the calculating data-quantity threshold, it is determining with it is described to be processed
The quantity of the corresponding calculate node of data.
Optionally, the processing module includes:
Allocation unit, for generating multiple tasks example according to the pending data, and by the multiple task instances
It is respectively allocated to multiple calculate nodes；
Call unit is transferred and the data processing request pair for being based on the calculate node from memory database
The regular configuration parameter answered；The rule configuration parameter is that the memory database is directed by disk database；
Processing unit, for based on the calculate node the task instances and the regular configuration parameter, start more
A calculate node parallel computation, obtains processing result corresponding with the data processing request after calculating.
Optionally, described device further include:
Update module for saving the processing result into the HDFS file, and updates the Hive table, so that
Updated Hive table is corresponding with including the HDFS file of the processing result.
Optionally, described device further include:
Reports module generates and the target account for being based on preset report tool and the updated Hive table
The corresponding Data Management Analysis report in family.
In addition, to achieve the above object, the present invention also proposes that a kind of finance data processing equipment, the equipment include: to deposit
Reservoir, processor and it is stored in the finance data processing routine that can be run on the memory and on the processor, it is described
The step of as above described in any item finance data processing methods are realized when finance data processing routine is executed by the processor.
In addition, to achieve the above object, the present invention also proposes a kind of computer readable storage medium, described computer-readable
It is stored with finance data processing routine on storage medium, realizes when the finance data processing routine is executed by processor and such as takes up an official post
The step of finance data processing method described in one.
If the present invention receives the data processing request for target account, the number to be processed of the target account is obtained
According to；Based on the pending data, calculate node is determined in distributed computing framework corresponding with the data processing request
Quantity；The multiple tasks example generated according to the pending data is respectively allocated to multiple calculate nodes, and is opened
Multiple calculate node parallel computations are moved, processing result corresponding with the data processing request is obtained after calculating；It is right as a result,
In the pending data of magnanimity, only it need to increase corresponding calculate node in distributed computing framework, and multiple calculate nodes are simultaneously
Row calculates, to greatly improve accounting speed of the financial institution when handling magnanimity finance data, meets business end to gold
Melt the high-timeliness demand of data processing.
Detailed description of the invention
Fig. 1 is the structural schematic diagram for the hardware running environment that the embodiment of the present invention is related to；
Fig. 2 is the flow diagram of finance data processing method first embodiment of the present invention；
Fig. 3 is the flow diagram of finance data processing method second embodiment of the present invention；
Fig. 4 is the flow diagram of finance data processing method 3rd embodiment of the present invention；
Fig. 5 is the flow diagram of finance data processing method fourth embodiment of the present invention；
Fig. 6 is the flow diagram of the 5th embodiment of finance data processing method of the present invention.
The embodiments will be further described with reference to the accompanying drawings for the realization, the function and the advantages of the object of the present invention.
Specific embodiment
It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not intended to limit the present invention.
As shown in Figure 1, Fig. 1 is the structural schematic diagram for the hardware running environment that the embodiment of the present invention is related to.
It should be noted that Fig. 1 can be the structural schematic diagram of the hardware running environment of finance data processing equipment.This hair
Bright embodiment finance data processing equipment can be PC, the terminal devices such as portable computer.
As shown in Figure 1, the finance data processing equipment may include: processor 1001, such as CPU, network interface 1004,
User interface 1003, memory 1005, communication bus 1002.Wherein, communication bus 1002 is for realizing between these components
Connection communication.User interface 1003 may include display screen (Display), input unit such as keyboard (Keyboard), optional
User interface 1003 can also include standard wireline interface and wireless interface.Network interface 1004 optionally may include standard
Wireline interface, wireless interface (such as WI-FI interface).Memory 1005 can be high speed RAM memory, be also possible to stable
Memory (non-volatile memory), such as magnetic disk storage.Memory 1005 optionally can also be independently of aforementioned
The storage device of processor 1001.
It will be understood by those skilled in the art that finance data processing equipment structure shown in Fig. 1 is not constituted to finance
The restriction of data processing equipment may include perhaps combining certain components or different than illustrating more or fewer components
Component layout.
As shown in Figure 1, as may include operating system, net in a kind of memory 1005 of computer readable storage medium
Network communication module, Subscriber Interface Module SIM and finance data processing routine.Wherein, operating system is to manage and control finance data
The program of processing equipment hardware and software resource supports the operation of finance data processing routine and other softwares or program.
In finance data processing equipment shown in Fig. 1, user interface 1003 is mainly used for carrying out data with each terminal
Communication；Network interface 1004 is mainly used for connecting background server, carries out data communication with background server；And processor 1001
It can be used for calling the finance data processing routine stored in memory 1005, and execute following operation:
If receiving the data processing request for target account, the pending data of the target account is obtained；
Based on the pending data, determines and calculate in distributed computing framework corresponding with the data processing request
The quantity of node；
The multiple tasks example generated according to the pending data is respectively allocated to multiple calculate nodes, and is opened
Multiple calculate node parallel computations are moved, processing result corresponding with the data processing request is obtained after calculating.
Further, processor 1001 can be also used for calling the finance data processing routine stored in memory 1005,
And execute following steps:
If receiving the data processing request for target account, the corresponding Tool for Data Warehouse of the target account is obtained
Hive table；
Based on preset access mode, the corresponding Hadoop distributed file system HDFS file of the Hive table is accessed,
Obtain the pending data of the target account.
Further, processor 1001 can be also used for calling the finance data processing routine stored in memory 1005,
And execute following steps:
According to the corresponding HDFS file of the Hive table, the quantity of access process corresponding with the HDFS file is determined；
Based on several determining described access processes, access the HDFS file with obtain the target account wait locate
Manage data.
Further, processor 1001 can be also used for calling the finance data processing routine stored in memory 1005,
And execute following steps:
Obtain the target data amount of the pending data；
According to the type of the data processing request, in distributed computing framework corresponding with the data processing request
The calculating data-quantity threshold of each calculate node is set；
According to the target data amount and the calculating data-quantity threshold, determine corresponding with the pending data described
The quantity of calculate node.
Further, processor 1001 can be also used for calling the finance data processing routine stored in memory 1005,
And execute following steps:
Multiple tasks example is generated according to the pending data, and the multiple task instances are respectively allocated to multiple
The calculate node；
Based on the calculate node, rule configuration ginseng corresponding with the data processing request is transferred from memory database
Number；The rule configuration parameter is that the memory database is directed by disk database；
The task instances and the regular configuration parameter based on the calculate node, start multiple calculate nodes
Parallel computation obtains processing result corresponding with the data processing request after calculating.
Further, processor 1001 can be also used for calling the finance data processing routine stored in memory 1005,
And execute following steps:
The processing result is saved into the HDFS file, and updates the Hive table, so that updated Hive
Table is corresponding with including the HDFS file of the processing result.
Further, processor 1001 can be also used for calling the finance data processing routine stored in memory 1005,
And execute following steps:
Based on preset report tool and the updated Hive table, generate at data corresponding with the target account
Manage analytical statement.
Based on above-mentioned structure, each embodiment of finance data processing method of the present invention is proposed.
It is the flow diagram of finance data processing method first embodiment of the present invention referring to Fig. 2, Fig. 2.
The embodiment of the invention provides the embodiments of finance data processing method, it should be noted that although in flow chart
In show logical order, but in some cases, shown or described step can be executed with the sequence for being different from herein
Suddenly.
The present embodiment finance data processing method includes:
Step S100 obtains the to be processed of the target account if receiving the data processing request for target account
Data；
In financial technology, there are the scenes of a large amount of finance data processing；Currently, more and more huger in finance data amount
Trend under, when the finance data to magnanimity is handled, generally existing accounting speed is slow for financial institution, is unable to satisfy business
The case where high-timeliness demand that end handles finance data.
For example, the purpose of financial institution is in order to realize calculation business fund cost or income, needs to carry out internal finance and turns
Price (FTP, Funds Transfer Pricing) is moved, for assets operation, FTP price represents its fund cost, needs to prop up
Pay FTP interest；For debt business, FTP represents its fund income, can therefrom obtain FTP interest income；In the prior art,
Fund transfer pricing method is using traditional oracle database, and account detailed data and rule configuration information are stored in
In oracle database, the pricing engine write when calculating FTP price using c++, TPS (Transaction Per Second,
Number of transactions per second) it can only achieve 2000, existing above-mentioned pricing method can have performance bottleneck when handling mass data, cannot
The requirement of finishing service end T+1 sunrise report.
In the present embodiment, if receiving the data processing request for target account, obtain the target account wait locate
Manage data；Specifically, if the data processing request for target account of upstream business system transmission is received, as a kind of reality
Apply mode, which can be the account detailed data that upstream business system gets out target account, i.e., it is described to
After handling data, the signal of the data preparation completion of sending；Further, the pending data of the present embodiment target account is to deposit
In HBase database, HBase is built upon the distributed data towards column on Hadoop distributed file system for storage
Library, it provides random real-time read and write access to data, the present embodiment financial institution upon receipt of the signal, from HBase
Obtain the account detailed data of the target account.
Step S200 is based on the pending data, in distributed computing framework corresponding with the data processing request
The quantity of middle determining calculate node；
Based on the pending data, determines and calculate in distributed computing framework corresponding with the data processing request
The quantity of node；Specifically, financial institution carries out data prediction, generation and data processing to the account detailed data got
Input required for corresponding computing engines is requested, for example, when data processing request is that FTP fixes a price, FTP price computing engines institute
The input data needed is the account of target account, subject, product, mechanism, currency type, remaining sum, value date, the Expiration Date, original
Time limit, reset valence frequency, last time resets valence day, next time resets valence day etc.；In distributed computing framework, according to data processing
Request the complexity of corresponding calculating logic (computing engines) that the max-thresholds of each calculate node processing data are set, it can be with
Understand, if calculating logic is complicated, such as FTP pricing logic is complex, and each target account is directed to when calculating will be into
The a series of logic judgment of row, rule match, rule calculating etc., if being assigned to the number of the pending data of each calculate node
Larger according to measuring, then the calculation procedure of the calculate node will last long and cannot all complete, and cause performance issue；As a kind of reality
Mode is applied, the calculating number of each calculate node is rationally arranged according to the complexity of current calculating logic for the present embodiment financial institution
According to amount, the calculation procedure of each calculate node is mutually indepedent under distributed computing framework, larger in the data volume of pending data
In the case where, then the quantity of calculate node is accordingly increased, the arithmetic speed of each calculate node is ensured with this.
The multiple tasks example generated according to the pending data is respectively allocated to multiple calculating by step S300
Node, and start multiple calculate node parallel computations, processing knot corresponding with the data processing request is obtained after calculating
Fruit.
Calculate node calls preset calculating logic algorithm corresponding with data processing request, and to being assigned to current calculating
The pending data of node is calculated, and under distributed computing framework, multiple calculate nodes start simultaneously, is concurrently counted
It calculates, processing result corresponding with the data processing request is obtained after calculating.
If the present embodiment receives the data processing request for target account, the number to be processed of the target account is obtained
According to；Based on the pending data, calculate node is determined in distributed computing framework corresponding with the data processing request
Quantity；The multiple tasks example generated according to the pending data is respectively allocated to multiple calculate nodes, and is opened
Multiple calculate node parallel computations are moved, processing result corresponding with the data processing request is obtained after calculating；It is right as a result,
In the pending data of magnanimity, only it need to increase corresponding calculate node in distributed computing framework, and multiple calculate nodes are simultaneously
Row calculates, and greatly improves accounting speed of the financial institution when handling magnanimity finance data, meets business end to financial number
According to the high-timeliness demand of processing.
Further, finance data processing method second embodiment of the present invention is proposed.
It is the flow diagram of finance data processing method second embodiment of the present invention referring to Fig. 3, Fig. 3, is based on above-mentioned gold
Melt data processing method first embodiment, in the present embodiment, step S100, if the data processing received for target account is asked
The step of asking, obtaining the pending data of the target account include:
Step S110 obtains the corresponding number of the target account if receiving the data processing request for target account
According to warehouse tool Hive table；
It is different from above-mentioned first embodiment, in the present embodiment, the pending data of target account is in the form of Hive table
It is stored in corresponding distributed file system, Hive is a Tool for Data Warehouse based on Hadoop, can be by structure
The data file of change is mapped as a database table, and provides simple sql query function, sql sentence can be converted to
MapReduce task is run；In the present embodiment, upstream business system gets out the account detailed data of target account,
After the i.e. described pending data, account detailed data is stored in corresponding HDFS file in the form of Hive table, and is issued
The data processing request for the target account, financial institution after receiving the data processing request for target account,
The corresponding Hive table of account detailed data is obtained from business system end.
Step S120 is based on preset access mode, accesses the corresponding Hadoop distributed file system of the Hive table
HDFS file obtains the pending data of the target account；
The corresponding HDFS file of the Hive table is accessed to obtain the pending data of the target account, specifically,
In this example, as an implementation, step S120 is based on preset access mode, it is corresponding to access the Hive table
HDFS file, the step of obtaining the pending data of the target account include:
Step a determines access process corresponding with the HDFS file according to the corresponding HDFS file of the Hive table
Quantity；
Step b accesses the HDFS file based on several determining described access processes to obtain the target account
Pending data.
Specifically, multiple access processes are arranged in the present embodiment, and control the maximum of each access process processing data file
Threshold value increases concurrent access process correspondingly then when the data volume of the corresponding HDFS file of target account is larger to mention
Rise the speed of data processing, each access process is obtained from HDFS file based on the data processing threshold value of itself it is a part of to
Data are handled, and pending data is further pre-processed, for example, when data processing request is that FTP fixes a price, financial machine
The pending data that structure is obtained from HDFS file is the account of target account, subject, product, mechanism, currency type, remaining sum, carries interest
Day, the Expiration Date, original maturity, reset valence frequency, last time resets valence day, next time resets the data such as valence day, then these data are spelled
Be connected into: account information, price result of upper day, input of the repayment schedule three parts as next step FTP price computing engines calculate
Engine can be what the udf function based on Hive was write, and access process setting maximum threshold can be to avoid individual access process
Performance issue caused by the overabundance of data of single treatment.
If the present embodiment receives the data processing request for target account, the corresponding data of the target account are obtained
Warehouse tool Hive table determines access process corresponding with the HDFS file according to the corresponding HDFS file of the Hive table
Quantity, based on several determining described access processes, access the HDFS file with obtain the target account wait locate
Data are managed, the pending data is based on, determines and calculates in distributed computing framework corresponding with the data processing request
The multiple tasks example generated according to the pending data is respectively allocated to multiple calculate nodes by the quantity of node,
And start multiple calculate node parallel computations, processing result corresponding with the data processing request is obtained after calculating；By
This, writes computing engines in conjunction with the udf function of Hive, can be by extending transversely when the data volume of pending data becomes larger
Mode increases access process, while increasing calculate node, i.e. computing resource (cpu+ memory), the concurrent processing of Lai Zengqiang system
Can, in FTP price application scenarios, the present embodiment can support the daily FTP calculation of price of hundred million grades or higher amount level account, T+
Producible T daily sheet is checked and is analyzed for related personnel before working on the 1.
Further, finance data processing method 3rd embodiment of the present invention is proposed.
It is the flow diagram of finance data processing method 3rd embodiment of the present invention referring to Fig. 4, Fig. 4, is based on above-mentioned gold
Melt data processing method first embodiment, in the present embodiment, step S200 is based on the pending data, with the data
The step of quantity of determining calculate node, includes: in the corresponding distributed computing framework of processing request
Step S210 obtains the target data amount of the pending data；
Step S220, according to the type of the data processing request, in distribution corresponding with the data processing request
The calculating data-quantity threshold of each calculate node is set in Computational frame；
Step S230, according to the target data amount and the calculating data-quantity threshold, the determining and pending data
The quantity of the corresponding calculate node.
In the present embodiment, the quantity of calculate node is determined in distributed computing framework corresponding with data processing request
The step of, specifically determined according to the calculating data-quantity threshold of the actual amount of data of pending data and each calculate node
, specifically, the target data amount of pending data is obtained, target data amount is pending data after data prediction
The total amount of data of actual needs input calculate node；According to the type of the data processing request, asked with the data processing
Ask the calculating data-quantity threshold that each calculate node is set in corresponding distributed computing framework, specifically, data processing request
Type it is different, corresponding calculating logic is also different, and the complexity and calculating speed of calculating are also different, the present embodiment according to
The concrete type of data processing request, that is, specific complexity determines the calculation amount maximum value i.e. calculation amount threshold of each calculate node
Value, the slow problem of calculating speed caused by the pending data amount for avoiding calculate node from being assigned to is excessive；According to the target
Data volume and the calculating data-quantity threshold determine the quantity of the calculate node corresponding with the pending data, for example,
It is 3M that the accessible data file size of each calculate node, which is arranged, if after obtaining pretreated pending data, if data
Size is 100M, then can start 34 calculate node parallel computations simultaneously, wherein 33 calculate nodes handle 3M file, remain
Calculate node of remaininging handles 1M file, and the data that each calculate node calculates are mutually indepedent, when pending data file becomes larger
When, as long as increasing relevant calculation resource (cpu+ memory), it can be increased concurrent calculate node, and then greatly improve financial machine
Accounting speed of the structure when handling magnanimity finance data, meets the high-timeliness demand that business end handles finance data.
Further, finance data processing method fourth embodiment of the present invention is proposed.
It is the flow diagram of finance data processing method fourth embodiment of the present invention referring to Fig. 5, Fig. 5, is based on above-mentioned gold
Melt data processing method first embodiment, in the present embodiment, step S300, multiple will generated according to the pending data
Pragmatic example is respectively allocated to multiple calculate nodes, and starts multiple calculate node parallel computations, obtained after calculating with
The step of data processing request corresponding processing result includes:
Step S310 generates multiple tasks example according to the pending data, and the multiple task instances is distinguished
It distributes to multiple calculate nodes；
Step S320 is based on the calculate node, transfers from memory database corresponding with the data processing request
Regular configuration parameter；The rule configuration parameter is that the memory database is directed by disk database；
Step S330, the task instances and the regular configuration parameter based on the calculate node, starts multiple institutes
Calculate node parallel computation is stated, processing result corresponding with the data processing request is obtained after calculating.
In the prior art, regular configuration parameter is stored in disk database, calculates be required to from disk every time
Primary regular configuration parameter is transferred in database, reduces calculating speed.
The present embodiment regular configuration parameter corresponding with the data processing request is stored in memory database, this
Embodiment calculate node stores the rule configuration parameter in memory database after transferring regular configuration parameter from disk database,
Thus the frequent inquiry to disk database is reduced in calculating process, and then improves the calculating speed of calculate node.
Further, the 5th embodiment of finance data processing method of the present invention is proposed.
It is the flow diagram of the 5th embodiment of finance data processing method of the present invention referring to Fig. 6, Fig. 6, is based on above-mentioned gold
Melt data processing method second embodiment, in the present embodiment, step S300, multiple will generated according to the pending data
Pragmatic example is respectively allocated to multiple calculate nodes, and starts multiple calculate node parallel computations, obtained after calculating with
After the step of data processing request corresponding processing result further include:
Step S400 saves the processing result into the HDFS file, and updates the Hive table, so as to update
Hive table afterwards is corresponding with including the HDFS file of the processing result.
Each calculate node is according to corresponding pending data and calculating logic, by corresponding logic judgment, rule
After being calculated with rule, calculated result is obtained, Hive is outputted results to.
Further, the processing result is saved into the HDFS file, and updates the Hive by step S400
Table so that updated Hive table with include the corresponding step of the HDFS file of the processing result after further include:
Step S500 is based on preset report tool and the updated Hive table, generates and the target account pair
The Data Management Analysis report answered.
For different data processing requests, corresponding report tool is chosen, for example, data processing request is FTP price
When, report tool selects the report tools such as MSTR or Bi@Report, by the result detailed data being calculated by product, subject
Equal latitudes, which summarize, generates FTP analytical statement, checks FTP analytical statement by MSTR.
If the present embodiment receives the data processing request for target account, the corresponding data of the target account are obtained
Warehouse tool Hive table；Based on preset access mode, the corresponding Hadoop distributed file system of the Hive table is accessed
HDFS file obtains the pending data of the target account；Based on the pending data, with the data processing request
The quantity of calculate node is determined in corresponding distributed computing framework；By the multiple tasks generated according to the pending data reality
Example is respectively allocated to multiple calculate nodes, and starts multiple calculate node parallel computations, obtained after calculating with it is described
The corresponding processing result of data processing request；The processing result is saved into the HDFS file, and updates the Hive
Table, so that updated Hive table is corresponding with including the HDFS file of the processing result；Based on preset report tool and
The updated Hive table generates Data Management Analysis report corresponding with the target account；Financial machine is improved as a result,
Accounting speed of the structure when handling magnanimity finance data, meets the high-timeliness demand that business end handles finance data.
In addition, the embodiment of the present invention also proposes that a kind of finance data processing unit, the finance data processing unit include:
Obtain module, if for receiving the data processing request for target account, obtain the target account to
Handle data；
Determining module, for being based on the pending data, in distributed computing corresponding with the data processing request
The quantity of calculate node is determined in frame；
Processing module, it is multiple described for the multiple tasks example generated according to the pending data to be respectively allocated to
Calculate node, and start multiple calculate node parallel computations, place corresponding with the data processing request is obtained after calculating
Manage result.
Preferably, the acquisition module includes:
First acquisition unit, if obtaining the target account for receiving the data processing request for target account
Corresponding Tool for Data Warehouse Hive table；
It is distributed to access the corresponding Hadoop of the Hive table for being based on preset access mode for second acquisition unit
File system HDFS file, obtains the pending data of the target account.
Preferably, the second acquisition unit includes:
Subelement is determined, for according to the corresponding HDFS file of the Hive table, determination to be corresponding with the HDFS file
The quantity of access process；
Subelement is accessed, for accessing the HDFS file to obtain based on several determining described access processes
State the pending data of target account.
Preferably, the determining module includes:
Third acquiring unit, for obtaining the target data amount of the pending data；
Setting unit, for the type according to the data processing request, at corresponding with the data processing request point
The calculating data-quantity threshold of each calculate node is set in cloth Computational frame；
Determination unit, for according to the target data amount and the calculating data-quantity threshold, it is determining with it is described to be processed
The quantity of the corresponding calculate node of data.
Preferably, the processing module includes:
Allocation unit, for generating multiple tasks example according to the pending data, and by the multiple task instances
It is respectively allocated to multiple calculate nodes；
Call unit is transferred and the data processing request pair for being based on the calculate node from memory database
The regular configuration parameter answered；The rule configuration parameter is that the memory database is directed by disk database；
Processing unit, for based on the calculate node the task instances and the regular configuration parameter, start more
A calculate node parallel computation, obtains processing result corresponding with the data processing request after calculating.
Preferably, described device further include:
Update module for saving the processing result into the HDFS file, and updates the Hive table, so that
Updated Hive table is corresponding with including the HDFS file of the processing result.
Preferably, described device further include:
Reports module generates and the target account for being based on preset report tool and the updated Hive table
The corresponding Data Management Analysis report in family.
It is realized at finance data as described above when the finance data processing unit modules operation that the present embodiment proposes
The step of reason method, details are not described herein.
In addition, the embodiment of the present invention also proposes a kind of computer readable storage medium, gold is stored on the storage medium
Melt data processor, the finance data processing routine realizes finance data processing side as described above when being executed by processor
The step of method.
Wherein, the finance data processing routine run on the processor, which is performed realized method, can refer to this
The each embodiment of invention finance data processing method, details are not described herein again.
It should be noted that, in this document, the terms "include", "comprise" or its any other variant are intended to non-row
His property includes, so that the process, method, article or the device that include a series of elements not only include those elements, and
And further include other elements that are not explicitly listed, or further include for this process, method, article or device institute it is intrinsic
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including being somebody's turn to do
There is also other identical elements in the process, method of element, article or device.
The serial number of the above embodiments of the invention is only for description, does not represent the advantages or disadvantages of the embodiments.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can be realized by means of software and necessary general hardware platform, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Based on this understanding, technical solution of the present invention substantially in other words does the prior art
The part contributed out can be embodied in the form of software products, which is stored in a storage medium
In (such as ROM/RAM, magnetic disk, CD), including some instructions are used so that a terminal device (can be mobile phone, computer, clothes
Business device, air conditioner or the network equipment etc.) execute method described in each embodiment of the present invention.
The above is only a preferred embodiment of the present invention, is not intended to limit the scope of the invention, all to utilize this hair
Equivalent structure or equivalent flow shift made by bright specification and accompanying drawing content is applied directly or indirectly in other relevant skills
Art field, is included within the scope of the present invention.
Claims (16)
1. a kind of finance data processing method, which is characterized in that the finance data processing method the following steps are included:
If receiving the data processing request for target account, the pending data of the target account is obtained；
Based on the pending data, calculate node is determined in distributed computing framework corresponding with the data processing request
Quantity；
The multiple tasks example generated according to the pending data is respectively allocated to multiple calculate nodes, and is started more
A calculate node parallel computation, obtains processing result corresponding with the data processing request after calculating.
2. finance data processing method as described in claim 1, which is characterized in that if described receive for target account
Data processing request, the step of obtaining the pending data of the target account include:
If receiving the data processing request for target account, the corresponding Tool for Data Warehouse Hive of the target account is obtained
Table；
Based on preset access mode, the corresponding Hadoop distributed file system HDFS file of the Hive table is accessed, is obtained
The pending data of the target account.
3. finance data processing method as claimed in claim 2, which is characterized in that it is described to be based on preset access mode, it visits
The step of asking the Hive table corresponding HDFS file, obtaining the pending data of the target account include:
According to the corresponding HDFS file of the Hive table, the quantity of access process corresponding with the HDFS file is determined；
Based on several determining described access processes, the HDFS file is accessed to obtain the number to be processed of the target account
According to.
4. finance data processing method as described in any one of claims 1-3, which is characterized in that described based on described to be processed
Data, in distributed computing framework corresponding with the data processing request determine calculate node quantity the step of include:
Obtain the target data amount of the pending data；
According to the type of the data processing request, it is arranged in distributed computing framework corresponding with the data processing request
The calculating data-quantity threshold of each calculate node；
According to the target data amount and the calculating data-quantity threshold, the calculating corresponding with the pending data is determined
The quantity of node.
5. finance data processing method as described in any one of claims 1-3, which is characterized in that it is described will be according to described wait locate
The multiple tasks example that reason data generate is respectively allocated to multiple calculate nodes, and it is parallel to start multiple calculate nodes
It calculates, the step of processing result corresponding with the data processing request is obtained after calculating includes:
Multiple tasks example is generated according to the pending data, and the multiple task instances are respectively allocated to multiple described
Calculate node；
Based on the calculate node, regular configuration parameter corresponding with the data processing request is transferred from memory database；
The rule configuration parameter is that the memory database is directed by disk database；
The task instances and the regular configuration parameter based on the calculate node, it is parallel to start multiple calculate nodes
It calculates, processing result corresponding with the data processing request is obtained after calculating.
6. finance data processing method as claimed in claim 2, which is characterized in that described to be given birth to according to the pending data
At multiple tasks example be respectively allocated to multiple calculate nodes, and start multiple calculate node parallel computations, count
After the step of obtaining processing result corresponding with the data processing request after calculation further include:
The processing result is saved into the HDFS file, and updates the Hive table so that updated Hive table with
HDFS file including the processing result is corresponding.
7. finance data processing method as claimed in claim 6, which is characterized in that described to save the processing result to institute
It states in HDFS file, and updates the Hive table, so that updated Hive table and the HDFS file for including the processing result
After corresponding step further include:
Based on preset report tool and the updated Hive table, data processing point corresponding with the target account is generated
Analyse report.
8. a kind of finance data processing unit, which is characterized in that the finance data processing unit includes:
Module is obtained, if obtaining the to be processed of the target account for receiving the data processing request for target account
Data；
Determining module, for being based on the pending data, in distributed computing framework corresponding with the data processing request
The quantity of middle determining calculate node；
Processing module, for the multiple tasks example generated according to the pending data to be respectively allocated to multiple calculating
Node, and start multiple calculate node parallel computations, processing knot corresponding with the data processing request is obtained after calculating
Fruit.
9. finance data processing unit as claimed in claim 8, which is characterized in that the acquisition module includes:
First acquisition unit, if it is corresponding to obtain the target account for receiving the data processing request for target account
Tool for Data Warehouse Hive table；
Second acquisition unit accesses the corresponding Hadoop distributed document of the Hive table for being based on preset access mode
System HDFS file, obtains the pending data of the target account.
10. finance data processing unit as claimed in claim 9, which is characterized in that the second acquisition unit includes:
Subelement is determined, for determining access corresponding with the HDFS file according to the corresponding HDFS file of the Hive table
The quantity of process；
Subelement is accessed, for accessing the HDFS file to obtain the mesh based on several determining described access processes
Mark the pending data of account.
11. such as the described in any item finance data processing units of claim 8-10, which is characterized in that the determining module packet
It includes:
Third acquiring unit, for obtaining the target data amount of the pending data；
Setting unit, for the type according to the data processing request, in distribution corresponding with the data processing request
The calculating data-quantity threshold of each calculate node is set in Computational frame；
Determination unit, for according to the target data amount and the calculating data-quantity threshold, the determining and pending data
The quantity of the corresponding calculate node.
12. such as the described in any item finance data processing units of claim 8-10, which is characterized in that the processing module packet
It includes:
Allocation unit for generating multiple tasks example according to the pending data, and the multiple task instances is distinguished
It distributes to multiple calculate nodes；
Call unit is transferred corresponding with the data processing request for being based on the calculate node from memory database
Regular configuration parameter；The rule configuration parameter is that the memory database is directed by disk database；
Processing unit, for based on the calculate node the task instances and the regular configuration parameter, start multiple institutes
Calculate node parallel computation is stated, processing result corresponding with the data processing request is obtained after calculating.
13. finance data processing unit as claimed in claim 9, which is characterized in that described device further include:
Update module for saving the processing result into the HDFS file, and updates the Hive table, so as to update
Hive table afterwards is corresponding with including the HDFS file of the processing result.
14. finance data processing unit as claimed in claim 13, which is characterized in that described device further include:
Reports module generates and the target account pair for being based on preset report tool and the updated Hive table
The Data Management Analysis report answered.
15. a kind of finance data processing equipment, which is characterized in that the equipment includes: memory, processor and is stored in described
On memory and the finance data processing routine that can run on the processor, the finance data processing routine is by the place
Manage the step of realizing the finance data processing method as described in any one of claims 1 to 7 when device executes.
16. a kind of computer readable storage medium, which is characterized in that be stored with financial number on the computer readable storage medium
According to processing routine, realized as described in any one of claims 1 to 7 when the finance data processing routine is executed by processor
The step of finance data processing method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910411302.0A CN110134516A (en) | 2019-05-16 | 2019-05-16 | Finance data processing method, device, equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910411302.0A CN110134516A (en) | 2019-05-16 | 2019-05-16 | Finance data processing method, device, equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110134516A true CN110134516A (en) | 2019-08-16 |
Family
ID=67574857
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910411302.0A Pending CN110134516A (en) | 2019-05-16 | 2019-05-16 | Finance data processing method, device, equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110134516A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110502347A (en) * | 2019-08-28 | 2019-11-26 | 中国银行股份有限公司 | Task load dispatching method and device |
| CN111400351A (en) * | 2020-03-18 | 2020-07-10 | 威讯柏睿数据科技(北京)有限公司 | Method and device for inquiring streaming data based on distributed parallel architecture |
| CN111414386A (en) * | 2020-03-18 | 2020-07-14 | 威讯柏睿数据科技(北京)有限公司 | Method and device for inquiring flow data based on distributed architecture |
| CN111475204A (en) * | 2020-04-26 | 2020-07-31 | 中国人民银行清算总中心 | Host interactive operation method, system and intermediate conversion device |
| CN111858630A (en) * | 2020-07-10 | 2020-10-30 | 山东云海国创云计算装备产业创新中心有限公司 | Data processing method, device and equipment and readable storage medium |
| CN112215511A (en) * | 2020-10-22 | 2021-01-12 | 杭州海康威视系统技术有限公司 | Attendance data calculation method, device and equipment |
| CN112613717A (en) * | 2020-12-17 | 2021-04-06 | 安徽兆尹信息科技股份有限公司 | Financial data processing method and storage medium |
| CN112711659A (en) * | 2020-12-31 | 2021-04-27 | 南京冰鉴信息科技有限公司 | Model calculation method and device based on mass graph data |
| CN115061825A (en) * | 2022-08-09 | 2022-09-16 | 深圳致星科技有限公司 | Heterogeneous computing system and method for private computing, private data and federal learning |
| CN115964181A (en) * | 2023-03-10 | 2023-04-14 | 之江实验室 | Data processing method and device, storage medium and electronic equipment |
| CN117076094A (en) * | 2023-10-16 | 2023-11-17 | 中国船舶集团有限公司第七〇七研究所 | Method for concurrently processing multiple tasks of cryptographic operation |
-
2019
- 2019-05-16 CN CN201910411302.0A patent/CN110134516A/en active Pending
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110502347A (en) * | 2019-08-28 | 2019-11-26 | 中国银行股份有限公司 | Task load dispatching method and device |
| CN110502347B (en) * | 2019-08-28 | 2022-05-27 | 中国银行股份有限公司 | Task load scheduling method and device |
| CN111414386B (en) * | 2020-03-18 | 2021-06-18 | 威讯柏睿数据科技(北京)有限公司 | Method and device for inquiring flow data based on distributed architecture |
| CN111400351A (en) * | 2020-03-18 | 2020-07-10 | 威讯柏睿数据科技(北京)有限公司 | Method and device for inquiring streaming data based on distributed parallel architecture |
| CN111414386A (en) * | 2020-03-18 | 2020-07-14 | 威讯柏睿数据科技(北京)有限公司 | Method and device for inquiring flow data based on distributed architecture |
| CN111475204A (en) * | 2020-04-26 | 2020-07-31 | 中国人民银行清算总中心 | Host interactive operation method, system and intermediate conversion device |
| CN111858630A (en) * | 2020-07-10 | 2020-10-30 | 山东云海国创云计算装备产业创新中心有限公司 | Data processing method, device and equipment and readable storage medium |
| CN111858630B (en) * | 2020-07-10 | 2022-06-17 | 山东云海国创云计算装备产业创新中心有限公司 | Data processing method, device and equipment and readable storage medium |
| CN112215511A (en) * | 2020-10-22 | 2021-01-12 | 杭州海康威视系统技术有限公司 | Attendance data calculation method, device and equipment |
| CN112613717A (en) * | 2020-12-17 | 2021-04-06 | 安徽兆尹信息科技股份有限公司 | Financial data processing method and storage medium |
| CN112711659A (en) * | 2020-12-31 | 2021-04-27 | 南京冰鉴信息科技有限公司 | Model calculation method and device based on mass graph data |
| CN112711659B (en) * | 2020-12-31 | 2024-03-15 | 南京冰鉴信息科技有限公司 | Model calculation method and device based on mass graph data |
| CN115061825A (en) * | 2022-08-09 | 2022-09-16 | 深圳致星科技有限公司 | Heterogeneous computing system and method for private computing, private data and federal learning |
| CN115964181A (en) * | 2023-03-10 | 2023-04-14 | 之江实验室 | Data processing method and device, storage medium and electronic equipment |
| CN117076094A (en) * | 2023-10-16 | 2023-11-17 | 中国船舶集团有限公司第七〇七研究所 | Method for concurrently processing multiple tasks of cryptographic operation |
| CN117076094B (en) * | 2023-10-16 | 2024-01-16 | 中国船舶集团有限公司第七〇七研究所 | Method for concurrently processing multiple tasks of cryptographic operation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110134516A (en) | Finance data processing method, device, equipment and computer readable storage medium | |
| US11106486B2 (en) | Techniques to manage virtual classes for statistical tests | |
| CN101587491A (en) | Hybrid database system using runtime reconfigurable hardware | |
| CN109240946A (en) | The multi-level buffer method and terminal device of data | |
| CN106462578A (en) | Method for querying and updating entries in database | |
| CN110389748A (en) | Business data processing method and terminal device | |
| CN111427971B (en) | Business modeling method, device, system and medium for computer system | |
| CN110019251A (en) | A kind of data processing system, method and apparatus | |
| CN113037877B (en) | Optimization method for time-space data and resource scheduling under cloud edge architecture | |
| CN109324905A (en) | Database operation method, device, electronic equipment and storage medium | |
| CN108830715A (en) | Disk processing method and system are returned in batch documents part | |
| CN103064955A (en) | Inquiry planning method and device | |
| CN112667612A (en) | Data quality checking method and device, electronic equipment and storage medium | |
| CN109032511A (en) | Data storage method, server and storage medium | |
| CN115422205A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN108509259A (en) | Obtain the method and air control system in multiparty data source | |
| CN110275771A (en) | A kind of method for processing business, Internet of Things billing infrastructure system and storage medium | |
| CN114969071A (en) | Data updating method and device | |
| CN109710884B (en) | Real-time index configuration method and system supporting multiple complex calculation modes | |
| CN116385156B (en) | Resource allocation method, apparatus, computer device and computer readable storage medium | |
| CN110377769A (en) | Modeling Platform system, method, server and medium based on graph data structure | |
| CN113572753B (en) | User equipment authentication method and device based on Newton's cooling law | |
| CN117372152A (en) | Resource return plan information generation method and device and computer equipment | |
| CN117151862A (en) | Data processing method, device, system, equipment and storage medium | |
| CN114186961A (en) | Business approval process configuration method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |