CN113435989A - Financial data processing method and device - Google Patents

Financial data processing method and device Download PDF

Info

Publication number
CN113435989A
CN113435989A CN202110714066.7A CN202110714066A CN113435989A CN 113435989 A CN113435989 A CN 113435989A CN 202110714066 A CN202110714066 A CN 202110714066A CN 113435989 A CN113435989 A CN 113435989A
Authority
CN
China
Prior art keywords
data
real
target
time
financial
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110714066.7A
Other languages
Chinese (zh)
Inventor
聂红丹
徐昕
张壹壹
景佳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Industrial and Commercial Bank of China Ltd ICBC
Original Assignee
Industrial and Commercial Bank of China Ltd ICBC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Industrial and Commercial Bank of China Ltd ICBC filed Critical Industrial and Commercial Bank of China Ltd ICBC
Priority to CN202110714066.7A priority Critical patent/CN113435989A/en
Publication of CN113435989A publication Critical patent/CN113435989A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

The embodiment of the application provides a financial data processing method and device, which can be used in the technical field of big data, and the method comprises the following steps: judging whether a target real-time event corresponding to the current financial service request relates to non-real-time data or not based on a preset rule engine, if so, calling target historical transaction data corresponding to service parameters of the target real-time event, and storing the target historical transaction data to a REDIS cluster in a batch processing system; the application flow processing system acquires target real-time data corresponding to a target real-time event and acquires target historical transaction data corresponding to service parameters from the REDIS cluster; and performing data association on the target real-time data and the target historical transaction data, and obtaining processing result data corresponding to the target real-time event based on the associated target real-time data and the target historical transaction data. The batch flow fusion method and the batch flow fusion system can effectively realize batch flow fusion between the batch processing system and the flow processing system under the large financial data processing architecture.

Description

Financial data processing method and device
Technical Field
The application relates to the technical field of data processing, in particular to the technical field of big data, and specifically relates to a financial data processing method and device.
Background
Commercial banks typically employ a streaming data processing framework for real-time online transaction data processing. The streaming data framework is currently composed of two sets of frameworks for batch processing and streaming processing and is used independently. The batch processing mainly uses a SPARK framework to process non-real-time data operation, and the stream processing uses FLINK to process quasi-real-time data operation with higher timeliness.
At present, because the two frames of batch processing and stream processing are independent of each other, two sets of different cluster environments need to be maintained in the development and operation processes, the development specifications of different frames are different, the types of received data are different, and the efficiency of data cleaning and conversion is low. Meanwhile, the existing independently-operated frames cannot meet the situation that batch flow fusion needs exist due to the fact that data of the existing independently-operated frames cannot be obtained timely, for example, when stream processing is used for real-time data analysis, non-real-time data before reference is needed, when batch processing is used for non-real-time data analysis, real-time data needs to be added, and the like.
That is to say, the existing financial data processing mode has the problems of low financial data processing efficiency, poor reliability, incapability of meeting the situation of batch flow fusion requirement and the like.
Disclosure of Invention
Aiming at the problems in the prior art, the application provides a financial data processing method and device, which can effectively realize batch flow fusion between a batch processing system and a flow processing system under a large financial data processing architecture, further can effectively improve the processing efficiency and reliability of the financial data processing process with the batch flow fusion requirement, and effectively meet the financial business processing situation with the batch flow fusion requirement.
In order to solve the technical problem, the application provides the following technical scheme:
in a first aspect, the present application provides a financial data processing method, including:
judging whether a target real-time event corresponding to the current financial service request relates to non-real-time data or not based on a preset rule engine, if so, calling target historical transaction data corresponding to service parameters of the target real-time event, and storing the target historical transaction data to a REDIS cluster in a batch processing system;
an application flow processing system acquires target real-time data corresponding to the target real-time event, and acquires target historical transaction data corresponding to the service parameters from the REDIS cluster;
and performing data association on the target real-time data and the target historical transaction data, and obtaining processing result data corresponding to the target real-time event based on the associated target real-time data and the target historical transaction data.
Further, before the determining, by the preset rule engine, whether the target real-time event corresponding to the current financial service request relates to non-real-time data, the method further includes:
receiving a financial service request;
determining a target processing type of a transaction code corresponding to the financial service request according to a preset transaction code configuration rule;
and if the target processing type is a preset first type or a preset second type, confirming the event to be processed corresponding to the financial service request as a target real-time event, wherein the first type is a real-time processing type, and the second type is a combination of a real-time processing type and a non-real-time processing type.
Further, still include:
if the target processing type is a preset third type, confirming the event to be processed corresponding to the financial service request as a target non-real-time event, wherein the third type is a non-real-time processing type;
and sending the service parameters of the target non-real-time event to a batch processing system so that the batch processing system can call data corresponding to the service parameters of the target non-real-time event within a preset processing time range and obtain processing result data corresponding to the target non-real-time event.
Further, still include:
and if the judgment shows that the target real-time event corresponding to the current financial service request does not relate to non-real-time data, sending the service parameters of the target real-time event to a stream processing system so that the stream processing system can call the data corresponding to the service parameters of the target real-time event in real time and obtain the processing result data corresponding to the target real-time event.
Further, the retrieving target historical transaction data corresponding to the service parameter of the target real-time event and storing the target historical transaction data to the REDIS cluster in the batch processing system includes:
sending the service parameters of the target real-time event to a preset batch processing data lake;
calling corresponding target historical transaction data from a preset distributed file system according to the service parameters in the batch data lake;
storing the target historical transaction data in a REDIS cluster in a batch processing system in the form of key-value pairs.
Further, before sending the service parameters of the target real-time event to a preset batch data lake, the method further includes:
respectively sending the financial service request to a real-time rule engine corresponding to a stream processing system and a non-real-time batch rule engine corresponding to a batch processing system;
correspondingly, the data association between the target real-time data and the target historical transaction data includes:
and according to the rule corresponding to the non-real-time batch rule engine and the rule corresponding to the real-time rule engine, performing data association on target historical transaction data corresponding to the non-real-time batch rule engine and target real-time data corresponding to the real-time rule engine.
Further, the acquiring, by the application stream processing system, target real-time data corresponding to the target real-time event includes:
and writing target real-time data corresponding to the target real-time event into the stream processing system by using a preset KAFKA distributed publishing and subscribing system.
In a second aspect, the present application provides a financial data processing apparatus comprising:
the data judgment module is used for judging whether a target real-time event corresponding to the current financial service request relates to non-real-time data or not based on a preset rule engine, if so, calling target historical transaction data corresponding to the service parameters of the target real-time event, and storing the target historical transaction data to a REDIS cluster in the batch processing system;
a data acquisition module, configured to acquire, by using a stream processing system, target real-time data corresponding to the target real-time event, and acquire target historical transaction data corresponding to the service parameter from the REDIS cluster;
and the data association module is used for performing data association on the target real-time data and the target historical transaction data and obtaining processing result data corresponding to the target real-time event based on the associated target real-time data and the target historical transaction data.
In a third aspect, the present application provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the financial data processing method when executing the program.
In a fourth aspect, the present application provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the financial data processing method described herein.
According to the technical scheme, the financial data processing method and device provided by the application comprise the following steps: judging whether a target real-time event corresponding to the current financial service request relates to non-real-time data or not based on a preset rule engine, if so, calling target historical transaction data corresponding to service parameters of the target real-time event, and storing the target historical transaction data to a REDIS cluster in a batch processing system; an application flow processing system acquires target real-time data corresponding to the target real-time event, and acquires target historical transaction data corresponding to the service parameters from the REDIS cluster; performing data association on the target real-time data and the target historical transaction data, and obtaining processing result data corresponding to the target real-time event based on the associated target real-time data and the target historical transaction data, by storing target historical transaction data into a REDIS cluster in a batch processing system, acquiring the target historical transaction data corresponding to the service parameters from the REDIS cluster by using a stream processing system and carrying out data association on the target real-time data and the target historical transaction data, batch-stream fusion between the batch processing system and the stream processing system under a large financial data processing architecture can be effectively realized, thereby effectively improving the processing efficiency and the intelligent degree of the financial business request with the batch flow fusion requirement, the reliability and the effectiveness of the financial data processing process can be effectively improved, and the financial service processing scene with batch flow fusion requirements can be effectively met; meanwhile, by adopting the REDIS cluster, the combination between the batch processing system and the stream processing system can be effectively realized on the basis of not changing the respective architecture modes of the batch processing system and the stream processing system, the defect that two sets of frames are required for batch processing and stream processing in the prior art is overcome, the defect of processing the financial service request of which part of services need to be associated with historical data and real-time data is overcome, the barriers of the traditional two sets of frames of batch data and real-time data are broken, the resources are saved, the development and maintenance cost is saved, and meanwhile, compared with the prior calculation of only real-time data, the consideration of historical data is added, the analysis and calculation of the financial service request can be carried out in more dimensions, better experience is provided for financial customers, and more visual and comprehensive customer behaviors can be provided for financial institutions or merchants and the like.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic diagram of interaction between a financial data processing apparatus and a client device in an embodiment of the present application.
Fig. 2 is a first flowchart of a financial data processing method according to an embodiment of the present application.
Fig. 3 is a second flowchart of the financial data processing method in the embodiment of the present application.
Fig. 4 is a third flowchart of the financial data processing method in the embodiment of the present application.
Fig. 5 is a fourth flowchart of the financial data processing method in the embodiment of the present application.
Fig. 6 is a fifth flowchart illustrating a financial data processing method according to an embodiment of the present application.
Fig. 7 is a sixth flowchart illustrating a financial data processing method according to an embodiment of the present application.
Fig. 8 is a seventh flowchart illustrating a financial data processing method according to an embodiment of the present application.
Fig. 9 is a schematic structural diagram of a financial data processing apparatus in an embodiment of the present application.
FIG. 10 is a logic function diagram of a financial data processing method according to an exemplary application of the present application.
Fig. 11 is a schematic structural diagram of an electronic device in an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the financial data processing method and apparatus disclosed in the present application may be used in the field of big data technology, and may also be used in any field other than the field of big data technology.
Because the two frames of batch processing and stream processing are independent of each other, two sets of different cluster environments need to be maintained in the development and operation processes, the maintenance cost is high, the development specifications of different frames are different, the types of received data are different, and the data cleaning and conversion cost is higher. Meanwhile, the existing independently-operated frames cannot meet the situation that batch flow fusion needs exist due to the fact that data of the existing independently-operated frames cannot be obtained timely, for example, when stream processing is used for real-time data analysis, non-real-time data before reference is needed, when batch processing is used for non-real-time data analysis, real-time data needs to be added, and the like. That is to say, the existing financial data processing mode has the problems of low financial data processing efficiency, poor reliability, incapability of meeting the situation of batch flow fusion requirement and the like. Based on this, the embodiment of the application provides a financial data processing method, which includes judging whether a target real-time event corresponding to a current financial service request relates to non-real-time data or not based on a preset rule engine, if so, calling target historical transaction data corresponding to service parameters of the target real-time event, and storing the target historical transaction data to a REDIS cluster in a batch processing system; an application flow processing system acquires target real-time data corresponding to the target real-time event, and acquires target historical transaction data corresponding to the service parameters from the REDIS cluster; performing data association on the target real-time data and the target historical transaction data, and obtaining processing result data corresponding to the target real-time event based on the associated target real-time data and the target historical transaction data, by storing target historical transaction data into a REDIS cluster in a batch processing system, acquiring the target historical transaction data corresponding to the service parameters from the REDIS cluster by using a stream processing system and carrying out data association on the target real-time data and the target historical transaction data, batch-stream fusion between the batch processing system and the stream processing system under a large financial data processing architecture can be effectively realized, thereby effectively improving the processing efficiency and the intelligent degree of the financial business request with the batch flow fusion requirement, the reliability and the effectiveness of the financial data processing process can be effectively improved, and the financial service processing scene with batch flow fusion requirements can be effectively met; meanwhile, by adopting the REDIS cluster, the combination between the batch processing system and the stream processing system can be effectively realized on the basis of not changing the respective architecture modes of the batch processing system and the stream processing system, the defect that two sets of frames are required for batch processing and stream processing in the prior art is overcome, the defect of processing the financial service request of which part of services need to be associated with historical data and real-time data is overcome, the barriers of the traditional two sets of frames of batch data and real-time data are broken, the resources are saved, the development and maintenance cost is saved, and meanwhile, compared with the prior calculation of only real-time data, the consideration of historical data is added, the analysis and calculation of the financial service request can be carried out in more dimensions, better experience is provided for financial customers, and more visual and comprehensive customer behaviors can be provided for financial institutions or merchants and the like.
Based on the above, the present application further provides a financial data processing apparatus for implementing the financial data processing method provided in one or more embodiments of the present application, referring to fig. 1, the financial data processing apparatus may be in communication connection with a client device, the financial data processing apparatus may receive a financial service request sent by the client device, and then the financial data processing apparatus invokes a batch processing system, a stream processing system, a distributed file system, and the like preset in a financial institution to perform a financial data processing process for the financial service request, and after obtaining processing result data of the financial service request, sends the processing result data to the client device of the financial institution for storage or display, and the like.
In a practical application scenario, the financial data processing device may be implemented by a server; the server may be communicatively coupled to at least one client device.
It is understood that the client devices may include smart phones, tablet electronic devices, network set-top boxes, portable computers, desktop computers, Personal Digital Assistants (PDAs), in-vehicle devices, smart wearable devices, and the like. Wherein, intelligence wearing equipment can include intelligent glasses, intelligent wrist-watch, intelligent bracelet etc..
In another practical application scenario, the aforementioned financial data processing apparatus may perform part of the financial data processing in the server as described above, or all operations may be performed in the client device. The selection may be specifically performed according to the processing capability of the client device, the limitation of the user usage scenario, and the like. This is not a limitation of the present application. If all operations are performed in the client device, the client device may further include a processor for detailed processing of financial data processing.
In a third practical application scenario, the aforementioned financial data processing apparatus may also be a functional module in a framework such as a streaming platform or a streaming processing system of a financial institution, and so on.
The client device may have a communication module (i.e., a communication unit), and may be communicatively connected to a remote server to implement data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, the server may also include a server on an intermediate platform, for example, a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may include a single computer device, or may include a server cluster formed by a plurality of servers, or a server structure of a distributed apparatus.
The server and the client device may communicate using any suitable network protocol, including network protocols not yet developed at the filing date of this application. The network protocol may include, for example, a TCP/IP protocol, a UDP/IP protocol, an HTTP protocol, an HTTPS protocol, or the like. Of course, the network Protocol may also include, for example, an RPC Protocol (Remote Procedure Call Protocol), a REST Protocol (Representational State Transfer Protocol), and the like used above the above Protocol.
In one or more embodiments of the present application, the non-real-time event refers to an event that does not need to be processed by the stream processing system in real time, and may include a quasi-real-time event, where a processing time rule of the quasi-real-time event is manually set in advance based on the batch processing system, for example, a batch process is performed every 10 minutes or every 24 hours, and the like, and may be specifically set according to an actual application situation.
The following embodiments and application examples are specifically and individually described in detail.
In order to solve the problems that in the existing financial data processing mode, due to the fact that a batch processing frame and a flow processing frame are independent of each other, financial data processing efficiency is low, reliability is poor, the situation that batch flow fusion requirements exist cannot be met, and the like, the financial data processing method is suitable for batch processing scenes that flow platform big data adopt the flow data processing frame to process online transaction data and non-real-time batch data in real time, such as various transaction scenes that commercial banks send account balance short message reminding in real time, blacklist transaction is intercepted in real time, and the like. Referring to fig. 2, the financial data processing method executed by the financial data processing apparatus specifically includes the following steps:
step 110: and judging whether the target real-time event corresponding to the current financial service request relates to non-real-time data or not based on a preset rule engine, if so, executing the step 120.
Step 120: and calling target historical transaction data corresponding to the service parameters of the target real-time event, and storing the target historical transaction data to a REDIS cluster in a batch processing system.
In step 110 and step 120, a Drools rule engine (open source code rule engine) may be introduced as the rule engine to perform engine rule routing for the transaction code corresponding to the target real-time event. And firstly, judging whether historical transaction data need to be read or not by a Drools engine according to transaction codes carried in the bank transaction data, if the historical transaction data do not need to be read, and only performing real-time calculation, directly applying a stream processing system to obtain target real-time data corresponding to the target real-time event, and obtaining processing result data corresponding to the target real-time event according to the target real-time data. If the Drolls engine determines that historical data needs to be read on the basis of reading the real-time data, the data is routed to the real-time rule engine and the non-real-time batch rule engine simultaneously.
Step 200: and the application flow processing system acquires target real-time data corresponding to the target real-time event and acquires target historical transaction data corresponding to the service parameters from the REDIS cluster.
In step 200, real-time data needing to read history data is routed to a non-real-time batch rule engine, while the data enters a FLINK calculation framework of an open source flow processing framework in real time, parameters marking characteristics of data to be requested, such as transaction codes, transaction numbers, application names, dates and the like, are transmitted to a HIVE in the batch processing framework, a reading thread acquires corresponding table data from a Hadoop Distributed File System (HDFS), the reading thread sequentially puts each row of data into a reading queue, a warehousing thread reads data from the corresponding reading queue, the warehousing thread updates a REDIS (remote Dictionary Server) cluster, the data to be acquired (such as data of the same event number, card number, guest code and the like in a certain period of time as the transaction) is stored into the REDIS cluster, the FLINK calls a REDIS class of the FLINK to read the data in the REDIS cluster, by adopting the key value pair storage database cluster such as the REDIS cluster, the stream processing system can quickly and accurately access the REDIS cluster to quickly acquire the historical data in the batch processing system, and the stream processing system does not need to access a data storage system or a database originally corresponding to the batch processing system, so that the stream processing system can quickly acquire the historical data without realizing the compatibility of the stream processing system and the batch processing system through a complex execution means, and the safety and the like of the data in the batch processing system are ensured.
Step 300: and performing data association on the target real-time data and the target historical transaction data, and obtaining processing result data corresponding to the target real-time event based on the associated target real-time data and the target historical transaction data.
In step 300, the data directly entering the FLINK before may be correlated according to the rules in the non-real-time rule engine, for example, the data of the same event number, card number, and guest code may be correlated to realize the service requirement.
As can be seen from the above description, in the financial data processing method provided in this embodiment of the present application, by storing target historical transaction data in a REDIS cluster in a batch processing system, and using a stream processing system to obtain target historical transaction data corresponding to the service parameter from the REDIS cluster and perform data association between the target real-time data and the target historical transaction data, batch stream fusion between the batch processing system and the stream processing system under a large financial data processing architecture can be effectively implemented, so that processing efficiency and intelligence degree of a financial service request with a batch stream fusion requirement can be effectively improved, reliability and effectiveness of a financial data processing process can be effectively improved, and a financial service processing scenario with a batch stream fusion requirement can be effectively satisfied; meanwhile, by adopting the REDIS cluster, the combination between the batch processing system and the stream processing system can be effectively realized on the basis of not changing the respective architecture modes of the batch processing system and the stream processing system, the defect that two sets of frames are required for batch processing and stream processing in the prior art is overcome, the defect of processing the financial service request of which part of services need to be associated with historical data and real-time data is overcome, the barriers of the traditional two sets of frames of batch data and real-time data are broken, the resources are saved, the development and maintenance cost is saved, and meanwhile, compared with the prior calculation of only real-time data, the consideration of historical data is added, the analysis and calculation of the financial service request can be carried out in more dimensions, better experience is provided for financial customers, and more visual and comprehensive customer behaviors can be provided for financial institutions or merchants and the like.
In order to improve the accuracy and efficiency of determining the to-be-processed event corresponding to the financial service request as the target real-time event, in an embodiment of the financial data processing method provided by the present application, referring to fig. 3, before step 100 of the financial data processing method, the following contents are further included:
step 010: a financial transaction request is received.
Step 020: and determining the target processing type of the transaction code corresponding to the financial service request according to a preset transaction code configuration rule.
Step 030: and if the target processing type is a preset first type or a preset second type, confirming the event to be processed corresponding to the financial service request as a target real-time event, wherein the first type is a real-time processing type, and the second type is a combination of a real-time processing type and a non-real-time processing type.
Specifically, for example, the sending of the personal balance account movement reminding short message is configured as a real-time event rule. And the client marketing carries out targeted real-time recommendation, saving and the like according to the transaction behavior of the client within a certain time at the moment of transaction, and is configured as a real-time event rule and a quasi-real-time event rule, and data of the same event number, the card number, the client code and the like within a certain time are calculated together with the real-time transaction.
As can be seen from the above description, in the financial data processing method provided in the embodiment of the present application, the target processing type of the transaction code corresponding to the financial service request is determined according to the preset transaction code configuration rule, so that the accuracy and efficiency of determining the to-be-processed event corresponding to the financial service request as the target real-time event can be effectively improved, and an accurate data basis can be provided for the subsequent financial data processing process, so as to further improve the efficiency and the result reliability of the financial data processing process.
In order to improve the reliability and accuracy of processing the non-real-time event, in an embodiment of the financial data processing method provided by the present application, referring to fig. 4, the following steps are further included after step 020 of the financial data processing method:
step 400: and if the target processing type is a preset third type, confirming the event to be processed corresponding to the financial service request as a target non-real-time event, wherein the third type is a non-real-time processing type.
Step 500: and sending the service parameters of the target non-real-time event to a batch processing system so that the batch processing system can call data corresponding to the service parameters of the target non-real-time event within a preset processing time range and obtain processing result data corresponding to the target non-real-time event.
For example, transaction data made by a client at 9 am needs to be distributed to other applications to be loaded in the next morning, and the rules are configured as quasi-real-time event rules.
As can be seen from the above description, in the financial data processing method provided in this embodiment of the present application, the service parameters of the target non-real-time event are sent to the batch processing system, so that the batch processing system retrieves the data corresponding to the service parameters of the target non-real-time event within a preset processing time range, and obtains the processing result data corresponding to the target non-real-time event, thereby effectively improving the reliability and accuracy of processing the non-real-time event, further effectively improving the comprehensive applicability and the universality of the financial data processing process, and further improving the intelligent degree of the financial data processing process.
In order to improve the efficiency, reliability and accuracy of processing the real-time event, in an embodiment of the financial data processing method provided by the present application, referring to fig. 5, the following steps are further included after step 110 of the financial data processing method:
step 130: and if the judgment shows that the target real-time event corresponding to the current financial service request does not relate to non-real-time data, sending the service parameters of the target real-time event to a stream processing system so that the stream processing system can call the data corresponding to the service parameters of the target real-time event in real time and obtain the processing result data corresponding to the target real-time event.
As can be seen from the above description, in the financial data processing method provided in this embodiment of the present application, the business parameters of the target real-time event are sent to the stream processing system, so that the stream processing system calls the data corresponding to the business parameters of the target real-time event in real time, and obtains the processing result data corresponding to the target real-time event, which can effectively improve the efficiency, reliability, and accuracy of processing the real-time event, and further can effectively improve the comprehensive applicability and the universality of the financial data processing process, so as to further improve the intelligence degree of the financial data processing process.
In order to improve the reliability and efficiency of the processing process of retrieving the target historical transaction data corresponding to the service parameter of the target real-time event and storing the target historical transaction data in the REDIS cluster in the batch processing system, in an embodiment of the financial data processing method provided by the present application, referring to fig. 6, step 120 in the financial data processing method further includes the following contents:
step 121: sending the service parameters of the target real-time event to a preset batch processing data lake;
step 122: and calling corresponding target historical transaction data from a preset distributed file system according to the service parameters in the batch data lake.
Step 123: storing the target historical transaction data in a REDIS cluster in a batch processing system in the form of key-value pairs.
As can be seen from the above description, the financial data processing method provided in the embodiment of the present application, through the use of the batch processing data lake, the distributed file system, and the REDIS cluster, can improve the reliability and efficiency of the processing process of the REDIS cluster in which the target historical transaction data corresponding to the service parameter of the target real-time event is called and stored in the batch processing system, and further can further improve the processing efficiency and reliability of the financial data processing process in which the batch flow fusion requirement exists.
In order to improve the reliability and efficiency of data association between the target real-time data and the target historical transaction data, in an embodiment of the financial data processing method provided by the present application, referring to fig. 7, before step 121 in the financial data processing method, the following is further included:
step 124: and respectively sending the financial service request to a real-time rule engine corresponding to the stream processing system and a non-real-time batch rule engine corresponding to the batch processing system.
Correspondingly, step 300 of the financial data processing method further includes the following steps:
step 310: and according to the rule corresponding to the non-real-time batch rule engine and the rule corresponding to the real-time rule engine, performing data association on target historical transaction data corresponding to the non-real-time batch rule engine and target real-time data corresponding to the real-time rule engine.
In step 310, the specific way of performing data association on the target historical transaction data corresponding to the non-real-time batch rule engine and the target real-time data corresponding to the real-time rule engine may be to perform FLINK operator calculation on the target historical transaction data and the target real-time data by applying a preset FLINK operator calculation way.
Step 320: and obtaining processing result data corresponding to the target real-time event based on the associated target real-time data and the target historical transaction data.
As can be seen from the above description, in the financial data processing method provided in the embodiment of the present application, according to the rule corresponding to the non-real-time batch rule engine and the rule corresponding to the real-time rule engine, the target historical transaction data corresponding to the non-real-time batch rule engine and the target real-time data corresponding to the real-time rule engine are subjected to data association, so that the reliability and efficiency of performing data association on the target real-time data and the target historical transaction data can be effectively improved, and further, the processing efficiency and reliability of the financial data processing process with the batch flow fusion requirement can be further improved.
In order to improve the efficiency and reliability of the stream processing system for acquiring the target real-time data corresponding to the target real-time event, in an embodiment of the financial data processing method provided by the present application, referring to fig. 8, step 200 in the financial data processing method specifically includes the following contents:
step 210: and writing target real-time data corresponding to the target real-time event into the stream processing system by using a preset KAFKA distributed publishing and subscribing system.
Step 220: and acquiring target historical transaction data corresponding to the service parameters from the REDIS cluster.
As can be seen from the above description, in the financial data processing method provided in this embodiment of the present application, the preset KAFKA distributed publish-subscribe system is used to write the target real-time data corresponding to the target real-time event into the stream processing system, so that the efficiency and reliability of the stream processing system for acquiring the target real-time data corresponding to the target real-time event can be effectively improved, and further, the processing efficiency and reliability of the financial data processing process requiring batch-stream fusion can be further improved.
In terms of software, in order to solve the problems that the existing financial data processing method has low financial data processing efficiency and poor reliability due to the fact that the two frames of batch processing and stream processing are independent of each other, and cannot meet the situation that batch-stream fusion needs exist, the present application provides an embodiment of a financial data processing apparatus for executing all or part of the content in the financial data processing method, and referring to fig. 9, the financial data processing apparatus specifically includes the following contents:
the data judging module 10 is configured to judge whether a target real-time event corresponding to the current financial service request relates to non-real-time data based on a preset rule engine, and if so, retrieve target historical transaction data corresponding to a service parameter of the target real-time event, and store the target historical transaction data to a REDIS cluster in the batch processing system.
In the data determining module 10, a Drools rule engine (open source code rule engine) may be introduced as the rule engine to perform engine rule routing for the transaction code corresponding to the target real-time event. And firstly, judging whether historical transaction data need to be read or not by a Drools engine according to transaction codes carried in the bank transaction data, if the historical transaction data do not need to be read, and only performing real-time calculation, directly applying a stream processing system to obtain target real-time data corresponding to the target real-time event, and obtaining processing result data corresponding to the target real-time event according to the target real-time data. If the Drolls engine determines that historical data needs to be read on the basis of reading the real-time data, the data is routed to the real-time rule engine and the non-real-time batch rule engine simultaneously.
A data obtaining module 20, configured to obtain, by the application flow processing system, target real-time data corresponding to the target real-time event, and obtain, from the REDIS cluster, target historical transaction data corresponding to the service parameter.
In the data acquisition module 20, real-time data that requires reading of historical data is routed to a non-real-time batch rules engine, while the data enter the FLINK calculation framework of the open source flow processing framework in real time, parameters marking the characteristics of the data to be requested are transmitted into the batch processing data lake HIVE in the batch processing framework, such as transaction codes, transaction numbers, application names, dates and the like, the reading thread acquires the corresponding table data from the Hadoop distributed file system HDFS, the reading thread sequentially puts each row of data into a reading queue, the warehousing thread reads data from the corresponding reading queue, updates a remote Dictionary service (REDIS) cluster, stores the data to be fetched (such as data in a certain period of time of the same event number, card number, guest code and the like as the transaction) to the REDIS cluster, the open source stream processing framework FLINK calls its own REDIS class to read the data in the REDIS cluster.
And the data association module 30 is configured to perform data association on the target real-time data and the target historical transaction data, and obtain processing result data corresponding to the target real-time event based on the associated target real-time data and the target historical transaction data.
In the data association module 30, the data directly entering the flip before may be associated according to the rule in the non-real-time rule engine, for example, the data with the same event number, card number, and guest code may be associated and calculated, so as to meet the service requirement.
The embodiment of the financial data processing apparatus provided in the present application may be specifically configured to execute the processing flow of the embodiment of the financial data processing method in the foregoing embodiment, and the functions of the processing flow are not described herein again, and reference may be made to the detailed description of the embodiment of the method.
As can be seen from the above description, in the financial data processing apparatus provided in this embodiment of the present application, by storing target historical transaction data in a REDIS cluster in a batch processing system, and using a stream processing system to obtain target historical transaction data corresponding to the service parameter from the REDIS cluster and perform data association between the target real-time data and the target historical transaction data, batch stream fusion between the batch processing system and the stream processing system under a large financial data processing architecture can be effectively implemented, so that processing efficiency and intelligence degree of a financial service request with a batch stream fusion requirement can be effectively improved, reliability and effectiveness of a financial data processing process can be effectively improved, and a financial service processing scenario with a batch stream fusion requirement can be effectively satisfied; meanwhile, by adopting the REDIS cluster, the combination between the batch processing system and the stream processing system can be effectively realized on the basis of not changing the respective architecture modes of the batch processing system and the stream processing system, the defect that two sets of frames are required for batch processing and stream processing in the prior art is overcome, the defect of processing the financial service request of which part of services need to be associated with historical data and real-time data is overcome, the barriers of the traditional two sets of frames of batch data and real-time data are broken, the resources are saved, the development and maintenance cost is saved, and meanwhile, compared with the prior calculation of only real-time data, the consideration of historical data is added, the analysis and calculation of the financial service request can be carried out in more dimensions, better experience is provided for financial customers, and more visual and comprehensive customer behaviors can be provided for financial institutions or merchants and the like.
For further explanation, the present application further provides a specific application example of the financial data processing method, and the application example is suitable for batch processing scenarios in which the streaming platform big data adopts a streaming data processing framework to process online transaction data and non-real-time batch data in real time, such as various transaction scenarios in which a commercial bank sends a moving account balance short message prompt in real time, and blacklist transactions are intercepted in real time.
The application example overcomes the defect that two sets of frames are required for batch processing and stream processing in the prior art, simultaneously makes up the defect that historical data and real-time data requirements need to be comprehensively considered for partial services, and provides the batch-stream fusion financial data processing method based on the FLINK open source stream processing frame and the REDIS memory key-value database cluster.
Referring to fig. 10, the financial data processing method mentioned in the application example of the present application specifically includes the following contents:
step 1: the business is configured according to the transaction code at the foreground (business foreground rule configuration);
for example: and sending the personal balance account-moving reminding short message, and configuring the short message as a real-time event rule.
For example: and the client marketing carries out targeted real-time recommendation, saving and the like according to the transaction behavior of the client within a certain time at the moment of transaction, and is configured as a real-time event rule and a quasi-real-time event rule, and data of the same event number, the card number, the client code and the like within a certain time are calculated together with the real-time transaction.
For example: transaction data made by a client at 9 am needs to be distributed to other applications to be loaded in the next morning, and the rules are configured as quasi-real-time event rules.
Step 2: a Drools rules engine (open source code rules engine) is introduced to perform engine rule routing on data according to transaction codes. A piece of bank transaction data enters a streaming platform, firstly, a Drools engine judges whether historical transaction data needs to be read or not according to transaction codes carried in the data, and if the historical transaction data does not need to be read and only real-time calculation is carried out, step 3 is executed; if the Drolls engine judges that historical data needs to be read on the basis of reading the real-time data, the data is simultaneously routed to the real-time rule engine and the quasi-real-time batch rule engine, namely, the step 3 and the step 6 are executed;
and step 3: routing the data that is only calculated in real time to a real-time rules engine;
and 4, step 4: data are written into corresponding TOPIC of KAFKA;
and 5: and directly entering an open source flow processing frame FLINK to obtain a real-time result through calculation by an operator.
Step 6: real-time data needing to read historical data is routed to a quasi-real-time batch rule engine;
and 7: when the data enter the FLINK calculation framework of the open source flow processing framework in real time, parameters marking data characteristics to be requested, such as transaction codes, transaction numbers, application names, dates and the like, are transmitted to the HIVE in the batch processing framework;
and 8: the reading thread acquires corresponding table data from a Hadoop Distributed File System (HDFS), and the reading thread sequentially puts each row of data into a reading queue;
and step 9: and the warehousing thread reads data from the corresponding reading queue and updates a remote Dictionary service (REDIS) (remote Dictionary Server) cluster.
Step 10: after data to be fetched (such as data in a certain period of time, such as an event number, a card number, a guest code and the like which are the same as the transaction) is stored in the REDIS cluster, the open source stream processing framework FLINK calls the REDIS class of the open source stream processing framework FLINK to read the data in the REDIS cluster, and the data which are directly entered into the FLINK before are associated according to rules in the quasi-real-time rule engine, such as the data of the same event number, the card number and the guest code are associated and calculated, so that service requirements are realized.
Step 11: and meanwhile, a user self-defined function UDF can be used for writing the FLINKSQL for the historical data, so that batch requirements are realized, namely when the data is routed to the quasi-real-time batch rule engine by the Drolls rule engine, the part of data is loaded to the REDIS, and the FLINK reads the REDIS data to perform the FLINKSQL batch calculation.
Based on the above, the implementation of the data processing example of specific batch stream combination is as follows:
step 01, constructing a memory database cluster based on REDIS for storing key-value data;
step 02, writing the real-time data into an open-source stream processing framework FLINK through a KAFKA distributed publishing and subscribing system;
step 03, transmitting data characteristic parameters to the batch data lake HIVE by the non-real-time data to obtain corresponding data;
step 04, writing the data acquired in the step 03 into the REDIS memory database cluster constructed in the step 01 in the form of key-value pairs to serve as a data source of an open source stream processing framework FLINK;
the data source of step 05.FLINK includes real-time data written in step 02 through KAFKA distributed publish-subscribe system and batch data obtained in step 04 through REDIS memory database cluster. And if only real-time stream processing is needed, carrying out FLINK operator calculation by using real-time data. And if historical data is needed while real-time calculation is carried out, the FLINK operator is calculated by using two data sources. If only batch processing is required, batch calculation of FLINKSQL is performed using data obtained from REDIS.
From the above, the financial data processing method based on batch-flow fusion of the FLINK and the REDIS provided by the application example of the application example establishes a processing mode covering both batch and streaming data, breaks through the barriers of two traditional frames of batch and real-time data, saves the development and maintenance costs while saving resources, and can analyze and calculate data with more dimensions compared with the conventional calculation of only real-time data and by adding the consideration of historical data, thereby providing better experience for customers and more intuitive and comprehensive customer behaviors for merchants. By taking the intelligent marketing business of the bank credit card as an example, the financial data processing method provided by the application can be used for analyzing the inflow or loss condition of a client by combining the historical data behavior of the client in real time when the client opens or sells the card in a financial institution, and the data in a certain period of time is visually displayed in a report form in batches, so that the bank can make targeted adjustment and improvement.
In terms of hardware, in order to solve the problems that the existing financial data processing method has low financial data processing efficiency and poor reliability due to the fact that the two frames of batch processing and stream processing are independent of each other, and cannot meet the situation that batch and stream fusion needs exist, the present application provides an embodiment of an electronic device for implementing all or part of the contents in the financial data processing method, where the electronic device specifically includes the following contents:
fig. 11 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application. As shown in fig. 11, the electronic device 9600 can include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this FIG. 11 is exemplary; other types of structures may also be used in addition to or in place of the structure to implement telecommunications or other functions.
In one embodiment, the financial data processing functions may be integrated into a central processor. Wherein the central processor may be configured to control:
step 110: and judging whether the target real-time event corresponding to the current financial service request relates to non-real-time data or not based on a preset rule engine, if so, executing the step 120.
Step 120: and calling target historical transaction data corresponding to the service parameters of the target real-time event, and storing the target historical transaction data to a REDIS cluster in a batch processing system.
Step 200: and the application flow processing system acquires target real-time data corresponding to the target real-time event and acquires target historical transaction data corresponding to the service parameters from the REDIS cluster.
Step 300: and performing data association on the target real-time data and the target historical transaction data, and obtaining processing result data corresponding to the target real-time event based on the associated target real-time data and the target historical transaction data.
As can be seen from the above description, in the electronic device provided in the embodiment of the present application, by storing target historical transaction data in a REDIS cluster in a batch processing system, and using a stream processing system to obtain target historical transaction data corresponding to the service parameter from the REDIS cluster and perform data association between the target real-time data and the target historical transaction data, batch stream fusion between the batch processing system and the stream processing system under a large financial data processing architecture can be effectively implemented, so that processing efficiency and intelligence degree of a financial service request with a batch stream fusion requirement can be effectively improved, reliability and effectiveness of a financial data processing process can be effectively improved, and a financial service processing scenario with a batch stream fusion requirement can be effectively satisfied; meanwhile, by adopting the REDIS cluster, the combination between the batch processing system and the stream processing system can be effectively realized on the basis of not changing the respective architecture modes of the batch processing system and the stream processing system, the defect that two sets of frames are required for batch processing and stream processing in the prior art is overcome, the defect of processing the financial service request of which part of services need to be associated with historical data and real-time data is overcome, the barriers of the traditional two sets of frames of batch data and real-time data are broken, the resources are saved, the development and maintenance cost is saved, and meanwhile, compared with the prior calculation of only real-time data, the consideration of historical data is added, the analysis and calculation of the financial service request can be carried out in more dimensions, better experience is provided for financial customers, and more visual and comprehensive customer behaviors can be provided for financial institutions or merchants and the like.
In another embodiment, the financial data processing apparatus may be configured separately from the central processor 9100, for example, the financial data processing apparatus may be configured as a chip connected to the central processor 9100, and the financial data processing function is realized by the control of the central processor.
As shown in fig. 11, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 also does not necessarily include all of the components shown in fig. 11; in addition, the electronic device 9600 may further include components not shown in fig. 11, which may be referred to in the prior art.
As shown in fig. 11, a central processor 9100, sometimes referred to as a controller or operational control, can include a microprocessor or other processor device and/or logic device, which central processor 9100 receives input and controls the operation of the various components of the electronic device 9600.
The memory 9140 can be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit 9100 can execute the program stored in the memory 9140 to realize information storage or processing, or the like.
The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. Power supply 9170 is used to provide power to electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, an LCD display, but is not limited thereto.
The memory 9140 can be a solid state memory, e.g., Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 9140 could also be some other type of device. Memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 being used for storing application programs and function programs or for executing a flow of operations of the electronic device 9600 by the central processor 9100.
The memory 9140 can also include a data store 9143, the data store 9143 being used to store data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, contact book applications, etc.).
The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. The communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as in the case of a conventional mobile communication terminal.
Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby implementing ordinary telecommunications functions. The audio processor 9130 may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100, thereby enabling recording locally through the microphone 9132 and enabling locally stored sounds to be played through the speaker 9131.
Embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps in the financial data processing method in the foregoing embodiments, where the computer-readable storage medium stores thereon a computer program, and when the computer program is executed by a processor, the computer program implements all the steps of the financial data processing method in the foregoing embodiments, where the execution subject is a server or a client, for example, the processor implements the following steps when executing the computer program:
step 110: and judging whether the target real-time event corresponding to the current financial service request relates to non-real-time data or not based on a preset rule engine, if so, executing the step 120.
Step 120: and calling target historical transaction data corresponding to the service parameters of the target real-time event, and storing the target historical transaction data to a REDIS cluster in a batch processing system.
Step 200: and the application flow processing system acquires target real-time data corresponding to the target real-time event and acquires target historical transaction data corresponding to the service parameters from the REDIS cluster.
Step 300: and performing data association on the target real-time data and the target historical transaction data, and obtaining processing result data corresponding to the target real-time event based on the associated target real-time data and the target historical transaction data.
As can be seen from the above description, in the computer-readable storage medium provided in this embodiment of the present application, by storing target historical transaction data in a REDIS cluster in a batch processing system, and using a stream processing system to obtain target historical transaction data corresponding to the service parameter from the REDIS cluster and perform data association between the target real-time data and the target historical transaction data, batch-stream fusion between the batch processing system and the stream processing system under a large financial data processing architecture can be effectively implemented, so that processing efficiency and intelligence of a financial service request with a batch-stream fusion requirement can be effectively improved, reliability and validity of a financial data processing process can be effectively improved, and a financial service processing scenario with a batch-stream fusion requirement can be effectively satisfied; meanwhile, by adopting the REDIS cluster, the combination between the batch processing system and the stream processing system can be effectively realized on the basis of not changing the respective architecture modes of the batch processing system and the stream processing system, the defect that two sets of frames are required for batch processing and stream processing in the prior art is overcome, the defect of processing the financial service request of which part of services need to be associated with historical data and real-time data is overcome, the barriers of the traditional two sets of frames of batch data and real-time data are broken, the resources are saved, the development and maintenance cost is saved, and meanwhile, compared with the prior calculation of only real-time data, the consideration of historical data is added, the analysis and calculation of the financial service request can be carried out in more dimensions, better experience is provided for financial customers, and more visual and comprehensive customer behaviors can be provided for financial institutions or merchants and the like.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principle and the implementation mode of the invention are explained by applying specific embodiments in the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims (10)

1. A method of financial data processing, comprising:
judging whether a target real-time event corresponding to the current financial service request relates to non-real-time data or not based on a preset rule engine, if so, calling target historical transaction data corresponding to service parameters of the target real-time event, and storing the target historical transaction data to a REDIS cluster in a batch processing system;
an application flow processing system acquires target real-time data corresponding to the target real-time event, and acquires target historical transaction data corresponding to the service parameters from the REDIS cluster;
and performing data association on the target real-time data and the target historical transaction data, and obtaining processing result data corresponding to the target real-time event based on the associated target real-time data and the target historical transaction data.
2. The method of claim 1, wherein before the determining whether the target real-time event corresponding to the current financial transaction request relates to non-real-time data based on the predetermined rules engine, the method further comprises:
receiving a financial service request;
determining a target processing type of a transaction code corresponding to the financial service request according to a preset transaction code configuration rule;
and if the target processing type is a preset first type or a preset second type, confirming the event to be processed corresponding to the financial service request as a target real-time event, wherein the first type is a real-time processing type, and the second type is a combination of a real-time processing type and a non-real-time processing type.
3. The financial data processing method of claim 2 further comprising:
if the target processing type is a preset third type, confirming the event to be processed corresponding to the financial service request as a target non-real-time event, wherein the third type is a non-real-time processing type;
and sending the service parameters of the target non-real-time event to a batch processing system so that the batch processing system can call data corresponding to the service parameters of the target non-real-time event within a preset processing time range and obtain processing result data corresponding to the target non-real-time event.
4. The financial data processing method of claim 1, further comprising:
and if the judgment shows that the target real-time event corresponding to the current financial service request does not relate to non-real-time data, sending the service parameters of the target real-time event to a stream processing system so that the stream processing system can call the data corresponding to the service parameters of the target real-time event in real time and obtain the processing result data corresponding to the target real-time event.
5. The financial data processing method of claim 1, wherein the retrieving the target historical transaction data corresponding to the business parameter of the target real-time event and storing the target historical transaction data to the REDIS cluster in the batch processing system comprises:
sending the service parameters of the target real-time event to a preset batch processing data lake;
calling corresponding target historical transaction data from a preset distributed file system according to the service parameters in the batch data lake;
storing the target historical transaction data in a REDIS cluster in a batch processing system in the form of key-value pairs.
6. The financial data processing method of claim 5 further comprising, before sending the business parameters of the target real-time event into a preset batch data lake:
respectively sending the financial service request to a real-time rule engine corresponding to a stream processing system and a non-real-time batch rule engine corresponding to a batch processing system;
correspondingly, the data association between the target real-time data and the target historical transaction data includes:
and according to the rule corresponding to the non-real-time batch rule engine and the rule corresponding to the real-time rule engine, performing data association on target historical transaction data corresponding to the non-real-time batch rule engine and target real-time data corresponding to the real-time rule engine.
7. The method of claim 1, wherein the obtaining of the target real-time data corresponding to the target real-time event by the application flow processing system comprises:
and writing target real-time data corresponding to the target real-time event into the stream processing system by using a preset KAFKA distributed publishing and subscribing system.
8. A financial data processing apparatus, comprising:
the data judgment module is used for judging whether a target real-time event corresponding to the current financial service request relates to non-real-time data or not based on a preset rule engine, if so, calling target historical transaction data corresponding to the service parameters of the target real-time event, and storing the target historical transaction data to a REDIS cluster in the batch processing system;
a data acquisition module, configured to acquire, by using a stream processing system, target real-time data corresponding to the target real-time event, and acquire target historical transaction data corresponding to the service parameter from the REDIS cluster;
and the data association module is used for performing data association on the target real-time data and the target historical transaction data and obtaining processing result data corresponding to the target real-time event based on the associated target real-time data and the target historical transaction data.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the financial data processing method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium on which a computer program is stored, the computer program, when being executed by a processor, implementing the financial data processing method according to any one of claims 1 to 7.
CN202110714066.7A 2021-06-25 2021-06-25 Financial data processing method and device Pending CN113435989A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110714066.7A CN113435989A (en) 2021-06-25 2021-06-25 Financial data processing method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110714066.7A CN113435989A (en) 2021-06-25 2021-06-25 Financial data processing method and device

Publications (1)

Publication Number Publication Date
CN113435989A true CN113435989A (en) 2021-09-24

Family

ID=77754694

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110714066.7A Pending CN113435989A (en) 2021-06-25 2021-06-25 Financial data processing method and device

Country Status (1)

Country Link
CN (1) CN113435989A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113836235A (en) * 2021-09-29 2021-12-24 平安医疗健康管理股份有限公司 Data processing method based on data center and related equipment thereof
CN114996260A (en) * 2022-08-05 2022-09-02 深圳市深蓝信息科技开发有限公司 Method and device for cleaning AIS data, terminal equipment and storage medium
CN115618396A (en) * 2022-11-28 2023-01-17 云账户技术(天津)有限公司 Data encryption method and device
CN116841753A (en) * 2023-08-31 2023-10-03 杭州迅杭科技有限公司 Stream processing and batch processing switching method and switching device

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113836235A (en) * 2021-09-29 2021-12-24 平安医疗健康管理股份有限公司 Data processing method based on data center and related equipment thereof
CN113836235B (en) * 2021-09-29 2024-04-09 平安医疗健康管理股份有限公司 Data processing method based on data center and related equipment thereof
CN114996260A (en) * 2022-08-05 2022-09-02 深圳市深蓝信息科技开发有限公司 Method and device for cleaning AIS data, terminal equipment and storage medium
CN114996260B (en) * 2022-08-05 2022-11-11 深圳市深蓝信息科技开发有限公司 Method and device for cleaning AIS data, terminal equipment and storage medium
CN115618396A (en) * 2022-11-28 2023-01-17 云账户技术(天津)有限公司 Data encryption method and device
CN115618396B (en) * 2022-11-28 2023-04-07 云账户技术(天津)有限公司 Data encryption method and device
CN116841753A (en) * 2023-08-31 2023-10-03 杭州迅杭科技有限公司 Stream processing and batch processing switching method and switching device
CN116841753B (en) * 2023-08-31 2023-11-17 杭州迅杭科技有限公司 Stream processing and batch processing switching method and switching device

Similar Documents

Publication Publication Date Title
CN113435989A (en) Financial data processing method and device
CN111031058A (en) Websocket-based distributed server cluster interaction method and device
CN110764881A (en) Distributed system background retry method and device
CN112615753B (en) Link abnormity tracking method, first node, second node and link
CN110908875B (en) Inspection method and device based on operation terminal
US20220391642A1 (en) Method and apparatus for evaluating joint training model
CN110990228A (en) Data interface monitoring method and device
CN111353892A (en) Transaction risk monitoring method and device
CN112163946A (en) Accounting processing method and device based on distributed transaction system
CN111405129A (en) Intelligent outbound risk monitoring method and device
CN111767558B (en) Data access monitoring method, device and system
CN113051094A (en) Supervision data submission testing method and device
CN114257532B (en) Method and device for detecting state of server
CN115099930A (en) Financial business data processing method and device
CN113434423A (en) Interface test method and device
CN113553152A (en) Job scheduling method and device
CN112927017A (en) Control method, device and system for outbound marketing process
CN111930624A (en) Test link message data processing method and device
CN112968954B (en) Flow control method, server and system for service migration
CN115731019A (en) Flexible transaction data processing method and device
CN114331458A (en) Suspicious transaction monitoring method and device
CN115278555A (en) Online transaction method and device based on 5G message
CN115719229A (en) Outbound transaction verification method and device
CN117041102A (en) Service jitter exception handling method and device
CN113378179A (en) Industrial control vulnerability scanning system based on active and passive nondestructive scanning technology

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination