CN115601172A

CN115601172A - Data processing method, device, equipment and storage medium

Info

Publication number: CN115601172A
Application number: CN202211301834.7A
Authority: CN
Inventors: 唐俊军; 宋龙飞; 孙海英; 陈凯; 何雷; 唐家鑫
Original assignee: Shanghai Pudong Development Bank Co Ltd
Current assignee: Shanghai Pudong Development Bank Co Ltd
Priority date: 2022-10-24
Filing date: 2022-10-24
Publication date: 2023-01-13

Abstract

The invention discloses a data processing method, a data processing device, data processing equipment and a storage medium. The method comprises the following steps: acquiring a transaction detail event set to be inquired from a compensation event table according to a preset acquisition condition; inquiring an association table according to the customer account number corresponding to each transaction detail event to be inquired in the transaction detail event set to be inquired, wherein the association table comprises: the corresponding relation between the customer account number and the customer number; if the target client number corresponding to the transaction detail event to be inquired is inquired, the target client number and the transaction detail corresponding to the transaction detail event to be inquired are stored in the database.

Description

Data processing method, device, equipment and storage medium

Technical Field

The embodiment of the invention relates to the technical field of computers, in particular to a data processing method, a data processing device, data processing equipment and a storage medium.

Background

Streaming computing is a data engine designed to handle infinite data sets. The current mainstream data out-of-order processing method comprises the following steps: window, watermark level, and allow to be deferred to three ways.

The window mode is as follows: the Window is a means for processing unlimited data into finite blocks, and can be opened to solve the problem of data disorder.

Watermark water level mode: the time semantic Watermark in the Flink takes the event time minus the allowed maximum disorder time as a WaterMark, the principle is equivalent to giving a certain time to data, and then a window is closed to trigger calculation.

Late mode allowed: the principle is that more data is given certain time which can be delayed on the basis of the watermark, when the watermark reaches the window size, calculation is triggered, but the window is not closed, and the window is really closed after the allowed delay time is reached.

The conventional data out-of-order processing method has the following disadvantages:

the window calculation, the size of the window or the time increases the delay of the overall data.

Watermark further increases the latency of the data on a window basis.

Late arrival is allowed, and the delay of data is also increased on the basis of window calculation.

Disclosure of Invention

Embodiments of the present invention provide a data processing method, an apparatus, a device, and a storage medium, so as to solve a problem of data disorder in processing of a high-concurrency data stream based on a Flink stream processing, and improve data accuracy without affecting most of data delay.

According to an aspect of the present invention, there is provided a data processing method including:

acquiring a transaction detail event set to be inquired from a compensation event table according to a preset acquisition condition;

inquiring an association table according to the customer account number corresponding to each transaction detail event to be inquired in the transaction detail event set to be inquired, wherein the association table comprises: the corresponding relation between the customer account number and the customer number;

and if the target customer number corresponding to the transaction detail event to be inquired is inquired, storing the target customer number and the transaction detail corresponding to the transaction detail event to be inquired in a database.

According to another aspect of the present invention, there is provided a data processing apparatus comprising:

the acquisition module is used for acquiring a transaction detail event set to be inquired from the compensation event table according to a preset acquisition condition;

a query module, configured to query an association table according to a customer account number corresponding to each transaction detail event to be queried in the transaction detail event set to be queried, where the association table includes: the corresponding relation between the client account number and the client number;

and the storage module is used for storing the target client number and the transaction detail corresponding to the transaction detail event to be inquired into a database if the target client number corresponding to the transaction detail event to be inquired is inquired.

According to another aspect of the present invention, there is provided an electronic apparatus including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the data processing method according to any of the embodiments of the present invention.

According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement a data processing method according to any one of the embodiments of the present invention when the computer instructions are executed.

According to the embodiment of the invention, a transaction detail event set to be inquired is obtained from a compensation event table according to a preset acquisition condition; inquiring an association table according to the customer account number corresponding to each transaction detail event to be inquired in the transaction detail event set to be inquired, wherein the association table comprises: the corresponding relation between the client account number and the client number; if the target client number corresponding to the transaction detail event to be inquired is inquired, the target client number and the transaction detail corresponding to the transaction detail event to be inquired are stored in the database, the problem of data disorder in high-concurrency data stream processing based on Flink streaming is solved, and the data accuracy can be improved under the condition that most data delay is not influenced.

It should be understood that the statements in this section are not intended to identify key or critical features of the embodiments of the present invention, nor are they intended to limit the scope of the invention. Other features of the present invention will become apparent from the following description.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.

FIG. 1 is a flow chart of a data processing method in an embodiment of the invention;

FIG. 2 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;

fig. 3 is a schematic structural diagram of an electronic device in an embodiment of the present invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in other sequences than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example one

Fig. 1 is a flowchart of a data processing method provided in an embodiment of the present invention, where this embodiment is applicable to a data processing situation, and the method may be executed by a data processing apparatus in an embodiment of the present invention, where the apparatus may be implemented in a software and/or hardware manner, and as shown in fig. 1, the method specifically includes the following steps:

and S110, acquiring a transaction detail event set to be inquired from the compensation event table according to a preset acquisition condition.

Wherein, the preset collecting condition may be: the retry times of the transaction detail events are less than or equal to the time threshold, and the event status of the transaction detail events is an unprocessed status, or the preset collection condition may be that the event status of the transaction detail events is an unprocessed status.

Wherein the compensation event table comprises: the transaction detail event that the client number is not inquired, the retry times corresponding to the transaction detail event that the client number is not inquired, the state information of the transaction detail event that the client number is not inquired, and the like.

The compensation event table may be generated in a manner as follows: acquiring a first event set from a first topoc through a Source component; filtering the first event set according to the table name field corresponding to each event to obtain a transaction detail event set; inquiring an association table according to a client account corresponding to each transaction detail event in the transaction detail event set; and generating a compensation event table according to the transaction detail event without inquiring the client number. The updating mode of the compensation event table comprises the following steps: and if the target client number corresponding to the transaction detail event to be inquired is not inquired, storing the transaction detail event to be inquired into a compensation event table, and updating the retry times of the transaction detail event to be inquired. The updating method of the compensation event table may further include: and if the target client number corresponding to the transaction detail event to be inquired is inquired, storing the target client number and the transaction detail corresponding to the transaction detail event to be inquired into a database, and modifying the state information corresponding to the transaction detail event to be inquired into a processed state.

Specifically, according to the preset collection condition, the manner of obtaining the transaction detail event set to be queried from the compensation event table may be: obtaining a preset number of transaction detail events to be inquired, of which the retry times are less than or equal to a time threshold value and the event states are unprocessed, from the compensation event table, and generating a transaction detail event set to be inquired according to the preset number of transaction detail events to be inquired.

And S120, inquiring an association table according to the customer account corresponding to each transaction detail event to be inquired in the transaction detail event set to be inquired.

Wherein the association table comprises: and the corresponding relation between the client account number and the client number.

Wherein the association table comprises: and the corresponding relation between the customer account number and the customer number. Wherein, the client number is the unique identification information of the client. The generation mode of the association table may be: acquiring a second event set from the second topoc through a Source component; filtering the second event set according to the table name field corresponding to each event to obtain a client information event set; and generating an association table according to the event type and the client information corresponding to each client information event in the client information event set.

Analyzing the transaction detail event to obtain a transaction detail, wherein the transaction detail comprises: the customer account number.

Specifically, the manner of querying the association table according to the customer account number corresponding to each transaction detail event to be queried in the transaction detail event set to be queried may be: analyzing each transaction detail event to be inquired in the transaction detail event set to obtain a transaction detail corresponding to each transaction detail event to be inquired, further obtaining a customer account corresponding to each transaction detail event to be inquired, and inquiring the association table according to the customer account corresponding to each transaction detail event to be inquired.

S130, if the target client number corresponding to the transaction detail event to be inquired is inquired, storing the target client number and the transaction detail corresponding to the transaction detail event to be inquired in a database.

Specifically, if a target customer number corresponding to a transaction detail event to be queried is queried, the target customer number and the transaction detail corresponding to the transaction detail event to be queried are stored in a database, and state information corresponding to the transaction detail event to be queried in a compensation event table is updated to be a processed state.

It should be noted that, if the target client number corresponding to the transaction detail event to be queried is not queried, the transaction detail event to be queried is stored in the compensation event table, and the retry number of the transaction detail event to be queried is updated.

Optionally, the preset collecting condition includes: the number of retries of the transaction detail event is less than or equal to a number threshold, and the event state of the transaction detail event is an unprocessed state.

And the retry times are increased by one after the transaction detail events are stored in the compensation event table. The number threshold is preset.

And if the transaction detail event is the transaction detail event of which the client number is not inquired, determining the event state of the transaction detail event as an unprocessed state.

Optionally, obtaining a transaction detail event set to be queried from the compensation event table according to a preset acquisition condition, including:

obtaining the transaction detail events to be inquired, of which the retry times are less than or equal to a time threshold value and the event states are unprocessed, in a preset number from the compensation event table;

and generating a transaction detail event set to be inquired according to the preset number of transaction detail events to be inquired.

The preset number may be a preset processing batch size.

In a specific example, the data in the compensation event table with unprocessed status is collected, and the data collection is performed by adopting a periodic collection and a method of limiting the size of the processing batch, where the collection conditions are as follows: the number of retries does not exceed the set size and the data state is unprocessed.

Optionally, before acquiring the transaction detail event set to be queried from the compensation event table according to the preset acquisition condition, the method further includes:

acquiring a first event set from a first topoc through a Source component;

filtering the first event set according to the table name field corresponding to each event to obtain a transaction detail event set;

inquiring an association table according to a client account corresponding to each transaction detail event in the transaction detail event set;

and generating a compensation event table according to the transaction detail event without inquiring the client number.

Specifically, the manner of obtaining the first event set from the first topoic through the Source component may be: acquiring first configuration information of the Source assembly, and determining that the topoc identifier is the first topoc according to the first configuration information of the Source assembly. A first set of events is obtained from the first topoc through a Source component.

Specifically, the manner of filtering the first event set according to the table name field corresponding to each event to obtain the transaction detail event set may be as follows: the method comprises the steps of obtaining a table name field corresponding to each event in a first event set, matching the table name field corresponding to each event with a first preset table name field list (the first preset table name field list is a table name field list corresponding to a transaction detail event), and obtaining the events in the first event set, which are the same as the table name fields in the first preset table name field list; and generating a transaction detail event set according to the events which are the same as the table name fields in the first preset table name field list.

Specifically, the manner of generating the compensation event table according to the transaction detail event without inquiring the customer number may be: acquiring a transaction detail event of which the client number is not inquired, state information of the transaction detail event and retry times (the initial retry times are 0); and generating a compensation event table according to the transaction detail event without inquiring the client number, the state information of the transaction detail event and the retry number.

Optionally, the method further includes:

acquiring a second event set from the second topoc through a Source component;

filtering the second event set according to the table name field corresponding to each event to obtain a client information event set;

and updating the association table according to the event type and the client information corresponding to each client information event in the client information event set.

The method for acquiring the second event set from the second topoc through the Source component may be: and acquiring second configuration information of the Source assembly, and determining that the topoc identifier is a second topoc according to the second configuration information of the Source assembly. And acquiring the second event set from the second topoc through the Source component.

The filtering of the second event set according to the table name field corresponding to each event may be performed to obtain the client information event set in the following manner: acquiring a table name field corresponding to each event in a second event set, matching the table name field corresponding to each event with a second preset table name field list (the second preset table name field list is a table name field list corresponding to the client information event), and acquiring events in the second event set, which are the same as the table name fields in the second preset table name field list; and generating a client information event set according to the events which are the same as the table name fields in the second preset table name field list.

Specifically, the manner of updating the association table according to the event type and the client information corresponding to each client information event in the client information event set may be: and generating a database operation statement according to the event type and the client information corresponding to each client information event in the client information event set, and synchronizing the client information into the database according to the database operation statement. The method for updating the association table according to the event type and the client information corresponding to each client information event in the client information event set may also be: if the event type is a new event, performing new operation on the association table according to the client information; if the event type is a deletion event, deleting the association table according to the client information; and if the event type is a modification event, modifying the association table according to the client information.

Optionally, the updating the association table according to the event type and the client information corresponding to each client information event in the client information event set includes:

if the event type is a new event, performing new addition operation on the association table according to the client information;

if the event type is a deletion event, deleting the association table according to the client information;

and if the event type is a modification event, modifying the association table according to the client information.

If the event type is a new event, the manner of performing new addition operation on the association table according to the client information may be: and if the event type is a newly added event, acquiring a customer account and a customer number included in customer information, and adding the customer account and the customer number included in the customer information into an association table.

If the event type is a delete event, the method of deleting the association table according to the client information may be: and if the event type is a deletion event, acquiring a client account number and a client number which are included in the client information, inquiring the association table according to the client number to obtain data in the association table corresponding to the client number, and deleting the data in the association table corresponding to the client number from the association table.

If the event type is a modification event, modifying the association table according to the client information, wherein the method may be: and if the event type is a modification event, taking the client account number and the client number included in the client information, inquiring the association table according to the client number to obtain data in the association table corresponding to the client number, and modifying the data in the association table according to the client information.

Optionally, the method further includes:

and if the target client number corresponding to the transaction detail event to be inquired is not inquired, storing the transaction detail event to be inquired into a compensation event table, and updating the retry times of the transaction detail event to be inquired.

Specifically, the manner of updating the retry number of the transaction detail event to be queried may be: the number of retries is incremented by one.

In one specific example, the transaction system OLTP implementation generates transaction details, which are stored in a database of the transaction system. And collecting the transaction detail by using a CDC data collection tool, storing the transaction detail into the specified topic of Kafka, and acquiring the transaction detail event from the corresponding topic through a Source component of Flink. During the Flink data stream Processing, a Flink-connect-jdbc component is used for inquiring the association table. And (4) sending the transaction details Sink to the TiDB database through a flink-connect-jdbc component. Wherein, the TiDB database is used by the front end through OLAP service or OLTP service.

In terms of the order of business development, the business system generates the customer information first, and the customer can actually generate the transaction, thereby generating the transaction detail. In actual processing, part of transaction detail data cannot be associated with a client number, and retrospective investigation finds that due to various factors such as a network, backpressure, an upstream system supply mode and the like, a client information event is not stored in an association table, and the transaction detail is consumed and processed by a Flink cluster, so that data disorder occurs, dirty data is generated, and the consistency of the data is influenced. The embodiment of the invention is used for independently processing the transaction detail which is not associated with the client number, so that the main data stream is not influenced, and most transaction detail data can be put in a warehouse in a quasi-real-time manner, thereby improving the user experience.

In another specific example, three Flink streaming processes are designed to implement data processing:

first, client information flow processing flow

The client information generated by the business system is persisted to a business system database in an OLTP mode; and the CDC tool collects and configures a customer information table of the service database, captures the operations of adding, modifying and deleting in the customer information table, assembles a single independent event according to an agreed event template, and stores the event into the corresponding topic in the Kafka cluster. And binding the appointed topic in the Kafka cluster through the Source component, and obtaining the read-only right in an authentication mode. In order to maintain data consistency, the configuration of consumption events is in a pre-commit manner, preventing repeated consumption from causing dirty data. And filtering to obtain the client information event according to the table name field in the event, and mapping the service field in the event to the context of the streaming processing flow. In the Processing layer, a database operation statement is generated according to a database operation mark (adding/updating/logic deleting) in the context reading event, and a Flink-connector-jdbc component is used for synchronizing the update to the TIDB database in the Sink layer.

Second, transaction detail stream processing flow

And the transaction detail generated by the business system is persisted into a database in an OLTP mode, and the CDC tool collects and configures the transaction detail table of the business database, assembles the transaction detail table into a single independent event according to an agreed event template and stores the single independent event into the corresponding topic in the Kafka cluster. And binding the appointed topic in the Kafka cluster through the Source component, and obtaining the read-only right in an authentication mode. In order to maintain data consistency, the configuration of consumption events is in a pre-commit manner, preventing repeated consumption from causing dirty data. And filtering to obtain the transaction detail event according to the table name field in the event, and mapping the service field in the event to the context of the streaming processing flow. In the Processing layer, a flink-connector-jdbc component is used for inquiring the association table according to the account number field in the context, if the client number can be inquired, the transaction detail is widened, an insert statement is assembled according to the service field in the context, the transaction detail is stored in the TIDB, and the online transaction inquiry of OLAPOLTP is provided. If the client number cannot be inquired, the number of initialization retries is 0, the state is unprocessed, information such as transaction detail events, the number of retries, the state and the like is output to a compensation event table by using a flink-connector-jdbc, and the data disorder compensation flow is waited for processing.

Third, data out-of-order compensation processing flow

The Source data Source is a custom data acquisition component, acquires data with unprocessed state in the compensation event table, and acquires the data by adopting a mode of periodic acquisition and limited processing batch size, wherein the acquisition condition is that the retry number does not exceed the set size and the data state is unprocessed. Analyzing the transaction detail event to be inquired acquired from the compensation event table to obtain fields such as account number, retry times, data state and the like, and mapping the fields to the context. When the transaction detail event is processed, if the corresponding client number can be inquired according to the client account number in the context, the data is stored in the database after being widened, and the state of the event in the compensation event table is updated to be processed. If the client number corresponding to the transaction detail is not associated, the retry times in the compensation event table are updated and added by one, and the event state is kept in an unprocessed state and waits for the batch processing of the next period.

It should be noted that the compensation process parameter configuration and the database custom query component can be adjusted according to the actual operation condition of actual production to achieve the optimal condition, and improve the accuracy and data consistency of the Flink streaming processing.

According to the technical scheme of the embodiment, a transaction detail event set to be inquired is obtained from a compensation event table according to a preset acquisition condition; inquiring an association table according to a customer account number corresponding to each transaction detail event to be inquired in the transaction detail event set to be inquired, wherein the association table comprises: the corresponding relation between the customer account number and the customer number; if the target client number corresponding to the transaction detail event to be inquired is inquired, the target client number and the transaction detail corresponding to the transaction detail event to be inquired are stored in a database, so that most data can be stored in a warehouse in a quasi-real-time mode, and the customer experience is improved. And the out-of-order data can be ensured to be persisted in the database, and even if the compensation process is not completed, the out-of-order data can be processed in the daily timing batch task.

Example two

Fig. 2 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention. The present embodiment may be applicable to the case of data processing, the apparatus may be implemented in a software and/or hardware manner, and the apparatus may be integrated in any device that provides a data processing function, as shown in fig. 2, where the data processing apparatus specifically includes: an acquisition module 210, a query module 220, and a storage module 230.

the query module is configured to query an association table according to a customer account number corresponding to each transaction detail event to be queried in the transaction detail event set to be queried, where the association table includes: the corresponding relation between the customer account number and the customer number;

and the storage module is used for storing the target customer number and the transaction detail corresponding to the transaction detail event to be inquired into a database if the target customer number corresponding to the transaction detail event to be inquired is inquired.

The product can execute the method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.

EXAMPLE III

FIG. 3 illustrates a schematic diagram of an electronic device 10 that may be used to implement an embodiment of the present invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.

As shown in fig. 3, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from a storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.

A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 11 performs the various methods and processes described above, such as a data processing method.

In some embodiments, the data processing method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the Internet.

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired result of the technical solution of the present invention can be achieved.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data processing method, comprising:

2. The method of claim 1, wherein the preset acquisition conditions comprise: the number of retries of the transaction detail event is less than or equal to a number threshold, and the event state of the transaction detail event is an unprocessed state.

3. The method of claim 1, wherein obtaining the transaction detail event set to be queried from the compensation event table according to a preset collection condition comprises:

4. The method according to claim 1, before obtaining the transaction detail event set to be queried from the compensation event table according to the preset collection condition, further comprising:

acquiring a first event set from a first topoc through a Source component;

5. The method of claim 1, further comprising:

acquiring a second event set from the second topoc through a Source component;

6. The method of claim 5, wherein updating the association table according to the event type and the customer information corresponding to each customer information event in the set of customer information events comprises:

if the event type is a new event, performing new operation on the association table according to the client information;

7. The method of claim 5, further comprising:

8. A data processing apparatus, comprising:

9. An electronic device, characterized in that the electronic device comprises:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,

the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data processing method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that it stores computer instructions for causing a processor to implement the data processing method of any of claims 1-7 when executed.