CN113034283A - Transaction data processing method, device and system - Google Patents
Transaction data processing method, device and system Download PDFInfo
- Publication number
- CN113034283A CN113034283A CN202110336438.7A CN202110336438A CN113034283A CN 113034283 A CN113034283 A CN 113034283A CN 202110336438 A CN202110336438 A CN 202110336438A CN 113034283 A CN113034283 A CN 113034283A
- Authority
- CN
- China
- Prior art keywords
- transaction data
- server
- application server
- transaction
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 19
- 238000012545 processing Methods 0.000 claims abstract description 78
- 238000000034 method Methods 0.000 claims abstract description 58
- 230000015654 memory Effects 0.000 claims description 27
- 230000005540 biological transmission Effects 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 29
- 238000010586 diagram Methods 0.000 description 15
- 238000004590 computer program Methods 0.000 description 10
- 239000000872 buffer Substances 0.000 description 8
- 230000006870 function Effects 0.000 description 5
- 230000000903 blocking effect Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000003139 buffering effect Effects 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000002688 persistence Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Accounting & Taxation (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Finance (AREA)
- Development Economics (AREA)
- Economics (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- Technology Law (AREA)
- General Business, Economics & Management (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a transaction data processing method, device and system, and belongs to the technical field of data processing. The method comprises the following steps: the method comprises the steps that a first server obtains transaction data generated by a transaction system and caches the transaction data in a target queue of the first server; the method comprises the steps that an application server pulls at least one piece of transaction data from a target queue, wherein the application server is associated with the target queue; and the application server sends the at least one transaction datum to the second server for caching, and deletes the at least one transaction datum in the target queue. The transaction data processing method provided by the application can avoid overload operation caused by overlarge transaction data amount passively received by the application server in a high-concurrency large-transaction-amount scene, so that the phenomenon that the application server collapses to cause data loss is reduced, and the stability of the application server in the process of counting the transaction data is improved.
Description
Technical Field
The application belongs to the technical field of data processing, and particularly relates to a method, a device and a system for processing transaction data.
Background
With the rapid development of mobile payment technology, online transaction has become the main transaction mode of people, and is used frequently especially in banking business. At present, for transaction data generated in an online transaction process, an application server generally counts the generated transaction data to obtain a statistical result, and then writes the statistical result into a database in a transaction manner for a user to query.
However, due to the rapid increase of the transaction amount and the transaction peak value, a scene with high concurrency and large transaction amount may exist, which may cause a phenomenon of breakdown caused by overload operation in the process of performing data statistics after the application server receives excessive transaction data, and further cause data loss in the process of performing statistics on the transaction data by the application server, thereby causing a reduction in system stability.
Disclosure of Invention
The embodiment of the application aims to provide a transaction data processing method, a transaction data processing device and a transaction data processing system, and the problem of low stability of an application server in the process of counting transaction data can be solved.
In order to solve the technical problem, the present application is implemented as follows:
in a first aspect, an embodiment of the present application provides a transaction data processing method, which is applied to a transaction data processing system including a first server, an application server, and a second server, and the method includes:
the first server acquires transaction data generated by the transaction system and caches the transaction data in a target queue of the first server;
the application server pulling at least one piece of transaction data from the target queue, wherein the application server is associated with the target queue;
and the application server sends the at least one piece of transaction data to the second server for caching, and deletes the at least one piece of transaction data in the target queue.
In a second aspect, an embodiment of the present application provides a transaction data processing apparatus, which is applied to a transaction data processing system including a first server, an application server, and a second server, and includes:
the first cache module is used for the first server to acquire transaction data generated by the transaction system and cache the transaction data in a target queue of the first server;
a data pull module, configured to pull at least one piece of transaction data from the target queue by the application server, where the application server is associated with the target queue;
and the second cache module is used for sending the at least one transaction data to the second server for caching by the application server and deleting the at least one transaction data in the target queue.
In a third aspect, embodiments of the present application provide a transaction data processing system comprising a processor, a memory, and a program or instructions stored on the memory and executable on the processor, which when executed by the processor implements the steps of the method according to the first aspect.
In a fourth aspect, embodiments of the present application provide a readable storage medium, on which a program or instructions are stored, which when executed by a processor implement the steps of the method according to the first aspect.
In the embodiment of the application, in the transaction data processing process, the transaction data generated by the transaction system is cached through the first server, the application server is associated with the target queue of the first server for caching the transaction data, when the transaction data is processed by the application server, the transaction data can be pulled from the target queue of the first server, caching is performed again in the second server, and the pulled transaction data in the first server is deleted after caching is completed, so that the problems that the data in the first server is processed repeatedly or the data is lost can be avoided. Therefore, by the transaction data processing method in the embodiment of the application, the transaction data generated by the transaction system is cached twice by the first server and the second server, rather than being directly sent to the application server for processing, and in a high-concurrency large-transaction-volume scene, overload operation caused by overlarge transaction data volume passively received by the application server can be avoided, so that the phenomenon of data loss caused by collapse of the application server is reduced, and the stability of the application server in the process of processing the transaction data is further improved.
Drawings
FIG. 1 is a block diagram of a transaction data processing system according to an embodiment of the present disclosure;
FIG. 2 is a schematic flow chart of a transaction data processing method according to an embodiment of the present disclosure;
FIG. 3 is a second schematic diagram of a transaction data processing system according to an embodiment of the present application;
fig. 4 is a second flowchart of a transaction data processing method according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a transaction data processing apparatus according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of a computing device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application.
The terms first, second and the like in the description and in the claims of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that embodiments of the application may be practiced in sequences other than those illustrated or described herein, and that the terms "first," "second," and the like are generally used herein in a generic sense and do not limit the number of terms, e.g., the first term can be one or more than one. In addition, "and/or" in the specification and claims means at least one of connected objects, a character "/" generally means that a preceding and succeeding related objects are in an "or" relationship.
The following describes in detail a transaction data processing method provided by the embodiment of the present application through a specific embodiment and an application scenario thereof with reference to the accompanying drawings.
Referring to fig. 1, which is a schematic structural diagram of a transaction data processing system according to an embodiment of the present disclosure, as shown in fig. 1, the transaction data processing system 10 is configured to receive transaction data generated by a transaction system 20, and perform data statistics on the received transaction data to obtain a statistical result.
The transaction data processing system 10 may include a first server 11, an application server 12, and a second server 13;
the first server 11 is connected to the transaction system 20, and the first server 11 is configured to receive and cache transaction data generated by the transaction system 20;
the application server 12 is connected to the first server 11 and the second server 13, respectively, and is configured to: the cached transaction data is pulled from the first server 11 to be cached in the second server 13, and the transaction data is obtained from the second server 13 for statistics to obtain a statistical result;
the second server 13 is used for caching the transaction data pulled by the application server 12 from the first server 11, and the application server 12 generates a statistical result according to the transaction data.
Referring to fig. 2, it is a schematic flow chart of a transaction data processing method according to an embodiment of the present application, applied to the transaction data processing system shown in fig. 1, and as shown in fig. 2, the transaction data processing method includes the following steps:
Therefore, in the transaction data processing process, the transaction data generated by the transaction system 20 is cached by the first server 11, and the application server is associated with the target queue of the first server 11 for caching the transaction data, when the transaction data is processed by the application server, the transaction data can be pulled from the target queue of the first server, the transaction data is cached again in the second server, and the pulled transaction data in the first server is deleted after the caching is completed, so that the problems that the data in the first server is processed repeatedly or the data is lost can be avoided. Thus, by the transaction data processing method in the embodiment of the present application, the transaction data generated by the transaction system 20 is cached twice by the first server and the second server, rather than being directly sent to the application server 12 for processing, and in a scenario with high concurrency and large transaction amount, overload operation caused by an excessively large amount of transaction data passively received by the application server 12 can be avoided, so that a phenomenon that the application server crashes to cause data loss is reduced, and stability of the application server 12 in a process of processing the transaction data is further improved.
In step 201, when the transaction system 20 generates transaction data, the transaction system 20 sends the generated transaction data to the first server 11, and the first server 11 receives the transaction data sent by the transaction system and buffers the transaction data in the target queue.
The transaction system 20 may be any system capable of implementing online transaction, and the transaction data is generated in the online transaction process, and the transaction data may include information such as payee information, transaction amount, transaction time, and transaction account name. For example, the transaction system may be a banking system, and the transaction data may be data generated in banking transactions, including deposit data, transfer data, withdrawal data, and the like.
The first server 11 may be any server having a data caching function. For example, the first server 11 may be a distributed cache server kafka or the like.
It should be noted that the first server 11 is provided with a target queue, and when the first server 11 receives the transaction data, the first server 11 buffers the received transaction data into the target queue. The first server 11 caches the received transaction data, and may cache the transaction data according to a transaction data type, and specifically, may cache the transaction data of the same transaction type in the same consumption queue. For example, in the case where the first server 11 is kafka, kafka may buffer the transaction data it receives into its one or more consumption queues according to the transaction data type.
In step 201, in the case that the first server 11 caches transaction data generated by the transaction system, the application server 12 may pull at least one piece of transaction data from the target queue where the transaction data is cached in the first server 11.
The application server 12 may be associated with the target queue of the first server 11, and the application server 12 and the target queue may be associated in advance. For example, in the case where the first server 11 is kafka, the application server 12 may subscribe to a consumption queue in kafka.
In addition, the above-mentioned application server 12 pulls at least one piece of transaction data from the target queue, it may be understood that the application server 12 initiates a transaction data pulling request to the first server 11, and the first server 11, in response to the transaction data pulling request, pulls out and sends the transaction data cached in the queue associated with the application server 12 to the application server 12.
It should be noted that, the application server 12 may pull the transaction data from the target queue, and may execute the operation of pulling the transaction data only by one thread; alternatively, to improve the efficiency of pulling transaction data, the application server 12 may set multiple threads to pull transaction data from the target queue at the same time, for example, the application server 12 may pull transaction data through two threads, and so on.
In this embodiment of the application, the application server 12 pulls the transaction data from the target queue, where the application server 12 pulls at least one piece of transaction data from the target queue according to a preset data pulling rule.
For example, a preset pull cycle and a preset pull number are set in the application server 12, and the application server 12 pulls transaction receipts of the preset pull number from the target queue in each preset pull cycle to realize the regular and quantitative pull of transaction data; alternatively, the application server 12 may pull transaction data from the target queue at a preset rate, and so on.
Or, the application server 12 may also control the application server 12 to pull the transaction data to the target queue in a manner of blocking the transaction data pull thread, that is, the application server 12 may determine to pull the transaction data from the target queue or stop pulling the transaction data according to its own needs, so as to further reduce the occurrence of a crash phenomenon in the transaction data processing process of the application server 12.
For example, when a queue of a thread of the application server 12 that pulls transaction data is full, the queue may be prevented from continuing to pull transaction data to the target queue.
Alternatively, before the step 202, the method may further include:
the application server acquires data processing capacity information;
the application server determines a data pulling rate based on the data processing capacity information;
the application server pulls at least one piece of transaction data from the target queue, and the pulling comprises the following steps:
and the application server pulls at least one piece of transaction data from the target queue according to the data pulling rate.
Based on this, in the process of pulling transaction data from the target queue, the application server 12 may determine the data pulling rate through its own data processing capability information, and pull the transaction data from the target queue according to the determined data pulling rate, so that the application server 12 may pull an appropriate amount of transaction data according to its own data processing capability and determine whether to stop pulling the transaction data (that is, the data pulling rate is zero), thereby realizing the controllability of the transaction data pulling process, and further improving the stability in the transaction data statistics.
The data processing capability information may be any information that can reflect the performance of the application server 12 in processing data, for example, the data processing capability information may be the memory occupancy rate of the processor of the application server, the current process number, and the like.
In addition, the application server 12 determines the data pulling rate based on the data processing capability information, which may be a preset relationship between the processing capability information and the pulling rate preset in the application server 12, and the application server 12 may determine, according to each preset relationship, a rate having a preset relationship with the acquired data processing capability information as the data pulling rate.
It should be noted that after the application server 12 determines the data pull rate, it may pull transaction data from the target queue according to the determined data pull rate. For example, where the data pull rate is determined to be 5 pieces/second, the application server 12 pulls 5 pieces of transaction data from the target queue every 5 seconds, and so on.
In step 203, after the application server 12 pulls the at least one transaction datum from the first server 11, the application server 12 may send the at least one pulled transaction datum to the second server 13 for caching, and after the application server 12 sends the at least one transaction datum to the second server 13 for caching, the first server 11 deletes the pulled transaction datum in the target queue.
The application server 12 sends the at least one transaction data to a second server for caching, where the second server 13 caches the transaction data, or caches each transaction data in the at least one transaction data; alternatively, the sending, by the application server 12, the at least one piece of transaction data to the second server 13 for caching may include: the application server 12 filters the at least one transaction data, and sends the filtered valid transaction data to the second server 13 for caching, so that the redundancy of the data can be reduced, and the validity of the transaction data can be ensured.
In addition, the second server 13 may be any server capable of storing data, for example, the second server 13 may be a redis, and the like.
In this embodiment of the application, after the application server 12 pulls the at least one piece of transaction data, statistical processing may be performed on the pulled transaction data.
The above-mentioned application server 12 may perform statistical processing on the pulled transaction data, and after the application server 12 sends the at least one piece of transaction data to the second server 13 for caching, the application server 12 obtains the cached transaction data from the second server 13 for statistics, so that in the case that the transaction data is lost (for example, the transaction data is lost due to crash or power failure of the application server) during the process of performing statistics on the transaction data by the application server 12, the application server 12 may also read the cached transaction data from the second server 13 for statistics, thereby further improving the stability of transaction data processing.
Since the transaction data cached by the second server 13 is the filtered valid transaction data, the application server 12 may also obtain only the valid transaction data for statistics, specifically, the method may further include: the application server obtains the effective transaction data from the second server, and counts the effective transaction data to generate a statistical result, so that the validity of the statistical result can be ensured.
It should be noted that, the application server 12 acquires the at least one transaction data from the second server 13, and performs statistics on the at least one transaction data to generate a statistical result, or the application server 12 may also send the generated statistical result to the second server 13, and may also establish an association relationship between the statistical result and the transaction data used for generating the statistical result.
Or, after the application server obtains the valid transaction data from the second server, and performs statistics on the valid transaction data, and generates a statistical result, the method may further include:
the application server transmits the statistical result to the second server;
and the second server caches the statistical result and deletes the effective transaction data cached by the second server.
Based on this, the application server 12 can cache the statistical result in the second server 13, and delete the valid transaction data for obtaining the statistical result, so that the user can not only inquire the statistical data, but also reduce the redundancy of the data in the second server 13.
It should be noted that, the application server 12 may perform statistics on the at least one transaction datum by using a preset statistical rule or method. For example, the at least one piece of transaction data may be counted according to the dimension of the merchant, such as merging the transaction data of the same merchant.
In addition, the application server 12 may count the transaction data by only one thread; the transaction data may be counted by a plurality of threads, for example, the transaction data may be counted by a dual thread, and so on.
In this embodiment of the application, after the application server 12 sends the pulled transaction data to the second server 13 for caching, the first server 11 may delete the pulled transaction data in its target queue.
The first server 11 deletes the pulled transaction data in the target queue, which may be that after the application server 12 pulls the at least one transaction data from the target queue, the first server 11 obtains an offset of the pulled transaction data, and deletes the transaction data corresponding to the offset in the target queue.
The offset of the pulled transaction data obtained by the first server 11 may be an offset of the pulled transaction data generated by the first server 11 in the data pulling process.
Or, after the application server sends the at least one piece of transaction data to the second server for caching, the method may further include:
the application server generating a target offset of the transaction data cached to the second server;
the application server sends the target offset to the first server;
the deleting the at least one transaction datum in the target queue may further include:
the first server updates the offset of the transaction data pulled by the application server based on the target offset to delete the at least one transaction data.
Based on this, the application server 12 may generate a target offset of the cached transaction data after caching the pulled transaction data in the second server 13, and update the offset automatically generated by the first server 11 through the target offset to delete the transaction data corresponding to the updated offset, so as to improve the accuracy of the offset acquired by the first server 11, and further reduce the possibility of missing or repeatedly pulling the transaction data.
It should be noted that, since the process of generating the offset is not an improvement of the present application, a person skilled in the art may implement the offset by an existing manner, and details are not described herein.
The generating of the target offset and the sending of the target offset to the first server may be completed within a preset time period after the application server 12 caches the pulled transaction data in the second server 13; alternatively, the application server may cache the pulled transaction data in the second server 13, and at the same time, generate the target offset and immediately send the target offset to the first server 11, so as to update the offset of the first server 11, thereby ensuring reliability of the statistical data and efficiency of processing the transaction data.
In addition, when the transaction data cached in the second server 13 is the transaction data obtained by filtering the at least one transaction data, the target offset may be an offset of the filtered transaction data.
The first server 11 updates the offset of the transaction data pulled by the application server 12 based on the target offset, and when the first server 11 generates the offset of the pulled transaction data, the first server 11 may replace the generated offset with the target offset; on the other hand, when the offset is not generated in the first server 11, the received target offset is directly used as the offset of the pulled transaction data.
For the convenience of understanding the transaction data processing method of the present application, a practical application process of the transaction data processing method of the present application is described herein with reference to fig. 3 and 4, specifically as follows:
as shown in fig. 3, the apparatus 100 (i.e. the transaction data processing system 10) of the real-time transaction data statistical scheme is composed of three modules, namely a transaction data buffering module 110 (i.e. the first server 11), a transaction data processing module 120 (i.e. the application server 12) and a statistical data persistence module 130 (i.e. the second server 13), wherein:
the transactional data buffering module 110 utilizes kafka as a buffer area for bulk transactional data that the transactional system writes into kafka. The kafka with high availability and high performance is utilized as a buffer for large volumes of data. The transaction data processing module 120 performs real-time transaction summary statistics, peak clipping and asynchronous transaction data processing from the batch of pulled data in the transaction data buffer module 110.
The transaction data processing module 120 subscribes to a consumption queue (i.e., a target queue) in kafka, extracts data from kafka, filters out invalid transaction data, retains valid data in a statistical data persistence module (e.g., redis)130, submits an offset (i.e., a target offset), prevents data loss or repeated consumption, accelerates consumption thread consumption, and performs summary statistics on valid data through thread pool multithreading. The specific implementation steps are shown in the flow chart of fig. 4:
and step 121, when the transaction system synchronizes a large amount of transaction data to the kafka, the transaction data buffer module 110 pulls the transaction data from the kafka in batch to perform processing statistics, so that the large-flow peak of the transaction system is avoided, and meanwhile, the decoupling is performed, and the transaction data are asynchronously counted.
Step 125, persisting the statistical data to the redis and deleting the same batch of transaction data. After the system counts the transaction data, the statistical data (i.e., the statistical result) is written into the redis, that is, the transaction data (i.e., the transaction data used for generating the statistical result) representing that the batch is successfully consumed, and the transaction data stored in the redis is deleted.
Referring to fig. 5, an embodiment of the present application provides a transaction data processing apparatus, which is applied to a transaction data processing system including a first server and an application server, as shown in fig. 5, the apparatus 500 includes:
a first cache module 501, configured to obtain, by the first server, transaction data generated by the transaction system and cache the transaction data in a target queue of the transaction system;
a data pulling module 502, configured to pull at least one piece of transaction data from the target queue by the application server, where the application server is associated with the target queue;
the second caching module 503 is configured to send the at least one piece of transaction data to the second server for caching, and delete the at least one piece of transaction data in the target queue.
Therefore, in the transaction data processing process, the transaction data generated by the transaction system is cached through the first server, the application server is associated with the target queue of the first server for caching the transaction data, when the transaction data is processed by the application server, the transaction data can be pulled from the target queue of the first server, caching is carried out again in the second server, the pulled transaction data in the first server is deleted after the caching is finished, and the problems that the data in the first server is processed repeatedly or the data is lost and the like can be avoided. Therefore, by the transaction data processing method in the embodiment of the application, the transaction data generated by the transaction system is cached twice by the first server and the second server, rather than being directly sent to the application server for processing, and in a high-concurrency large-transaction-volume scene, overload operation caused by overlarge transaction data volume passively received by the application server can be avoided, so that the phenomenon of data loss caused by collapse of the application server is reduced, and the stability of the application server in the process of processing the transaction data is further improved.
In some embodiments, the second cache module 503 is specifically configured to:
and the application server filters the at least one piece of transaction data and sends the filtered effective transaction data to the second server for caching.
Based on this, the pulled transaction data is filtered and then cached, so that the redundancy of the data can be reduced, and the validity of the transaction data is ensured.
In some embodiments, the apparatus 500 further comprises:
and the data statistics module is used for acquiring the effective transaction data from the second server by the application server, and performing statistics on the effective transaction data to generate a statistical result.
Based on this, by counting the valid transaction data, the validity of the statistical result can be ensured.
In some embodiments, the apparatus 500 further comprises:
a statistical result transmission module, configured to transmit the statistical result to the second server by the application server;
and the statistical result caching module is used for caching the statistical result by the second server and deleting the effective transaction data cached by the second server.
Based on the above, the application server can cache the statistical result in the second server, and delete the effective transaction data for obtaining the statistical result through statistics, so that the user can inquire the statistical data, and the redundancy of the data in the second server can be reduced.
In some embodiments, the apparatus 500 further comprises:
an offset generation module, configured to generate, by the application server, a target offset of the transaction data cached to the second server;
an offset sending module, configured to send the target offset to the first server by the application server;
the second cache module 503 is specifically configured to:
the first server updates the offset of the transaction data pulled by the application server based on the target offset to delete the at least one transaction data.
Based on this, the application server may generate a target offset of the cached transaction data after caching the pulled transaction data in the second server, and update the offset automatically generated by the first server through the target offset to delete the transaction data corresponding to the updated offset, so as to improve the accuracy of the offset acquired by the first server, and further reduce the possibility of missing or repeated pulling of the transaction data.
In some embodiments, the apparatus 500 further comprises:
the capability information acquisition module is used for acquiring data processing capability information by the application server;
a rate determination module, configured to determine, by the application server, a data pull rate based on the data processing capability information;
the data pulling module is specifically configured to:
and the application server pulls at least one piece of transaction data from the target queue according to the data pulling rate.
Based on this, in the process of pulling transaction data from the target queue, the application server can determine the data pulling rate through the data processing capacity information of the application server, and pull the transaction data from the target queue according to the determined data pulling rate, so that the application server can pull a proper amount of transaction data according to the data processing capacity of the application server and determine whether to stop pulling the transaction data (namely, the data pulling rate is zero), the controllability of the transaction data pulling process is realized, and the stability in the transaction data statistics is further improved.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Based on the same inventive concept, the embodiment of the present disclosure further provides a computing device, which is specifically described in detail with reference to fig. 6.
FIG. 6 is a block diagram illustrating a computing device, according to an example embodiment.
As shown in fig. 6, the computing device 600 is capable of implementing a method for processing transaction data and a block diagram of an exemplary hardware architecture of a computing device of a transaction data processing apparatus according to the embodiments of the present disclosure. The computing device may refer to a transaction data processing system in an embodiment of the disclosure, and the computing device 600 includes a first server, an application server, and a second server in an embodiment of the disclosure.
The computing device 600 may comprise a processor 601 and a memory 602 storing computer program instructions, the processor 601 being understood to comprise a processor of a first server, a processor of an application server and a processor of a second server; likewise, the memory 602 may be understood to include the memory of a first server, the memory of an application server, and the memory of a second server.
Specifically, each processor may include a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
The various memories described above may include mass storage for information or instructions. By way of example, and not limitation, memory may include a Hard Disk Drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. The memory may include removable or non-removable (or fixed) media, where appropriate. The memory may be internal or external to the integrated gateway device, where appropriate. In a particular embodiment, the memory is non-volatile solid-state memory. In a particular embodiment, the memory includes Read Only Memory (ROM). Where appropriate, the ROM may be mask-programmed ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.
The processor 601, by reading and executing the computer program instructions stored in the memory 602, performs the following steps:
the processor 601 executes the first server to acquire the transaction data generated by the transaction system and buffer the transaction data in the target queue of the transaction system;
the application server pulling at least one piece of transaction data from the target queue, wherein the application server is associated with the target queue;
and the application server sends the at least one piece of transaction data to the second server for caching, and deletes the at least one piece of transaction data in the target queue.
Therefore, in the transaction data processing process, the transaction data generated by the transaction system is cached through the first server, the application server is associated with the target queue of the first server for caching the transaction data, when the transaction data is processed by the application server, the transaction data can be pulled from the target queue of the first server, caching is carried out again in the second server, the pulled transaction data in the first server is deleted after the caching is finished, and the problems that the data in the first server is processed repeatedly or the data is lost and the like can be avoided. Therefore, by the transaction data processing method in the embodiment of the application, the transaction data generated by the transaction system is cached twice by the first server and the second server, rather than being directly sent to the application server for processing, and in a high-concurrency large-transaction-volume scene, overload operation caused by overlarge transaction data volume passively received by the application server can be avoided, so that the phenomenon of data loss caused by collapse of the application server is reduced, and the stability of the application server in the process of processing the transaction data is further improved.
In some embodiments, the processor 601 specifically executes the application server to filter the at least one transaction data, and send the filtered valid transaction data to the second server for caching.
Based on this, the pulled transaction data is filtered and then cached, so that the redundancy of the data can be reduced, and the validity of the transaction data is ensured.
In some embodiments, the processor 601 further executes the application server to obtain the valid transaction data from the second server, and perform statistics on the valid transaction data to generate a statistical result.
Based on this, by counting the valid transaction data, the validity of the statistical result can be ensured.
In some embodiments, processor 601, further executes the application server to transmit the statistical result to the second server;
and the second server caches the statistical result and deletes the effective transaction data cached by the second server.
Based on the above, the application server can cache the statistical result in the second server, and delete the effective transaction data for obtaining the statistical result through statistics, so that the user can inquire the statistical data, and the redundancy of the data in the second server can be reduced.
In some embodiments, processor 601, further executes the application server to generate a target offset for the transaction data cached to the second server;
the application server sends the target offset to the first server;
the deleting the at least one transaction datum in the target queue further comprises:
the first server updates the offset of the transaction data pulled by the application server based on the target offset to delete the at least one transaction data.
Based on this, the application server may generate a target offset of the cached transaction data after caching the pulled transaction data in the second server, and update the offset automatically generated by the first server through the target offset to delete the transaction data corresponding to the updated offset, so as to improve the accuracy of the offset acquired by the first server, and further reduce the possibility of missing or repeated pulling of the transaction data.
In some embodiments, the processor 601:
the application server acquires data processing capacity information;
the application server determines a data pulling rate based on the data processing capacity information;
and the application server pulls at least one piece of transaction data from the target queue according to the data pulling rate.
Based on this, in the process of pulling transaction data from the target queue, the application server can determine the data pulling rate through the data processing capacity information of the application server, and pull the transaction data from the target queue according to the determined data pulling rate, so that the application server can pull a proper amount of transaction data according to the data processing capacity of the application server and determine whether to stop pulling the transaction data (namely, the data pulling rate is zero), the controllability of the transaction data pulling process is realized, and the stability in the transaction data statistics is further improved.
The embodiment of the disclosure also provides a computer storage medium, in which computer-executable instructions are stored, and the computer-executable instructions are used for implementing the transaction data processing method described in the embodiment of the disclosure.
Embodiments of the present disclosure also provide a computer program product comprising a computer program/instructions which, when executed by a processor, implement the method of processing transaction data according to the first aspect.
Wherein the computer program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus and computer program products according to the present disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable transaction data statistics apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable transaction data statistics apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable transaction data accounting device to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable statistical apparatus for transaction data to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (14)
1. A transaction data processing method is applied to a transaction data processing system comprising a first server, an application server and a second server, and the method comprises the following steps:
the first server acquires transaction data generated by the transaction system and caches the transaction data in a target queue of the first server;
the application server pulling at least one piece of transaction data from the target queue, wherein the application server is associated with the target queue;
and the application server sends the at least one piece of transaction data to the second server for caching, and deletes the at least one piece of transaction data in the target queue.
2. The method of claim 1, wherein the sending, by the application server, the at least one transaction datum to the second server for caching comprises:
and the application server filters the at least one piece of transaction data and sends the filtered effective transaction data to the second server for caching.
3. The method of claim 2, further comprising:
and the application server acquires the effective transaction data from the second server, and counts the effective transaction data to generate a statistical result.
4. The method of claim 3, wherein the application server obtains the valid transaction data from the second server, performs statistics on the valid transaction data, and after generating a statistical result, the method further comprises:
the application server transmits the statistical result to the second server;
and the second server caches the statistical result and deletes the effective transaction data cached by the second server.
5. The method of claim 1, wherein after the application server sends the at least one transaction datum to the second server for caching, the method further comprises:
the application server generating a target offset of the transaction data cached to the second server;
the application server sends the target offset to the first server;
the deleting the at least one transaction datum in the target queue further comprises:
the first server updates the offset of the transaction data pulled by the application server based on the target offset to delete the at least one transaction data.
6. The method of claim 1, wherein before the application server pulls at least one transaction datum from the target queue, the method further comprises:
the application server acquires data processing capacity information;
the application server determines a data pulling rate based on the data processing capacity information;
the application server pulls at least one piece of transaction data from the target queue, and the pulling comprises the following steps:
and the application server pulls at least one piece of transaction data from the target queue according to the data pulling rate.
7. An apparatus for processing transaction data, applied to a transaction data processing system including a first server, an application server and a second server, the apparatus comprising:
the first cache module is used for the first server to acquire transaction data generated by the transaction system and cache the transaction data in a target queue of the first server;
a data pull module, configured to pull at least one piece of transaction data from the target queue by the application server, where the application server is associated with the target queue;
and the second cache module is used for sending the at least one transaction data to the second server for caching by the application server and deleting the at least one transaction data in the target queue.
8. The apparatus of claim 7, wherein the second cache module is specifically configured to:
and the application server filters the at least one piece of transaction data and sends the filtered effective transaction data to the second server for caching.
9. The apparatus of claim 8, further comprising:
and the data statistics module is used for acquiring the effective transaction data from the second server by the application server, and performing statistics on the effective transaction data to generate a statistical result.
10. The apparatus of claim 9, further comprising:
a statistical result transmission module, configured to transmit the statistical result to the second server by the application server;
and the statistical result caching module is used for caching the statistical result by the second server and deleting the effective transaction data cached by the second server.
11. The apparatus of claim 7, further comprising:
an offset generation module, configured to generate, by the application server, a target offset of the transaction data cached to the second server;
an offset sending module, configured to send the target offset to the first server by the application server;
the second cache module is specifically configured to:
the first server updates the offset of the transaction data pulled by the application server based on the target offset to delete the at least one transaction data.
12. The apparatus of claim 7, further comprising:
the capability information acquisition module is used for acquiring data processing capability information by the application server;
a rate determination module, configured to determine, by the application server, a data pull rate based on the data processing capability information;
the data pulling module is specifically configured to:
and the application server pulls at least one piece of transaction data from the target queue according to the data pulling rate.
13. A transaction data processing system, characterized in that the system comprises a processor, a memory and a program or instructions stored on the memory and executable on the processor, which program or instructions, when executed by the processor, implement the steps of the method of processing transaction data according to any of claims 1 to 6.
14. A readable storage medium, on which a program or instructions are stored, which when executed by a processor implement the steps of the method of processing transaction data according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110336438.7A CN113034283A (en) | 2021-03-29 | 2021-03-29 | Transaction data processing method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110336438.7A CN113034283A (en) | 2021-03-29 | 2021-03-29 | Transaction data processing method, device and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113034283A true CN113034283A (en) | 2021-06-25 |
Family
ID=76452772
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110336438.7A Pending CN113034283A (en) | 2021-03-29 | 2021-03-29 | Transaction data processing method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113034283A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114296910A (en) * | 2021-12-02 | 2022-04-08 | 武汉物易云通网络科技有限公司 | Multi-partition queue sequential consumption method and device |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2002014865A (en) * | 2000-04-28 | 2002-01-18 | Fujitsu Ltd | Device for switching object control method and method for the same |
| KR20150040172A (en) * | 2013-10-04 | 2015-04-14 | 주식회사 솔박스 | Method for caching contents based on content request record and request priority and HTTP cache server using the method |
| CN106528792A (en) * | 2016-11-10 | 2017-03-22 | 福州智永信息科技有限公司 | Big data acquisition and high-speed processing method and system based on multi-layer caching mechanism |
| CN110196769A (en) * | 2018-04-19 | 2019-09-03 | 财付通支付科技有限公司 | A kind of information processing method, device, server and storage medium |
| CN111078697A (en) * | 2019-12-11 | 2020-04-28 | 秒针信息技术有限公司 | Data storage method and device, storage medium and electronic equipment |
| CN111629019A (en) * | 2019-08-13 | 2020-09-04 | 广州凡科互联网科技股份有限公司 | Method for asynchronously processing big data and high concurrency |
| CN111880947A (en) * | 2020-07-20 | 2020-11-03 | 中国银联股份有限公司 | Data transmission method and device |
| CN112486707A (en) * | 2020-12-14 | 2021-03-12 | 中国人民银行清算总中心 | Redis-based message asynchronous consumption method and device |
-
2021
- 2021-03-29 CN CN202110336438.7A patent/CN113034283A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2002014865A (en) * | 2000-04-28 | 2002-01-18 | Fujitsu Ltd | Device for switching object control method and method for the same |
| KR20150040172A (en) * | 2013-10-04 | 2015-04-14 | 주식회사 솔박스 | Method for caching contents based on content request record and request priority and HTTP cache server using the method |
| CN106528792A (en) * | 2016-11-10 | 2017-03-22 | 福州智永信息科技有限公司 | Big data acquisition and high-speed processing method and system based on multi-layer caching mechanism |
| CN110196769A (en) * | 2018-04-19 | 2019-09-03 | 财付通支付科技有限公司 | A kind of information processing method, device, server and storage medium |
| CN111629019A (en) * | 2019-08-13 | 2020-09-04 | 广州凡科互联网科技股份有限公司 | Method for asynchronously processing big data and high concurrency |
| CN111078697A (en) * | 2019-12-11 | 2020-04-28 | 秒针信息技术有限公司 | Data storage method and device, storage medium and electronic equipment |
| CN111880947A (en) * | 2020-07-20 | 2020-11-03 | 中国银联股份有限公司 | Data transmission method and device |
| CN112486707A (en) * | 2020-12-14 | 2021-03-12 | 中国人民银行清算总中心 | Redis-based message asynchronous consumption method and device |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114296910A (en) * | 2021-12-02 | 2022-04-08 | 武汉物易云通网络科技有限公司 | Multi-partition queue sequential consumption method and device |
| CN114296910B (en) * | 2021-12-02 | 2022-11-22 | 武汉物易云通网络科技有限公司 | Multi-partition queue sequential consumption method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020258665A1 (en) | Process guarantee method and device for distributed transactions | |
| US7624112B2 (en) | Asynchronously storing transaction information from memory to a persistent storage | |
| WO2019141186A1 (en) | Data processing method and device | |
| JP2003516581A (en) | Continuous flow checkpoint setting data processing method | |
| WO2021135745A1 (en) | Method and apparatus for carrying out transaction data processing at blockchain node | |
| CN109615514A (en) | Hot spot account trading system and method | |
| CN101604437A (en) | Account is real time processing system and account batch real-time processing method in batches | |
| CN110188110A (en) | A kind of method and device constructing distributed lock | |
| CN113034283A (en) | Transaction data processing method, device and system | |
| CN111897838A (en) | Transaction query method and device, electronic equipment and readable storage medium thereof | |
| CN111177158B (en) | Method for matching order with stock consistently | |
| CN106716369B (en) | Variable timing schedule to accommodate randomly occurring event delays | |
| CN109583873B (en) | Transaction communication method, server, POS machine and electronic equipment | |
| KR101499890B1 (en) | Low Latency Framework System | |
| CN113672401A (en) | Triggering method and system for batch processing tasks and computer readable storage medium | |
| CN107194712B (en) | Method and device for recording change information of shared account and method and system for supplementing account of internal account | |
| CN110297824B (en) | Data recording method, device, equipment and storage medium based on resource transfer | |
| CN115907949A (en) | Bank transaction data processing method and device | |
| CN116028530A (en) | Object resource reading method and device, electronic equipment and readable storage medium | |
| CN111681008B (en) | Risk control method, apparatus, device and storage medium | |
| US20210240698A1 (en) | Asynchronous remote calls with undo data structures | |
| CN112650568A (en) | Merchant netting difference settlement method, system, computer equipment and storage medium | |
| CN112162988A (en) | Distributed transaction processing method and device and electronic equipment | |
| CN110457116B (en) | Method and device for processing transaction request | |
| JP2001125749A (en) | Recording medium with recorded external storage device driver program and computer having external storage device access function |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |