CN109034993B - Account checking method, account checking equipment, account checking system and computer readable storage medium - Google Patents
Account checking method, account checking equipment, account checking system and computer readable storage medium Download PDFInfo
- Publication number
- CN109034993B CN109034993B CN201811153706.6A CN201811153706A CN109034993B CN 109034993 B CN109034993 B CN 109034993B CN 201811153706 A CN201811153706 A CN 201811153706A CN 109034993 B CN109034993 B CN 109034993B
- Authority
- CN
- China
- Prior art keywords
- transaction
- data
- reconciliation
- flow data
- distributed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/12—Accounting
Abstract
The invention discloses an account checking method, which comprises the following steps: reading transaction flow data generated by a transaction system in real time; extracting transaction data to be checked from the transaction flow data; acquiring transaction identification information from the extracted transaction data to be checked, and grouping the extracted transaction data to be checked according to the transaction identification information, wherein the transaction identification information is used for uniquely identifying one transaction; and performing reconciliation processing on the transaction data to be reconciled in each group. The invention also discloses account checking equipment, a system and a computer readable storage medium. The invention improves the real-time performance of transaction reconciliation.
Description
Technical Field
The invention relates to the technical field of big data processing, in particular to a reconciliation method, device and system and a computer-readable storage medium.
Background
With the advent of the big data era, big data related application shows explosive growth, particularly for a large-scale transaction system, massive transaction data needs to be processed every day, and under the condition, how to analyze and process transaction data generated by the transaction system is required, so that the problem that the transaction reconciliation requirement is urgently required to be solved at present is met.
In the existing financial network, the account checking scheme of the transaction system is mainly used for carrying out batch processing account checking on offline transaction data, for example, a 'T + 1' day mode, the transaction data is collected in the first day, and the account checking can be carried out on the transaction data in the next day.
Disclosure of Invention
The invention mainly aims to provide a reconciliation method, a device, a system and a computer-readable storage medium, aiming at improving the real-time performance of transaction reconciliation.
In order to achieve the above object, the present invention provides an account checking method, which comprises the following steps:
reading transaction flow data generated by a transaction system in real time;
extracting transaction data to be checked from the transaction flow data;
acquiring transaction identification information from the extracted transaction data to be checked, and grouping the extracted transaction data to be checked according to the transaction identification information, wherein the transaction identification information is used for uniquely identifying one transaction;
and performing reconciliation processing on the transaction data to be reconciled in each group.
Preferably, the step of reading transaction flow data generated by the transaction system in real time comprises:
reading transaction flow data generated by a transaction system in real time from a preset message acquisition node, wherein the message acquisition node is used for acquiring the transaction flow data generated by the transaction system in real time;
or, reading transaction flow data generated by the transaction system in real time from a distributed message queue, wherein the distributed message queue is used for caching the transaction flow data sent by the message acquisition node.
Preferably, the transaction flow data is obtained by synchronizing a transaction log file from a slave node of a data center node corresponding to the transaction system by the message acquisition node, and analyzing and standardizing the transaction log file.
Preferably, the step of reading transaction flow data generated by the transaction system from the distributed message queue in real time comprises:
creating a sliding window and setting the sliding step length of the sliding window;
and reading transaction flow data generated by the transaction system from the distributed message queue based on the sliding window and the sliding step length, wherein each sliding window is used for collecting the transaction flow data generated by the transaction system within a preset time length.
Preferably, after the step of reading the transaction pipeline data generated by the transaction system from the distributed message queue based on the sliding window and the sliding step size, the method further includes:
recording the data latency and the data waiting times of the transaction running data through a preset check point in the sliding process of the sliding window;
when the data latency reaches a preset maximum latency, executing the following steps: extracting transaction data to be checked from the transaction flow data;
deleting the check point when the data waiting times reach a preset maximum latency;
wherein the data latency and the maximum latency are represented in a count.
Preferably, the step of extracting the transaction data to be reconciled from the transaction flow data comprises:
converting the transaction flow data into a distributed data set;
splitting the distributed data set according to a database table of the transaction flow data source to obtain a plurality of sub-distributed data sets;
extracting a data field value to be checked from the sub-distributed data set, and converting the extracted data field value into a preset standard data format;
carrying out duplicate removal processing on the data field value converted into the standard data format to obtain a data field value corresponding to a preset final transaction state;
and summarizing the data field values subjected to the deduplication processing in each sub-distributed data set to obtain a new distributed data set, and taking the data in the new distributed data set as transaction data to be checked.
Preferably, the step of reconciling the transaction data to be reconciled in each group includes:
and in each group, reconciliation processing is carried out on the transaction data to be reconciled according to the transaction amount and the transaction state.
In addition, to achieve the above object, the present invention further provides a reconciliation apparatus, including: the system comprises a memory, a processor and a reconciliation program stored on the memory and capable of running on the processor, wherein the reconciliation program realizes the steps of the reconciliation method when being executed by the processor.
In addition, in order to achieve the above object, the present invention further provides an account checking system, where the account checking system includes an account checking device and a message collecting node, where:
the account checking equipment is the account checking equipment;
the message acquisition node is used for acquiring transaction flow data generated by the transaction system in real time so that the reconciliation equipment can read the transaction flow data generated by the transaction system through the message acquisition node.
Preferably, the message collection node is further configured to synchronize a transaction log file generated by a slave node in a data center node of the transaction system, and the transaction log file records transaction flow data generated by the transaction system;
analyzing and standardizing the transaction flow data in the transaction log file, and storing the analyzed and standardized transaction flow data in a local storage area or sending the analyzed and standardized transaction flow data to a distributed message queue so that a reconciliation device can read the transaction flow data generated by a transaction system from a message acquisition node or the distributed message queue.
Furthermore, to achieve the above object, the present invention also provides a computer readable storage medium having a reconciliation program stored thereon, which when executed by a processor implements the steps of the reconciliation method as described above.
The invention reads the transaction flow data generated by the transaction system in real time; extracting transaction data to be checked from the transaction flow data; acquiring transaction identification information from the extracted transaction data to be checked, and grouping the extracted transaction data to be checked according to the transaction identification information, wherein the transaction identification information is used for uniquely identifying one transaction; and performing reconciliation processing on the transaction data to be reconciled in each group. Because the transaction running data is read in real time and the transaction identification information is used for uniquely identifying one transaction, the invention can realize real-time account checking of each transaction generated by the transaction system by the mode and improve the real-time performance of transaction account checking.
Drawings
FIG. 1 is a schematic diagram of an apparatus architecture of a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a reconciliation method according to a first embodiment of the present invention;
FIG. 3 is a schematic diagram of a detailed step of step S20 in FIG. 2;
FIG. 4 is a diagram illustrating the collection of transaction flow data via a sliding window in accordance with an embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The main solution of the embodiment of the invention is as follows: reading transaction flow data generated by a transaction system in real time; extracting transaction data to be checked from the transaction flow data; acquiring transaction identification information from the extracted transaction data to be checked, and grouping the extracted transaction data to be checked according to the transaction identification information, wherein the transaction identification information is used for uniquely identifying one transaction; and performing reconciliation processing on the transaction data to be reconciled in each group.
In the existing financial network, the account checking scheme of the transaction system is mainly used for carrying out batch processing account checking on offline transaction data, for example, a 'T + 1' day mode, the transaction data is collected in the first day, and the account checking can be carried out on the transaction data in the next day.
The invention reads the transaction flow data generated by the transaction system in real time; extracting transaction data to be checked from the transaction flow data; acquiring transaction identification information from the extracted transaction data to be checked, and grouping the extracted transaction data to be checked according to the transaction identification information, wherein the transaction identification information is used for uniquely identifying one transaction; and performing reconciliation processing on the transaction data to be reconciled in each group. Because the transaction running data is read in real time and the transaction identification information is used for uniquely identifying one transaction, the invention can realize real-time account checking of each transaction generated by the transaction system by the mode and improve the real-time performance of transaction account checking.
As shown in fig. 1, fig. 1 is a schematic device structure diagram of a hardware operating environment according to an embodiment of the present invention.
The account checking equipment in the embodiment of the invention can be server, PC machine or virtual machine equipment.
As shown in fig. 1, the reconciliation device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a storage device separate from the processor 1001.
Those skilled in the art will appreciate that the configuration of the apparatus shown in fig. 1 is not intended to be limiting of the apparatus and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is a kind of computer storage medium, may include therein an operating system, a network communication module, a user interface module, and a reconciliation program.
In the device shown in fig. 1, the network interface 1004 is mainly used for connecting to a backend server and performing data communication with the backend server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and the processor 1001 may be configured to call a reconciliation program stored in the memory 1005 and perform operations in the embodiments of the reconciliation method described below.
Based on the hardware structure, the embodiment of the reconciliation method is provided.
Referring to fig. 2, fig. 2 is a schematic flowchart of a reconciliation method according to a first embodiment of the present invention, where the reconciliation method includes:
step S10, reading transaction flow data generated by the transaction system in real time;
the account checking method is applied to account checking equipment in an account checking system running on a big data platform, wherein the big data platform can be a Spark platform which is a popular big data calculation and statistics platform at present, and the Spark platform can realize various machine learning and data mining by calculating and counting big data; the transaction system can be a business system of financial institutions such as banks, security companies, insurance companies and the like, and is used for processing and recording various financial businesses such as savings, account transfer, investment and the like of a user and generating corresponding transaction flow data. The reconciliation device includes distributed streaming computing components including, but not limited to: spark Streaming, Storm, Samza, Flink and Kafka Streams.
The step S10 may specifically include: reading transaction flow data generated by a transaction system in real time from a preset message acquisition node, wherein the message acquisition node is used for acquiring the transaction flow data generated by the transaction system in real time;
or, reading transaction flow data generated by the transaction system in real time from a distributed message queue, wherein the distributed message queue is used for caching the transaction flow data sent by the message acquisition node.
It should be noted that, in the above two modes, the message acquisition node acquires the transaction flow data for the reconciliation device to directly read, or the message acquisition node transmits the transaction flow data to the distributed message queue for the reconciliation device to read in the distributed message queue, and both the two modes do not need to change the code of the transaction system, do not invade the transaction system, and are more convenient to access.
In other embodiments, before the step S10, the code may also be modified in the transaction system, so as to store the transaction flow data in the distributed message queue in real time, so that the reconciliation device can read from the distributed message queue in real time;
optionally, the transaction running data is written into a local log file in the transaction system, and an agent (flash agent) is deployed on a host of the transaction system to monitor the local log file, collect the transaction running data in the log file in real time, and write the transaction running data into the distributed message queue, so that the reconciliation equipment can read the transaction running data from the distributed message queue in real time.
In this embodiment, it is preferable that transaction flow data generated by the transaction system is acquired in real time through a message acquisition node, and after the transaction flow data is acquired by the message acquisition node, the transaction flow data is stored in a preset space of the message acquisition node or sent to a distributed message queue, so that the reconciliation equipment can read from the message acquisition node and also can read from the distributed message queue, where the reading after the data is cached in the distributed message queue is advantageous in that when the reconciliation equipment is recovered after a fault, the data is directly read from the distributed message queue, and it is possible to prevent the data from being over-stored in the message acquisition node. In this case, the reconciliation equipment is equivalent to be in butt joint with the trading system through the distributed message queue middleware, and can acquire the trading flow data generated by the trading system in real time.
Further, the transaction flow data can be obtained by synchronizing a transaction log file from a slave node of a data center node corresponding to the transaction system by the message collection node, and analyzing and standardizing the transaction log file.
In particular implementation, the message collection node may be disguised as a slave node of a slave node in a data center node of the transaction system, and synchronizes a transaction log file generated by the slave node in the data center node of the transaction system, where the transaction log file may be a Binlog log file, in which transaction flow data generated by the transaction system is recorded. After the transaction flow data are acquired, the message acquisition node analyzes and standardizes the transaction flow data in the transaction log file, and stores the analyzed and standardized transaction flow data in a local storage area or sends the analyzed and standardized transaction flow data to a distributed message queue, so that the reconciliation equipment can read the transaction flow data generated by the transaction system from the message acquisition node or the distributed message queue and perform subsequent reconciliation operation.
Specifically, taking a distributed message queue Apache Kafka (a distributed publish-subscribe message system, which is mainly used for processing active streaming data) as an example, transaction streaming data generated by a transaction system is stored in binlog (a binary log file used for recording SQL statement information updated by a user on a database), a message collection node analyzes the binlog to obtain standardized transaction streaming data, and the standardized transaction streaming data is sent to the Kafka message queue, wherein a process of analyzing the data into the standardized transaction streaming data by the message collection node is as follows: determining each field according to binary data bits, then determining each attribute (such as a dcn number, a table name and a library name) corresponding to each field to obtain standardized transaction flow data, and reading the transaction flow data from the Kafka message queue by the account checking equipment, wherein the part is real-time.
It should be noted that the transaction system database may further include a backup library corresponding thereto, where the backup library is used to backup and store the transaction data, and when the Apache Kafka reads the online streaming data, the online streaming data may be read from the backup library, so as to avoid occupying transaction system resources. Besides, in addition to Apache Kafka, the distributed Message Queue in the embodiment of the present invention may also be another type of distributed Message Queue, such as an ActiveMQ (Message Queue, Message bus), RabbitMQ, rocktmq, and the like, and may be flexibly set in specific implementation.
Step S20, transaction data to be checked out are extracted from the transaction flow data;
in this step, the reconciliation equipment extracts transaction data to be reconciled from the acquired transaction flow data, where the transaction data to be reconciled includes, but is not limited to, data such as transaction amount, transaction status, and transaction identifier (e.g., transaction flow number).
Specifically, referring to fig. 3, fig. 3 is a schematic diagram illustrating a detailed step of step S20 in fig. 2, where the step S20 may include:
step S21, converting the transaction flow data into a distributed data set;
step S22, splitting the distributed data set according to the database table of the transaction flow data source to obtain a plurality of sub-distributed data sets;
step S23, extracting the data field value to be checked from the sub-distributed data set, and converting the extracted data field value into a preset standard data format;
step S24, carrying out duplication elimination processing on the data field value converted into the standard data format to obtain a data field value corresponding to the preset final transaction state;
and step S25, summarizing the data field values after the deduplication processing in each sub-distributed data set to obtain a new distributed data set, and taking the data in the new distributed data set as the transaction data to be reconciled.
In this embodiment, the reconciliation device reads and processes transaction running data through the distributed stream computing component, and the distributed stream computing component is Spark Streaming as an example for explanation. The step of extracting the transaction data to be checked from the transaction stream data by Spark Streaming comprises the following steps:
1. spark Streaming converts the read transaction pipeline data into a distributed data set. The Spark Streaming may first convert the read transaction flow data into an elastic Distributed data set rdd (resource Distributed data set), and then encapsulate a layer of data structure in the elastic Distributed data set to obtain a Distributed data set DStream.
2. The separation was as shown. The DStream has transaction running data of different database tables, and the data structures of the transaction running data are different, so that the DStream can be split according to the difference of the database tables of the transaction running data sources to obtain a plurality of sub-DStreams, wherein the data structures of the transaction running data in each sub-DStream are the same.
3. And (6) standardizing data. The spare Streaming extracts a data field value to be checked from each sub-DSstream according to a preset extraction rule corresponding to the data structure, and converts the extracted data field value into a preset standard data format (such as JSON format), thereby realizing the unification of the data structures of different sub-DSstreams.
4. And removing the heavy matter and taking the final state. Because data in a database table is subjected to the processes of creation and multiple updates, multiple records may exist in the same data in the Binlog, but for account checking equipment, only final-state data is concerned, so that the step needs to perform deduplication processing on a data field value converted into a standard data format to obtain a data field value corresponding to a preset final transaction state, wherein the final transaction state can be flexibly set according to business, such as transaction success, transaction failure and the like.
5. And (6) merging the data. The account checking needs to compare data from different database tables, so that data in each sub Dstream needs to be summarized into one Dstream, so as to obtain a new distributed data set, and the data in the new distributed data set is used as transaction data to be checked.
It should be noted that, in addition to Spark Streaming, the distributed Streaming components used above may also be replaced by the following similar Streaming components: storm, Samza, Flink, Kafka Streams.
S30, acquiring transaction identification information from the extracted transaction data to be checked, and grouping the extracted transaction data to be checked according to the transaction identification information, wherein the transaction identification information is used for uniquely identifying one transaction;
in the step, the transaction data to be checked can be grouped based on the transaction identification information, such as the transaction serial number, and the transaction identification information is used for uniquely identifying one transaction, so that the step can realize that the transaction data to be checked can be divided according to different transactions.
In addition, it should be noted that, after grouping the transaction data to be checked, the method further includes:
if only one transaction flow data is detected in the grouped transaction data, which indicates that the transaction flow data of another transaction system is not acquired currently, the transaction flow data is cached in a preset storage area at the moment, so that the two transaction flow data are detected in the group, and then the subsequent reconciliation is performed.
And step S40, performing reconciliation processing on the transaction data to be reconciled in each group.
In the step, account checking processing is carried out on the transaction data to be checked in each group. As an embodiment, the step may include: in each group, account checking processing is carried out on the transaction data to be checked according to the transaction amount and the transaction state, namely, the account checking equipment obtains an account checking result by comparing the transaction amount and the transaction state in the same transaction data. Of course, other transaction data, such as transaction types, transaction time and the like, can be compared according to actual needs, so that an account checking result can be obtained, and the method can be flexibly set in specific implementation.
In the embodiment, the account checking device reads transaction flow data generated by the transaction system in real time; extracting transaction data to be checked from the transaction flow data; acquiring transaction identification information from the extracted transaction data to be checked, and grouping the extracted transaction data to be checked according to the transaction identification information, wherein the transaction identification information is used for uniquely identifying one transaction; and performing reconciliation processing on the transaction data to be reconciled in each group. Because the embodiment reads the transaction running data in real time and the transaction identification information is used for uniquely identifying one transaction, the invention can realize real-time account checking of each transaction generated by the transaction system by the mode and improve the real-time performance of transaction account checking.
Further, based on the first embodiment of the reconciliation method of the present invention, a second embodiment of the reconciliation method of the present invention is provided.
In this embodiment, the step of reading transaction flow data generated by the transaction system from the distributed message queue in real time may include:
creating a sliding window and setting the sliding step length of the sliding window;
and reading transaction flow data generated by the transaction system from the distributed message queue based on the sliding window and the sliding step length, wherein each sliding window is used for collecting the transaction flow data generated by the transaction system within a preset time length.
In this embodiment, it is considered that Spark Streaming performs data calculation once in a time unit, but the Streaming data of the same transaction may not arrive in the same time unit due to network delay and the like, and at this time, a problem of data boundary occurs when processing data, and for this reason, the transaction Streaming data generated by the transaction system may be read from the distributed message queue in a sliding window manner.
Referring to fig. 4, fig. 4 is a schematic diagram illustrating the collection of transaction flow data through a sliding window according to an embodiment of the present invention. The length of the sliding Window is fixed and at least includes 2 latches (minimum acquisition time unit of the original data stream), for example, the sliding Window may include 3 latches, and the sliding step size of the sliding Window may be set to 1 latch, at this time, every time the sliding Window slides by one step size, a corresponding Window data stream is obtained, as shown in windows 1, windows 2, and windows 3 in the figure. By the method, the problem of data boundary is solved, the real-time performance of account checking is ensured,
further, after the step of reading the transaction pipeline data generated by the transaction system from the distributed message queue based on the sliding window and the sliding step size, the method may further include: recording the data latency and the data waiting times based on the transaction running data through a preset check point in the sliding process of the sliding window; when the data latency reaches a preset maximum latency, executing the following steps: extracting transaction data to be checked from the transaction flow data; deleting the check point when the data waiting times reach a preset maximum latency; wherein the data latency and the maximum latency are represented in a count.
In this embodiment, since the sliding window may generate a situation of repeatedly calculating the same data for multiple times, in order to ensure reliability and timeliness of a final reconciliation result, a corresponding Checkpoint (a Checkpoint established by a calculation engine in the calculation process to avoid data loss, which is equivalent to a data snapshot) may be established for key indexes such as the waiting times, latency, maximum latency, and the like of transaction running data, and in the sliding window sliding process, the data latency and the data waiting times based on the transaction running data are recorded through the preset Checkpoint; when the data latency reaches the preset maximum latency, the transaction can be considered to reach the final state (transaction success, transaction failure and the like), the reconciliation condition is met, at the moment, the step of extracting the transaction data to be reconciled from the transaction stream data can be executed, and then the reconciliation operation is carried out, so that the transaction data participating in the reconciliation can be ensured to be the transaction data corresponding to the final state of the transaction, and the accuracy of the reconciliation result is ensured; when the data waiting times reach the preset maximum latency, the reconciliation can be considered to be finished, and the corresponding check point is deleted, so that unnecessary system resource occupation can be reduced.
The invention also provides account checking equipment.
The reconciliation equipment of the invention comprises: the system comprises a memory, a processor and a reconciliation program stored on the memory and capable of running on the processor, wherein the reconciliation program realizes the steps of the reconciliation method when being executed by the processor.
The method implemented when the account checking program running on the processor is executed may refer to each embodiment of the account checking method of the present invention, and details are not described here.
The invention also provides an account checking system.
In an embodiment of the reconciliation system of the present invention, the system comprises a reconciliation device and a message collection node, wherein:
the account checking equipment is the above account checking equipment;
the message acquisition node is used for acquiring transaction flow data generated by the transaction system in real time so that the reconciliation equipment can read the transaction flow data generated by the transaction system through the message acquisition node.
In particular implementation, the message collection node may be disguised as a slave node of a slave node in a data center node of the transaction system, and synchronizes a transaction log file generated by the slave node in the data center node of the transaction system, where the transaction log file may be a Binlog log file, in which transaction flow data generated by the transaction system is recorded.
After the transaction flow data are acquired, the message acquisition node analyzes and standardizes the transaction flow data in the transaction log file, and stores the analyzed and standardized transaction flow data in a local storage area or sends the analyzed and standardized transaction flow data to a distributed message queue, so that the reconciliation equipment can read the transaction flow data generated by the transaction system from the message acquisition node or the distributed message queue and perform subsequent reconciliation operation. The reconciliation operation performed by the reconciliation device can refer to each embodiment of the reconciliation method of the present invention, and details are not described herein.
The acquisition mode of the transaction flow data has no invasion to the transaction system, does not need to change any code in the transaction system, and is convenient to access.
In the embodiment, the transaction flow data is read in real time, and the transaction identification information is used for uniquely identifying one transaction, so that each transaction generated by the transaction system can be checked in real time through the mode, and the real-time performance of transaction checking is improved.
The invention also provides a computer readable storage medium.
The computer readable storage medium of the present invention has stored thereon a reconciliation program which, when executed by a processor, implements the steps of the reconciliation method as described above.
The method implemented when the account checking program running on the processor is executed may refer to each embodiment of the account checking method of the present invention, and details are not described here.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (9)
1. An account checking method is characterized by comprising the following steps:
reading transaction flow data generated by a transaction system in real time, wherein the transaction flow data is obtained by synchronizing a transaction log file from a slave node of a data center node corresponding to the transaction system by a message acquisition node and analyzing and standardizing the transaction log file;
extracting transaction data to be checked out from the transaction flow data, wherein the step of extracting the transaction data to be checked out from the transaction flow data comprises the following steps:
converting the transaction flow data into a distributed data set; splitting the distributed data set according to a database table of the transaction flow data source to obtain a plurality of sub-distributed data sets;
extracting a data field value to be checked from the sub-distributed data set, and converting the extracted data field value into a preset standard data format;
carrying out duplicate removal processing on the data field value converted into the standard data format to obtain a data field value corresponding to a preset final transaction state;
summarizing the data field values subjected to the deduplication processing in each sub-distributed data set to obtain a new distributed data set, and taking data in the new distributed data set as transaction data to be checked;
acquiring transaction identification information from the extracted transaction data to be checked, and grouping the extracted transaction data to be checked according to the transaction identification information, wherein the transaction identification information is used for uniquely identifying one transaction;
performing reconciliation processing on the transaction data to be reconciled in each group;
the transaction system maintains a transaction system database and is used for storing the service information in a plurality of storage devices in a distributed mode, the transaction system also maintains a standby database corresponding to the transaction system database, and the standby database is used for backing up and storing the transaction data so as to ensure that a large amount of service data are stored.
2. The reconciliation method of claim 1 wherein the step of reading in real time transaction flow data generated by the transaction system comprises:
reading transaction flow data generated by a transaction system in real time from a preset message acquisition node, wherein the message acquisition node is used for acquiring the transaction flow data generated by the transaction system in real time;
or, reading transaction flow data generated by the transaction system in real time from a distributed message queue, wherein the distributed message queue is used for caching the transaction flow data sent by the message acquisition node.
3. The reconciliation method of claim 2 wherein the step of reading transaction stream data generated by the transaction system from the distributed message queue in real time comprises:
creating a sliding window and setting the sliding step length of the sliding window;
and reading transaction flow data generated by the transaction system from the distributed message queue based on the sliding window and the sliding step length, wherein each sliding window is used for collecting the transaction flow data generated by the transaction system within a preset time length.
4. The reconciliation method of claim 3 wherein the step of reading transaction flow data generated by the transaction system from the distributed message queue based on the sliding window and the sliding step size is followed by further comprising:
recording the data latency and the data waiting times of the transaction running data through a preset check point in the sliding process of the sliding window;
when the data latency reaches a preset maximum latency, executing the following steps: extracting transaction data to be checked from the transaction flow data;
deleting the check point when the data waiting times reach a preset maximum latency;
wherein the data latency and the maximum latency are represented in a count.
5. The reconciliation method of claim 1, wherein the reconciliation processing of the transaction data to be reconciled in each group comprises:
and in each group, reconciliation processing is carried out on the transaction data to be reconciled according to the transaction amount and the transaction state.
6. A reconciliation apparatus, wherein the reconciliation apparatus comprises: a memory, a processor, and a reconciliation program stored on the memory and executable on the processor, the reconciliation program when executed by the processor implementing the steps of the reconciliation method of any of claims 1 to 5.
7. A reconciliation system, comprising a reconciliation device and a message collection node, wherein:
the reconciliation equipment is the reconciliation equipment of claim 6;
the message acquisition node is used for acquiring transaction flow data generated by the transaction system in real time so that the reconciliation equipment can read the transaction flow data generated by the transaction system through the message acquisition node.
8. The reconciliation system of claim 7,
the message acquisition node is also used for synchronizing transaction log files generated by slave nodes in a data center node of the transaction system, and transaction stream data generated by the transaction system is recorded in the transaction log files;
analyzing and standardizing the transaction flow data in the transaction log file, and storing the analyzed and standardized transaction flow data in a local storage area or sending the analyzed and standardized transaction flow data to a distributed message queue so that a reconciliation device can read the transaction flow data generated by a transaction system from a message acquisition node or the distributed message queue.
9. A computer-readable storage medium, having stored thereon a reconciliation program which, when executed by a processor, implements the steps of the reconciliation method of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811153706.6A CN109034993B (en) | 2018-09-29 | 2018-09-29 | Account checking method, account checking equipment, account checking system and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811153706.6A CN109034993B (en) | 2018-09-29 | 2018-09-29 | Account checking method, account checking equipment, account checking system and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109034993A CN109034993A (en) | 2018-12-18 |
| CN109034993B true CN109034993B (en) | 2022-04-01 |
Family
ID=64615561
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811153706.6A Active CN109034993B (en) | 2018-09-29 | 2018-09-29 | Account checking method, account checking equipment, account checking system and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109034993B (en) |
Families Citing this family (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109615495B (en) * | 2018-10-11 | 2023-08-18 | 创新先进技术有限公司 | Data reconciliation method, device, equipment and system |
| CN109783701A (en) * | 2018-12-20 | 2019-05-21 | 平安信托有限责任公司 | Identical flowing water determination method, device, computer equipment and storage medium |
| CN109816481A (en) * | 2019-01-04 | 2019-05-28 | 平安科技（深圳）有限公司 | Bill processing method, device and computer readable storage medium |
| CN109934693A (en) * | 2019-02-22 | 2019-06-25 | 东莞市盟大塑化科技有限公司 | Classification account checking method and system, storage medium, electronic equipment |
| CN110060140A (en) * | 2019-03-15 | 2019-07-26 | 深圳壹账通智能科技有限公司 | Mass data account checking method, device, medium and computer equipment |
| CN110084690A (en) * | 2019-03-18 | 2019-08-02 | 深圳壹账通智能科技有限公司 | Data processing method, device, computer installation and storage medium |
| CN110175165A (en) * | 2019-04-16 | 2019-08-27 | 平安普惠企业管理有限公司 | Transaction account checking method, device, computer equipment and storage medium |
| CN110298740A (en) * | 2019-06-24 | 2019-10-01 | 深圳乐信软件技术有限公司 | Data account checking method, device, equipment and storage medium |
| CN110378778A (en) * | 2019-07-10 | 2019-10-25 | 中信百信银行股份有限公司 | Multi-data source account checking method, system, electronic equipment and storage medium |
| CN110544164A (en) * | 2019-08-27 | 2019-12-06 | 中信百信银行股份有限公司 | Full link account checking method and system |
| CN110647544A (en) * | 2019-09-10 | 2020-01-03 | 四川新网银行股份有限公司 | Account checking method based on stream data |
| CN110880131A (en) * | 2019-11-11 | 2020-03-13 | 深圳前海微众银行股份有限公司 | Invoice generation method and device |
| CN110889754B (en) * | 2019-11-19 | 2023-07-11 | 中信百信银行股份有限公司 | Method for improving processing efficiency of non-overdraft hot spot account |
| CN110991871A (en) * | 2019-11-29 | 2020-04-10 | 深圳前海微众银行股份有限公司 | Risk monitoring method, device, equipment and computer readable storage medium |
| CN112419018B (en) * | 2020-01-03 | 2022-11-18 | 上海哔哩哔哩科技有限公司 | General data reconciliation method, server and storage medium in distributed environment |
| CN111899087A (en) * | 2020-06-16 | 2020-11-06 | 中国建设银行股份有限公司 | Data providing method and device, electronic equipment and computer readable storage medium |
| CN111695904A (en) * | 2020-06-24 | 2020-09-22 | 深圳壹账通智能科技有限公司 | Data risk control method and system and computer equipment |
| CN112116306A (en) * | 2020-07-01 | 2020-12-22 | 神思旭辉医疗信息技术有限责任公司 | Medical reconciliation method based on T +0/N combination |
| CN112328545B (en) * | 2020-11-03 | 2022-11-22 | 河北幸福消费金融股份有限公司 | Account checking file generation method, system, device and storage medium |
| CN112734544A (en) * | 2020-12-31 | 2021-04-30 | 四川新网银行股份有限公司 | Distributed real-time account checking method based on message queue |
| CN113448713B (en) * | 2021-07-12 | 2024-02-13 | 中国银行股份有限公司 | Asynchronous processing method, device, storage medium and equipment for stream data |
| CN113590602A (en) * | 2021-07-30 | 2021-11-02 | 上海微盟企业发展有限公司 | Message duplicate removal method, device, equipment and storage medium |
| CN115115433B (en) * | 2022-07-19 | 2023-09-05 | 深圳市享多多网络技术有限公司 | Order data processing method, device, computer equipment and storage medium |
| CN115115457B (en) * | 2022-08-29 | 2022-11-22 | 天津金城银行股份有限公司 | Method, system and medium for processing business transaction flow |
| CN117251120B (en) * | 2023-11-17 | 2024-03-01 | 杭州乒乓智能技术有限公司 | Accounting system optimization method, device, equipment and medium based on jvm out-of-heap memory |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1763776A2 (en) * | 2004-05-21 | 2007-03-21 | Bea Systems, Inc. | Service oriented architecture |
| CN101650795A (en) * | 2009-09-09 | 2010-02-17 | 浪潮集团山东通用软件有限公司 | Method for reducing account in transit based on real time conformation of intratransaction |
| CN106227765A (en) * | 2016-07-13 | 2016-12-14 | 广州唯品会网络技术有限公司 | The implementation method that time window is accumulative |
| CN107665460A (en) * | 2016-07-24 | 2018-02-06 | 平安科技（深圳）有限公司 | Accounting method and device based on real-time deal |
-
2018
- 2018-09-29 CN CN201811153706.6A patent/CN109034993B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1763776A2 (en) * | 2004-05-21 | 2007-03-21 | Bea Systems, Inc. | Service oriented architecture |
| CN101650795A (en) * | 2009-09-09 | 2010-02-17 | 浪潮集团山东通用软件有限公司 | Method for reducing account in transit based on real time conformation of intratransaction |
| CN106227765A (en) * | 2016-07-13 | 2016-12-14 | 广州唯品会网络技术有限公司 | The implementation method that time window is accumulative |
| CN107665460A (en) * | 2016-07-24 | 2018-02-06 | 平安科技（深圳）有限公司 | Accounting method and device based on real-time deal |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109034993A (en) | 2018-12-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109034993B (en) | Account checking method, account checking equipment, account checking system and computer readable storage medium | |
| CN107506451B (en) | Abnormal information monitoring method and device for data interaction | |
| CN107220142B (en) | Method and device for executing data recovery operation | |
| CN107832406B (en) | Method, device, equipment and storage medium for removing duplicate entries of mass log data | |
| CN107818431B (en) | Method and system for providing order track data | |
| CN110569214B (en) | Index construction method and device for log file and electronic equipment | |
| US9031997B2 (en) | Log file compression | |
| CN110795257A (en) | Method, device and equipment for processing multi-cluster operation records and storage medium | |
| CN109656999B (en) | Method, device, storage medium and apparatus for synchronizing large data volume data | |
| CN111309550A (en) | Data acquisition method, system, equipment and storage medium of application program | |
| CN110928851B (en) | Method, device and equipment for processing log information and storage medium | |
| CN112685433B (en) | Metadata updating method and device, electronic equipment and computer-readable storage medium | |
| EP3937022A1 (en) | Method and apparatus of monitoring interface performance of distributed application, device and storage medium | |
| CN113590556A (en) | Database-based log processing method, device and equipment | |
| CN109783457B (en) | CGI interface management method, device, computer equipment and storage medium | |
| CN103716384A (en) | Method and device for realizing cloud storage data synchronization in cross-data-center manner | |
| CN111400361A (en) | Data real-time storage method and device, computer equipment and storage medium | |
| CN113760847A (en) | Log data processing method, device, equipment and storage medium | |
| US10915510B2 (en) | Method and apparatus of collecting and reporting database application incompatibilities | |
| CN113806301A (en) | Data synchronization method, device, server and storage medium | |
| CN113535677B (en) | Data analysis query management method, device, computer equipment and storage medium | |
| CN113836237A (en) | Method and device for auditing data operation of database | |
| CN117271584A (en) | Data processing method and device, computer readable storage medium and electronic equipment | |
| CN110309206B (en) | Order information acquisition method and system | |
| US10360234B2 (en) | Recursive extractor framework for forensics and electronic discovery |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Shi Zuheng Inventor after: Hu Dongdong Inventor after: Yin Song Inventor after: Fan Ruibin Inventor after: Wan Lei Inventor before: Shi Zuheng Inventor before: Hu Dongdong Inventor before: Yin Song Inventor before: Fan Ruibin Inventor before: Wan Lei |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |