CN113961566A

CN113961566A - Transaction flow processing method and system

Info

Publication number: CN113961566A
Application number: CN202111329347.7A
Authority: CN
Inventors: 杜雪
Original assignee: China Construction Bank Corp
Current assignee: China Construction Bank Corp
Priority date: 2021-11-10
Filing date: 2021-11-10
Publication date: 2022-01-21

Abstract

The invention relates to a processing method and system of transaction pipelining, comprising decoupling a service system and a pipelining processing system by using a message queue, improving the response speed of the service system and reducing the pressure of the pipelining system during a transaction peak; hessian is used for serialization and deserialization of a flow object, so that the speed is improved, the serialized byte stream is shorter, the transmission speed is higher, the whole-flow processing efficiency reduced due to decoupling is made up, a large amount of transaction flow data is stored in a database-partitioning and table-partitioning mode in a storage mode, the access performance is improved, a memory base table corresponding to a flow number is calculated by a MurMurHash algorithm, the operation performance is higher, the collision rate is lower, the flow data can be more uniformly distributed into a plurality of tables, and the database resources are more effectively utilized. Meanwhile, the method provides a fault-tolerant mechanism for the failure condition of twice pipelining storage, and ensures that the transaction pipelining cannot be lost.

Description

Transaction flow processing method and system

Technical Field

The invention relates to the technical field of internet, in particular to a method and a system for processing transaction flow.

Background

The processing and storage of transaction pipelining have wide application, most business systems generate a large amount of transaction pipelining data every day, but the processing and storage of the pipelining data can seriously affect the system performance. Service systems often have higher requirements on response speed, system stability and the like. Meanwhile, the transaction flow record is used as a data base for statistical analysis and prediction of related services, and has an important role.

At present, transaction flow processing and storage functions of most systems are coupled with business functions, and flow data has the characteristics of large data volume and long storage time, so that the response speed of a business system is seriously influenced, and the stability of the system is also influenced due to a long business processing flow. Meanwhile, in the processing process, once abnormal conditions such as the incapability of connecting the databases and the like occur, the pipelining storage fails, data is lost and difficult to retrieve, and therefore the statistical analysis result of subsequent services is influenced.

Disclosure of Invention

In order to solve the defects of the prior art, the invention provides a transaction flow processing method and a transaction flow processing system, wherein a service system is decoupled from a flow processing system by using a message queue, so that the response speed of the service system is improved, and the pressure of the flow processing system during a transaction peak value can be reduced; hessian is used for serialization and deserialization of the flow object, so that the speed is improved, the serialized byte stream is shorter, the transmission speed is higher, and the whole-process processing efficiency reduced by decoupling is compensated. The storage adopts a mode of database division, table division and partition division to store a large amount of transaction flow data, so that the access performance is improved; the memory base table corresponding to the serial number is calculated by adopting a MurMurHash algorithm, the operation performance is higher, the collision rate is lower, the serial data can be more uniformly distributed into a plurality of tables, and the database resources are more effectively utilized. Meanwhile, the method provides a fault-tolerant mechanism for the failure condition of twice pipelining storage, and ensures that the transaction pipelining cannot be lost.

In order to achieve the above purpose, the technical scheme adopted by the invention comprises the following steps:

a method of processing a transaction pipeline, comprising:

the service system sends the stream object to the message queue, and the stream system starts a Socket port to receive the stream object; the message queue is Kafka message queue interface integrated Hessian, and serializes and deserializes the pipeline object;

judging the base table in which the stream object should be stored, and connecting a database to store the stream object into the corresponding base table; the storage mode comprises a storage mode of sub-base, sub-table and sub-area;

judging whether the stream object is stored in a base table or not, and when the stream object is stored in the base table, sending the stream object to finish the transaction stream processing successfully;

when the flow object is not stored in the base table, marking the flow object as an unstuffed flow object, writing the unstuffed flow object into an unstuffed flow object file and storing the unstuffed flow object in a directory;

scanning the flow object, reading the flow object file which is not put in storage when the flow object file which is not put in storage meets the naming rule and is less than or equal to the preset sending times, connecting a flow system and sending the flow object file which is not put in storage;

when the transmission is successful, deleting the non-warehousing pipelining object file;

and when the sending fails, modifying the name of the non-warehousing stream object file, and marking the sending times.

Further, the storage mode of the database, table and partition comprises that the pipeline objects are respectively stored into partitions by taking a month as a unit, all the partitions of the database are checked through a script, the number of the partitions is increased when the number of the partitions is smaller than three, and the script is repeatedly executed by taking a day as a unit;

the pipelining system calculates a hash value corresponding to the pipelining number by using a hash algorithm according to the pipelining number of each pipelining object, calculates a section number of the pipelining object according to the hash value, and takes the section number as a table number of the pipelining object;

and obtaining a corresponding database label according to the table number, connecting the corresponding database label with the corresponding database according to the database label, and storing the stream object to the database.

Further, the hash algorithm is a non-cryptographic hash algorithm.

Furthermore, the receiving of the pipeline object by the pipeline system starting Socket port comprises starting a thread to asynchronously process the pipeline object; the thread scans the pipeline object at regular intervals.

Further, the chronological object includes a serial number, a transaction date, a business status, and a transaction amount

Further, the preset number of times of transmission is 3.

Further, the database includes an Oracle database and a Mysql database.

The invention also relates to a transaction pipeline processing system, which is characterized by comprising:

the acquisition module is used for acquiring the pipeline object;

the judging module is used for judging whether the stream object is successfully stored in the corresponding base table and the sending times of the stream object;

the processing module is used for calculating a base table in which the pipeline object is stored;

and the storage module is used for storing the pipeline objects.

The invention also relates to a computer-readable storage medium, characterized in that the storage medium has stored thereon a computer program which, when being executed by a processor, carries out the above-mentioned method.

The invention also relates to an electronic device, characterized in that it comprises a processor and a memory;

the memory is used for storing the pipeline object;

the processor is configured to execute the computer program stored in the memory and execute the method described above.

The invention has the beneficial effects that:

by adopting the transaction flow processing method and the transaction flow processing system, the service system and the flow processing system are decoupled by using the message queue, so that the response speed of the service system is improved, and the pressure of the flow system during a transaction peak value can be reduced; hessian is used for serialization and deserialization of the flow object, so that the speed is improved, the serialized byte stream is shorter, the transmission speed is higher, and the whole-process processing efficiency reduced by decoupling is compensated. The storage adopts a mode of database division, table division and partition division to store a large amount of transaction flow data, so that the access performance is improved; the MurMurHash algorithm is adopted to calculate the memory base table corresponding to the serial number, the operation performance is higher, the collision rate is lower, the serial data can be more uniformly distributed into a plurality of tables, and the database resources are more effectively utilized. Meanwhile, the method provides a fault-tolerant mechanism for the failure condition of twice pipelining storage, and ensures that the transaction pipelining cannot be lost.

Drawings

FIG. 1 is a flow chart of a transaction pipeline processing method according to the present invention.

FIG. 2 is a schematic diagram of a transaction pipeline processing system according to the present invention.

Detailed Description

For a clearer understanding of the contents of the present invention, reference will be made to the accompanying drawings and examples.

The invention relates to a processing method of transaction flow shown in figure 1, comprising the following steps:

using Kafka as message middleware, the service system sends messages to Kafka, and the pipeline system listens and receives messages. Therefore, the service system is not influenced by the flow processing process, the response speed of the service system and the stability of the system can be improved, a buffering effect is achieved, and the pressure of the flow system during the transaction peak value is reduced.

Specifically, Kafka can be expanded horizontally as message middleware, but Kafka receives and sends messages in byte arrays, and other data types need to specify a serialization mode. The pipelining data generated by the business system is an object, comprises a pipelining number, a transaction date, a business state and a transaction amount field, and cannot use a serialization mode originally supported by Kafka, so that the system adopts a self-defined serialization mode to serialize and deserialize the pipelining object. The invention serializes and deserializes the pipeline object by implementing the interface provided by Kafka and integrating Hessian. Hessian is a lightweight remoting onhttp tool, provides the function of RMI by using a simple method, is simpler and quicker compared with WebService, and has the characteristics of independence from a platform and the like. Serialization is the process of converting a pipeline object into a byte sequence; deserialization refers to the process of restoring a byte sequence to a pipelined object. Compared with native serialization of Java, the serialization and deserialization speed is higher, the serialized byte stream is shorter, data transmission is carried out based on the mode of the serialized byte stream, the data transmission efficiency is obviously improved, and further the whole-process processing efficiency reduced by adding message middleware can be compensated.

the storage mode of the sub-database, sub-table and sub-area comprises that each table is divided according to months, data of each month is in one sub-area, operation is carried out according to the sub-areas during query and insertion, the automatic expansion of the sub-areas is realized, all the existing table sub-areas of the database are checked through scripts, when the number of the sub-areas is less than three, six sub-areas are newly added, and the scripts are executed at regular time every day; as a table evolves over time and traffic, the amount of data in the table in the library becomes larger and larger. The data operations are also getting larger and larger. The resources of one physical machine are limited, and the amount of data that can be finally carried and the processing capacity of the data are limited. At this time, the sub-base sub-tables are used for accepting the ultra-large-scale tables. The database and table partitioning are not conflicted and can be used in combination.

the pipelining system calculates a hash value corresponding to the pipelining number by using a hash algorithm according to the pipelining number of each pipelining object, calculates a section number of the pipelining object according to the hash value, and takes the section number as a table number of the pipelining object; and obtaining a corresponding database label according to the table number, connecting the corresponding database label with the corresponding database according to the database label, and storing the stream object to the database.

Specifically, the flow object can be designed to be stored in 1024 tables of 6 databases, the last 6 bits of each table are numbers and represent the serial numbers of the flow table, and the rest bits are supplemented with 0, for example, the names of the first flow meter and the last flow meter are xxx _000001 and xxx _001024 respectively; the number of the databases and the number of the tables can be set in the configuration file according to the situation, and the testing environment can only configure 2 databases because the data volume is small so as to save resources; because the structure of the pipeline objects is the same, all the pipeline objects are uniformly distributed in the 1024 tables as much as possible, so that the resource occupation of each library is uniform, the pipeline system calculates the hash value corresponding to the pipeline number by using a MurMurHash algorithm according to the globally unique pipeline number in each pipeline object, and the MurMurHash algorithm is a non-encryption Hash algorithm and has higher operation performance and lower collision rate compared with the traditional Hash algorithm; after the hash value is obtained through calculation, the 32 th power hash space of 2 is averagely divided into the number of tables, the number of the falling interval of the hash value is calculated according to the hash value and serves as the table number, the mapping relation between the table number and the database number is in a configuration file, the database number corresponding to the table can be obtained according to the table number, the corresponding database connection is obtained according to the database number, and the running water is put in storage.

Specifically, when the service system fails to send the pipeline object to Kafka, the pipeline object is written into a file and stored in a specified directory. The other thread asynchronously scans the files under the designated path, when the files exist, the files are read, the running system is connected, the running files are sent after the connection is successful, and if the running system returns success, the files are deleted; if the return fails, the file name is modified, the sending times are marked, and the sending is not continued after the sending times reach 3 times.

Another aspect of the present invention relates to a transaction pipeline processing system, which is structured as shown in fig. 2, and includes:

the acquisition module is used for acquiring the pipeline object;

and the storage module is used for storing the pipeline objects. .

By using this system, the above-described arithmetic processing method can be executed and a corresponding technical effect can be achieved.

An embodiment of the present invention also provides a computer-readable storage medium capable of implementing all the steps in the processing method of the transaction flow in the above embodiment, where the computer-readable storage medium has stored thereon a computer program, and when being executed by a processor, the computer program implements all the steps of the processing method of the transaction flow in the above embodiment.

The embodiment of the present invention further provides an electronic device for executing the method, as an implementation apparatus of the method, the electronic device at least has a processor and a memory, and particularly, the memory stores data and related computer programs, such as pipeline objects, etc., required for executing the method, and the processor calls the data and programs in the memory to execute all steps of implementing the method, and obtains corresponding technical effects.

Preferably, the electronic device may comprise a bus architecture, which may include any number of interconnected buses and bridges linking together various circuits including one or more processors and memory. The bus may also link various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same element, i.e., a transceiver, providing a means for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used for storing data used by the processor in performing operations.

Additionally, the electronic device may further include a communication module, an input unit, an audio processor, a display, a power source, and the like. The processor (or controller, operational controls) employed may include a microprocessor or other processor device and/or logic device that receives input and controls the operation of various components of the electronic device; the memory may be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices, and may store the above-mentioned related data information, and may also store a program for executing the related information, and the processor may execute the program stored in the memory to realize information storage or processing, etc.; the input unit is used for providing input to the processor, and can be a key or a touch input device; the power supply is used for supplying power to the electronic equipment; the display is used for displaying display objects such as images and characters, and may be an LCD display, for example. The communication module is a transmitter/receiver that transmits and receives signals via an antenna. The communication module (transmitter/receiver) is coupled to the processor to provide an input signal and receive an output signal, which may be the same as in the case of a conventional mobile communication terminal. Based on different communication technologies, a plurality of communication modules, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be disposed in the same electronic device. The communication module (transmitter/receiver) is also coupled to a speaker and a microphone via an audio processor to provide audio output via the speaker and receive audio input from the microphone to implement the usual telecommunication functions. The audio processor may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor is also coupled to the central processor, so that recording on the local machine can be realized through the microphone, and sound stored on the local machine can be played through the loudspeaker.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a system for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction system which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.

The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method of processing a transaction pipeline, comprising:

2. The method according to claim 1, wherein the storage manner of the sub-library and sub-table partitions comprises that the pipeline objects are respectively stored into the partitions by taking a month as a unit, all the partitions of the database are checked through a script, the number of the partitions is increased when the number of the partitions is smaller than three, and the script is repeatedly executed by taking a day as a unit;

3. The method of claim 2, wherein the hash algorithm is a non-cryptographic hash algorithm.

4. The method of claim 3, wherein the pipelined system-initiated Socket port accepting the pipelined object comprises initiating a thread to asynchronously process the pipelined object; the thread scans the pipeline object at regular intervals.

5. The method of claim 1, wherein the chronological object includes a serial number, a transaction date, a business status, and a transaction amount.

6. The method of claim 1, wherein the predetermined number of transmissions is 3.

7. The method of claim 1, wherein the database comprises an Oracle database and a Mysql database.

8. A transaction pipelining processing system, comprising:

the acquisition module is used for acquiring the pipeline object;

the judging module is used for judging whether the stream object is successfully stored in a corresponding base table and the sending times of the stream object;

and the storage module is used for storing the pipeline objects.

9. A computer-readable storage medium, characterized in that the storage medium has stored thereon a computer program which, when being executed by a processor, carries out the method of any one of claims 1 to 7.

10. An electronic device comprising a processor and a memory;

the memory is used for storing the pipeline object;

the processor, for executing a computer program stored in a memory, performs the method of any of claims 1 to 7.