CN115344633A - Data processing method, device, equipment and storage medium - Google Patents
Data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN115344633A CN115344633A CN202210959739.XA CN202210959739A CN115344633A CN 115344633 A CN115344633 A CN 115344633A CN 202210959739 A CN202210959739 A CN 202210959739A CN 115344633 A CN115344633 A CN 115344633A
- Authority
- CN
- China
- Prior art keywords
- data
- transaction
- data processing
- payment
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
Abstract
The invention discloses a data processing method, a device, equipment and a storage medium, aiming at the problems that an HTTP synchronous interface is adopted by the interface of the prior prepayment system, the interface is completely open, the system pressure is high, the interface is easy to overtime, the resource is occupied and crashed, and the transaction type of the transaction data is identified by acquiring the transaction data in real time; classifying the transaction types according to real-time account arrival and offline settlement; performing data processing on the transaction type of the real-time account arrival by adopting a synchronous data transmission mode; processing the transaction type of off-line settlement by adopting an asynchronous data transmission mode or a file data transmission mode; therefore, the interface pressure of the prepayment system is relieved, and the data processing efficiency of the prepayment system is improved.
Description
Technical Field
The invention belongs to the technical field of data processing, and particularly relates to a data processing method, a data processing device, data processing equipment and a data processing storage medium for a prepayment system.
Background
At present, an HTTP synchronous interface is adopted as a prepayment interface, the interface is completely open, and the system pressure is high. The interface covers various service scenes, including recharging, withdrawing, deducting, refunding, canceling, transferring and the like, and meanwhile, the service of sending prepayment is provided for 68 upstream systems (all service systems), and the daily transaction amount is 21W.
Based on a huge and growing business system, when the business peak is high and the prepayment interface is accessed concurrently, the interface is easy to overtime, resources are occupied and crashed, and even a server is down, so that the problems of incomplete data storage, abnormal data updating and the like are caused.
Disclosure of Invention
The invention aims to provide a data processing method, a data processing device, data processing equipment and a data processing storage medium, which are used for building a pre-payment prepositive service, reducing pressure of a pre-payment system, ensuring the stability of the pre-payment system, providing multi-dimensional white list configuration, reducing pressure of the pre-payment system in a peak period, prepositioning data verification, reducing the logic complexity of the pre-payment system, improving the interface timeliness, reducing the pressure of an upstream system for retransmission due to the abnormality of an original pre-payment interface and ensuring the pre-payment core function.
In order to solve the problems, the technical scheme of the invention is as follows:
a data processing method for a prepaid system, comprising:
acquiring transaction data in real time, and identifying the transaction type of the transaction data;
classifying the transaction types according to real-time account arrival and offline settlement;
performing data processing on the transaction type of the real-time account arrival by adopting a synchronous data transmission mode;
processing the transaction type of off-line settlement by adopting an asynchronous data transmission mode or a file data transmission mode; the interface pressure of the prepayment system is relieved, and the data processing efficiency of the prepayment system is improved.
According to an embodiment of the present invention, the obtaining transaction data in real time, and identifying the transaction type of the transaction data further includes:
carrying out standardized character filtering operation on the acquired transaction data to obtain target transaction data;
carrying out object identification operation on the target transaction data to obtain a transaction object;
and calling a preset transaction type identification model to process the transaction data of the transaction object in the target transaction data to obtain the transaction type of the transaction object.
According to an embodiment of the present invention, the classifying the transaction types according to the real-time account and the off-line settlement further comprises:
marking the transaction data with the transaction types of recharging, withdrawing or transferring as real-time account arrival, and transmitting the transaction data to a first data processing channel for real-time processing;
and marking the transaction data with the transaction type of deduction, refund or reimbursement as off-line settlement, and transmitting the transaction data to a message queue in a second data processing channel for processing.
According to an embodiment of the present invention, the performing data processing on the transaction type of the real-time account by using a synchronous data transmission manner further includes:
the prepayment system is used as a server side, HTTP interface service is provided, and an interface API is issued;
and after receiving the data sent by the client according to the interface API, the server side performs validity check on the multiple data, generates a transaction record and returns a prepayment result.
According to an embodiment of the present invention, the processing the transaction type of the offline settlement by using the asynchronous data transmission further includes:
building a pre-payment prepositive service;
setting a pre-payment white list according to the transaction type;
the pre-payment prepositive service receives pre-payment data sent by a client in a kafka mode, and performs data validity check;
the pre-payment preposition service transmits the data after the verification is passed to the pre-payment system.
According to an embodiment of the present invention, the data processing for the transaction type of offline settlement in a file data transmission manner further includes:
building a pre-payment prepositive service;
setting a pre-payment white list according to the transaction type, flexibly configuring a pre-payment sending state, and reducing pressure for the peak period of a pre-payment system;
loading a CSV file through a database and performing data processing, wherein the CSV file is a file generated in advance according to the prepayment data; generating the CSV file again by the processed data and uploading the CSV file to a designated path;
the pre-payment prepositive service loads the CSV file under the appointed path and carries out data validity check;
the pre-payment prepositive service summarizes the data passing the verification according to the charging department, the charging item and the sending pre-payment time dimension;
the pre-payment front-end service sends the aggregated data to a pre-payment system.
A data processing apparatus for use in a prepaid system, comprising:
the data identification module is used for acquiring transaction data in real time and identifying the transaction type of the transaction data;
the data classification module is used for classifying the transaction types according to real-time account arrival and offline settlement;
the first data processing module is used for processing the transaction type of the real-time account arrival in a synchronous data transmission mode;
the second data processing module is used for processing the transaction type of the offline settlement in an asynchronous data transmission mode or a file data transmission mode; the interface pressure of the prepayment system is relieved, and the data processing efficiency of the prepayment system is improved.
According to an embodiment of the present invention, the data identification module further includes:
the data filtering unit is used for carrying out standardized character filtering operation on the acquired transaction data to obtain target transaction data;
the object identification unit is used for carrying out object identification operation on the target transaction data to obtain a transaction object;
and the data processing unit is used for calling a preset transaction type identification model to process the transaction data of the transaction object in the target transaction data to obtain the transaction type of the transaction object.
A data processing apparatus comprising: a memory and a processor, the memory having stored therein computer readable instructions, which when executed by the processor, cause the processor to perform the steps of the data processing method in an embodiment of the present invention.
A storage medium having stored thereon computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform steps in a data processing method in an embodiment of the invention.
Due to the adoption of the technical scheme, compared with the prior art, the invention has the following advantages and positive effects:
aiming at the problems that an HTTP synchronous interface is adopted by an interface of a current prepayment system, the interface is completely open, the system pressure is high, the interface is easy to overtime, resources are occupied and crashed, the data processing method in one embodiment of the invention identifies the transaction type of the transaction data by acquiring the transaction data in real time; classifying the transaction types according to real-time account arrival and offline settlement; performing data processing on the transaction type of the real-time account arrival by adopting a synchronous data transmission mode; processing the transaction type of off-line settlement by adopting an asynchronous data transmission mode or a file data transmission mode; therefore, the interface pressure of the prepayment system is relieved, and the data processing efficiency of the prepayment system is improved.
Drawings
FIG. 1 is a flow diagram of a data processing method in an embodiment of the invention;
FIG. 2 is a block diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention.
Detailed Description
A data processing method, an apparatus, a device and a storage medium according to the present invention are further described in detail below with reference to the accompanying drawings and specific embodiments. Advantages and features of the present invention will become apparent from the following description and from the claims.
Example one
The embodiment aims at the problems that an HTTP synchronous interface is adopted by an existing prepayment system interface, the interface is completely open, the system pressure is high, the interface is easy to overtime, and resources are occupied and crashed, and provides a data processing method.
Referring to fig. 1, the data processing method includes the following steps:
s1: acquiring transaction data in real time, and identifying the transaction type of the transaction data;
s2: classifying the transaction types according to real-time account arrival and offline settlement;
s3: performing data processing on the transaction type of the real-time account arrival by adopting a synchronous data transmission mode;
s4: processing the transaction type of off-line settlement by adopting an asynchronous data transmission mode or a file data transmission mode; the interface pressure of the prepayment system is relieved, and the data processing efficiency of the prepayment system is improved.
In step S1, transaction data is obtained in real time, and a transaction type of the transaction data is identified, which may be implemented by the following steps:
carrying out standardized character filtering operation on the acquired transaction data to obtain target transaction data;
carrying out object identification operation on the target transaction data to obtain a transaction object;
and calling a preset transaction type identification model to process the transaction data of the transaction object in the target transaction data to obtain the transaction type of the transaction object.
In this embodiment, an extraction rule device is first constructed in the application system, where the extraction rule device includes a log acquisition level, a log acquisition program name, an extraction time range, and the like, and when the log file content is newly added, each rule in the extraction rule device is matched, and if the matching is successful, the transaction in the log is output to the data processor.
For example, the extraction rule engine obtains the information-level transaction or interaction information, the name of the log obtaining program is TransactionBusiness, and the extraction time is 0:00-24:00, the extraction rule device extracts the Info information of the designated program in the log, and ignores the rest information by default. At this time, the output transaction is the transaction data to be analyzed, and the transaction data to be analyzed is stored.
After the transaction data to be analyzed are obtained, the transaction object and the transaction type are extracted from the transaction data to be analyzed.
And performing standardized character filtering operation on the transaction data to obtain target transaction data.
In practical applications, some sensitive words, stop words, etc. may exist in the transaction data, and the transaction data needs to be subjected to a standardized character filtering operation.
Specifically, the standardized character filtering operation refers to sensitive word filtering, synonym replacement, stop word removal and the like on the transaction data to be analyzed, so that a plurality of sentences with high semantic value are obtained, and the target transaction data is obtained.
And carrying out object identification operation on the target transaction data to obtain a transaction object.
Specifically, the target transaction data is matched according to a preset template in the knowledge base, and a transaction object, namely which client currently performs the transaction, is identified. If the current transaction objects are A, B, C, D, E, F and G.
And calling a preset transaction type identification model to process the transaction data of the transaction object in the target transaction data to obtain the transaction type of the transaction object.
In this embodiment, historical transaction data and transaction types may be collected, and the constructed neural network model may be trained until the loss function of the model is less than a set value. The neural network model obtained at this time is called a preset transaction type identification model.
When the model is used, the transaction data of the transaction object in the target transaction data is input into the preset transaction type identification model, so that the transaction type of the transaction object can be obtained,
under a general scene, the transaction types can be various, such as recharging, withdrawing, transferring, refunding and the like. The same transaction object may involve multiple transaction types or may involve only one transaction type.
In step S2, transaction types are classified according to real-time account arrival and offline settlement.
In this embodiment, the transaction data of which the transaction type is a recharge type, a withdrawal type or a transfer type is marked as a real-time account, and is transmitted to the first data processing channel for real-time processing.
And marking the transaction data with the transaction type of deduction, refund or reimbursement as off-line settlement, and transmitting the transaction data to a message queue in the second data processing channel for processing.
In the classification of the real-time account arrival, in practical application, based on the industrial characteristics of express delivery, the account state of a website is guaranteed to be updated in time from the standpoint of customers, and the real-time account arrival is realized according to the corresponding transaction types of recharging, withdrawing and transferring.
The transaction data of the transaction type which can be received in real time can be stored in a transaction information table of an Oracle database, the transaction information table is monitored in real time, and incremental transaction information data are obtained in real time. The incremental data in the transaction information table is copied to the message queue of Kafka in real time by using OGG (Oracle golden gate). Data were read from the brooker in Kafka by either the real-time ETL program or the Spark program. And packaging and counting the acquired data according to a service scene (such as recharging, withdrawing or transferring), storing the counted stored data into the MongoDB in a JSON format, and adding a last _ up _ time field for each piece of data to store the time for storing the data into the MongoDB.
The timing scanning is carried out on the mongo DB through a Spring Boot program, and the scanning interval of the timing scanning of the mongo DB is set to be in the second level. And querying to obtain incremental data according to last _ up _ time, and synchronizing the incremental data to an Aeroscope database in real time.
The packaged transaction data passes through a first data processing channel (namely a synchronous data transmission channel) in real time, and the first data processing channel calls JSON format data in an Aeroform database to process, so that the account state of a website is guaranteed to be updated in time.
For the classification of offline settlement, in practical application, based on business scenes such as ticket collection, examination and the like, pre-payment is sent after data processing, and the transaction types corresponding to deduction, refund and reimbursement are offline settlement.
Transaction data for the transaction types for offline settlement can be aggregated offline and synchronously into an operational data warehouse ODS. In this way, direct manipulation of source data in the prepayment system can be reduced.
Synchronizing data in the data warehouse ODS into a wide table of the data warehouse DW, establishing the wide table, and summarizing all fields to be counted in different scenes (such as deduction, refund or reimbursement) into the wide table. Therefore, the table connection operation during data processing of a single scene can be reduced, and the efficiency of data synchronous processing is ensured to the maximum extent.
And counting and summarizing the data stored in the wide table on Hive by combining with service requirements, and inserting a summarized result into a service table in a database DM, wherein the service table in a data warehouse DM stores data of different service scenes. Each partition of the service table is a result of statistics for different specific service scenarios, and the statistical data of different partitions are taken for different service scenarios. And further, data of different service scenes can be accurately processed in a targeted manner.
And synchronizing different tags on the data bands of all the partitions in the service table into an Aero pike database, and storing the tags in a JSON form. And (3) enabling the transaction data in the service table to pass through a second data processing channel (namely an asynchronous data transmission channel or a file data transmission channel), calling JSON format data in the Aeroskip database to be processed by the second data processing channel, and sending the processed data to a prepayment system.
In step S3, a synchronous data transmission mode is adopted for the transaction type of the real-time account arrival, and data processing is performed.
In the step, data processing is carried out on the transaction data of the transaction type of the real-time account arrival by a synchronous data transmission method, so that the real-time account arrival business is realized, and the timely updating of the account state of the network point is ensured. The synchronous data transmission method is as follows:
1) The prepayment system is used as a server side, provides HTTP Interface service and issues an Interface API (application Programming Interface);
2) The client sends data according to the interface API;
3) After receiving the data, the server side performs multiple data validity checks to generate a transaction record;
4) And the server returns the result of the prepayment transmission.
In the synchronous data transmission method, the message transmission between the client and the server is realized by adopting polling technology, traditional polling and long polling message transmission modes. The polling mode continuously sends HTTP requests to the server, and the server feeds back the HTTP requests every time but does not necessarily carry the latest data. The long polling mode also continuously sends HTTP requests to the server, and the server returns when new data exists.
In step S4, the transaction type of offline settlement is processed by using an asynchronous data transmission mode or a file data transmission mode.
According to the method, for the transaction data of the off-line settlement transaction type, data processing is carried out in an asynchronous data transmission mode or a file data transmission mode, the pre-payment system is depressurized by building a pre-payment preposed service, the stability of the pre-payment system is guaranteed, multi-dimensional white list configuration is provided, and depressurization is carried out for the rush hour pre-payment system through flexible configuration. By means of the prepositive data verification, the logic complexity of the prepayment system is reduced, the interface timeliness is improved, the pressure of the upstream system for retransmitting due to the abnormality of the original prepayment interface is reduced, and the prepayment core function is guaranteed.
The asynchronous data transmission method comprises the following steps:
1) Building a pre-payment prepositive service;
2) Setting a pre-payment white list based on the transaction type, and flexibly configuring a pre-payment sending state to reduce the peak period pressure of a pre-payment system;
3) The client sends the prepayment data in a kafka mode;
4) The pre-payment prepositive service receives data and carries out data validity check;
5) The pre-payment front-end service sends the verification-passed data to the pre-payment system.
All settlement forces of the traditional transaction business model are concentrated in a prepayment system, such as charging, withdrawing, deducting, refunding, reimbursing, transferring and the like, the prepayment system simultaneously provides a service of sending prepayment for 68 upstream systems, and daily transaction amount is up to 21W. Based on a huge and growing business system, when the business peak is high and the prepayment interface is accessed concurrently, the interface is easy to overtime, resources are occupied and crashed, and even a server is down, so that the problems of incomplete data storage, abnormal data updating and the like are caused.
The embodiment improves the method, and the method is used for carrying out prepositive identification of transaction types and prepositive verification of data validity by building a prepositive service of prepayment, reducing pressure for the prepayment system and ensuring the stability of the prepayment system.
And setting a prepayment white list based on the transaction type, and flexibly configuring the prepayment sending state. If the transaction type of the deduction, the refund or the reimbursement is set as the white list, the prepayment system can process the transaction data of the deduction, the refund or the reimbursement separated in the step S2 in real time, that is, send the transaction data in real time. Of course, the transaction types corresponding to the recharging, the withdrawal and the transfer may also be set as a white list, and then, the prepaid system may perform real-time processing, that is, send the transaction data of the recharging, the withdrawal and the transfer separated in step S2 in real time. The method can be specifically set according to actual requirements.
In the asynchronous data transmission method of the embodiment, when the client sends the prepayment data in the kafka mode, the prepayment prepositive service identifies the transaction type of the received prepayment data and performs data validity check. The data validity check comprises checking information of a sender, a receiver, a transaction amount, transaction time and the like of the transaction data.
In particular, the transaction data may be verified by means of a digital token. For example, the client interacts with the authentication module in advance to authenticate the user identity of the transaction. After the authentication is passed, the authentication module runs a preset algorithm according to the transaction data uploaded by the client to generate a corresponding group of character strings (such as tokens), namely digital tokens.
The pre-payment prepositive service receives the digital token from the client, verifies the digital token, operates a symmetric algorithm of an algorithm of the digital token generated by the authentication module according to the digital token, and obtains transaction data corresponding to the digital token, namely data to be verified. And comparing the received transaction data with the transaction data obtained by analyzing the digital token, and judging whether the received transaction data and the transaction data are consistent. And if the transaction data is consistent with the data to be verified, the digital token passes the verification, otherwise, the digital token cannot pass the verification.
The pre-payment prepositive service sends the data after the verification is passed to the pre-payment system, so that the identification of the transaction type and the related calculation of data validity verification by the pre-payment system are saved, the logic complexity of the pre-payment system is reduced, the interface timeliness is improved, and the stability of the pre-payment system is ensured.
The file data transmission method is as follows:
1) Building a pre-payment prepositive service;
2) Setting a pre-payment white list based on the transaction type, and flexibly configuring a pre-payment sending state to reduce the peak period pressure of a pre-payment system;
3) The upstream system generates the pre-payment data into a CSV file and uploads the CSV file to a designated path;
4) Loading a CSV file by a GP (GreenPlum) database and carrying out deep processing (such as forced leveling settlement of express industry);
5) The GP database generates the processed data into CSV files and uploads the CSV files to a designated path;
6) Loading a CSV file and carrying out data validity check by the pre-payment prepositive service;
7) The prepayment prepositive service carries out data summarization on the data passing the verification according to the charging department, the charging project and the sending prepayment time dimension;
8) The pre-payment preposition service sends the summarized data to a pre-payment system.
The GP is the fastest and highest cost performance relational distributed database in the industry, and has the following advantages:
mass storage
GreenPlum supports the storage and management of 50PB (1PB = 1024TB) level mass data, integrates data from different departments and different platforms of different source systems into a database for centralized storage, and stores detailed historical data tracks, so that a service user does not face one information island or another information island, is not confused about deviation caused by data of different versions, and reduces the complexity of management and maintenance work for IT personnel.
High concurrency
Greenplus provides a resource management function (workload manager) to manage database resources, and resource queue management is utilized to realize resource allocation according to user groups, such as Session simultaneous activation number, maximum resource value, and the like. Through the resource management function, the resource allocation and the user SQL query priority level can be carried out according to the user level, and meanwhile, the consumption of system resources by low-quality SQL (such as unconditional multi-table join and the like) can be prevented.
High cost performance
The GreenPlum database software system node can reach high performance on a common x86Sercer based on various open hardware platforms in the industry, such as PC Sercer of SUN/HP/DELL and other manufacturers, so that the cost performance is high, compared with other special systems of closed data warehouses, the investment of Greenplus per TB is 1/5 of that of the Greenplus or even lower, and similarly, the maintenance cost of the Greenplus product is much lower compared with that of the same manufacturer.
The system is easy to use
The GreenPlum product is developed based on popular PostgreSQL, almost all PostgreSQL client tools and PostgreSQL applications can run on a Greenplus platform, and rich PostgreSQL resources are provided for users to refer to on the Internet.
High availability
Greenplus is a highly available system, in existing cases using a clustered MPP environment of up to 96 machines. Besides the Raid technology at the hardware level, greenplus also provides protection of a database layer Mirror mechanism, that is, data of each node is synchronously mirrored in other nodes, and an error of a single node does not affect the use of the whole system.
For the main node, the Greenplus provides a Master/Stand by mechanism to carry out main node fault tolerance, and when the main node has an error, the main node can be switched to the Stand by node to continue service.
The reaction speed is high
The greenplus realizes real-time updating of a data warehouse through a quasi-real-time and real-time data loading mode, and further realizes an dynamic data warehouse (ADW). Based on the dynamic data warehouse, the business user can perform BI real-Time analysis- "Just In Time BI" on the current business data, so that the enterprise can be enabled to sensitively sense the change of the market, and the decision support reaction speed is accelerated.
In the embodiment, the GP database is adopted to load the CSV file and carry out deep processing (such as forced leveling settlement of express industry), the processed data is generated into the CSV file and is uploaded to a specified path (such as an address for specially storing the CSV file or an address for facilitating pre-payment pre-service loading), the data processing of the CSV file can be rapidly and effectively realized, and the data processing efficiency is improved.
The file data transmission method and the asynchronous data transmission method need to build a pre-payment preposition service, and the difference is that the file data transmission method needs to carry out deep processing on upstream system data, and the description is omitted here.
The data processing method in the embodiment reduces the pressure of the prepayment system by building the prepayment prepositive service, so as to ensure the stability of the prepayment system; the sending state of the prepayment system is flexibly configured by providing multi-dimensional white list configuration, and the prepayment system is depressurized for peak periods; through the preposition of data verification, the logic complexity of a prepayment system is reduced, the interface timeliness is improved, the pressure of retransmission of an upstream system due to the abnormality of an original prepayment interface is reduced, and the prepayment core function is guaranteed.
Example two
The present embodiment provides a data processing apparatus for use in a prepaid system, please refer to fig. 2, the data processing apparatus comprising:
the data identification module 1 is used for acquiring transaction data in real time and identifying the transaction type of the transaction data;
the data classification module 2 is used for classifying the transaction types according to real-time account arrival and offline settlement;
the first data processing module 3 is used for processing the transaction type of the real-time account arrival in a synchronous data transmission mode;
the second data processing module 4 is used for processing the transaction type of the off-line settlement by adopting an asynchronous data transmission mode or a file data transmission mode; the interface pressure of the prepayment system is relieved, and the data processing efficiency of the prepayment system is improved.
Wherein the data identification module further comprises:
the data filtering unit is used for carrying out standardized character filtering operation on the acquired transaction data to obtain target transaction data;
the object identification unit is used for carrying out object identification operation on the target transaction data to obtain a transaction object;
and the data processing unit is used for calling a preset transaction type identification model to process the transaction data of the transaction object in the target transaction data to obtain the transaction type of the transaction object.
The functions and implementation manners of the data identification module 1, the data classification module 2, the first data processing module 3, and the second data processing module 4 are as described in the first embodiment, and are not described herein again.
EXAMPLE III
The embodiment provides a data processing apparatus. Referring to fig. 3, the data processing apparatus 500 may have a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 510 (e.g., one or more processors) and a memory 520, one or more storage media 530 (e.g., one or more mass storage devices) storing applications 533 or data 532. Memory 520 and storage media 530 may be, among other things, transient or persistent storage. The program stored on the storage medium 530 may include one or more modules (not shown), each of which may include a sequence of instructions operating on the data processing apparatus 500.
Further, the processor 510 may be arranged to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the data processing device 500.
The data processing apparatus 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input-output interfaces 560, and/or one or more operating systems 531, such as Windows server, vista, and the like.
Those skilled in the art will appreciate that the data processing device architecture shown in fig. 3 is not meant to be limiting of the data processing device, and may include more or fewer components than shown, or some components may be combined, or a different arrangement of components.
Another embodiment of the present invention also provides a computer-readable storage medium.
The computer readable storage medium may be a non-volatile computer readable storage medium, which may also be a volatile computer readable storage medium. The computer readable storage medium has instructions stored therein, which when executed on a computer, cause the computer to perform the steps of the data processing method in the first embodiment.
The data processing method, if implemented in the form of program instructions and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present embodiment essentially or partially contributes to the prior art, or all or part of the technical solution may be embodied in the form of software, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present disclosure. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
For convenience and brevity of description, it is clear to those skilled in the art that the above-described system and apparatus may be referred to in the foregoing method embodiments for identifying specific implementations.
The embodiments of the present invention have been described in detail with reference to the drawings, but the present invention is not limited to the embodiments. Even if various changes are made to the present invention, it is still within the scope of the present invention if they fall within the scope of the claims of the present invention and their equivalents.
Claims (10)
1. A data processing method for use in a prepaid system, comprising:
acquiring transaction data in real time, and identifying the transaction type of the transaction data;
classifying the transaction types according to real-time account arrival and offline settlement;
performing data processing on the transaction type of the real-time account arrival by adopting a synchronous data transmission mode;
processing the transaction type of off-line settlement by adopting an asynchronous data transmission mode or a file data transmission mode; the interface pressure of the prepayment system is relieved, and the data processing efficiency of the prepayment system is improved.
2. The data processing method of claim 1, wherein the obtaining transaction data in real-time, the identifying the transaction type of the transaction data further comprises:
carrying out standardized character filtering operation on the acquired transaction data to obtain target transaction data;
carrying out object identification operation on the target transaction data to obtain a transaction object;
and calling a preset transaction type identification model to process the transaction data of the transaction object in the target transaction data to obtain the transaction type of the transaction object.
3. The data processing method of claim 1, wherein classifying the transaction types by real-time ledger and offline settlement further comprises:
the transaction data with the transaction type of recharging, withdrawing or transferring is marked as real-time account arrival and transmitted to a first data processing channel for real-time processing;
and marking the transaction data with the transaction type of deduction, refund or reimbursement as off-line settlement, and transmitting the transaction data to a message queue in the second data processing channel for processing.
4. The data processing method of claim 1, wherein the performing data processing on the real-time account-arriving transaction type by synchronous data transmission further comprises:
the prepayment system is used as a server side, HTTP interface service is provided, and an interface API is issued;
and after receiving the data sent by the client according to the interface API, the server side performs validity check on the multiple data, generates a transaction record and returns a prepayment result.
5. The data processing method of claim 1, wherein the data processing for the transaction type of offline settlement using asynchronous data transmission further comprises:
building a pre-payment prepositive service;
setting a pre-payment white list according to the transaction type;
the pre-payment prepositive service receives pre-payment data sent by a client in a kafka mode, and performs data validity check;
the pre-payment preposition service transmits the data after the verification is passed to the pre-payment system.
6. The data processing method of claim 1, wherein the data processing for the transaction type of offline settlement by file data transmission further comprises:
building a pre-payment prepositive service;
setting a pre-payment white list according to the transaction type, flexibly configuring a pre-payment sending state, and reducing pressure for the peak period of a pre-payment system;
loading a CSV file through a database and performing data processing, wherein the CSV file is a file generated in advance according to pre-payment data; generating the CSV file again and uploading the CSV file to a specified path according to the processed data;
the pre-payment prepositive service loads the CSV file under the specified path and performs data validity check;
the pre-payment prepositive service summarizes the data passing the verification according to the charging department, the charging item and the sending pre-payment time dimension;
the prepaid prefix service sends the aggregated data to a prepaid system.
7. A data processing apparatus for use in a prepaid system, comprising:
the data identification module is used for acquiring transaction data in real time and identifying the transaction type of the transaction data;
the data classification module is used for classifying the transaction types according to real-time account arrival and offline settlement;
the first data processing module is used for processing the transaction type of the real-time account arrival in a synchronous data transmission mode;
the second data processing module is used for processing the transaction type of the offline settlement in an asynchronous data transmission mode or a file data transmission mode; the interface pressure of the prepayment system is relieved, and the data processing efficiency of the prepayment system is improved.
8. The data processing apparatus of claim 7, wherein the data identification module further comprises:
the data filtering unit is used for carrying out standardized character filtering operation on the acquired transaction data to obtain target transaction data;
the object identification unit is used for carrying out object identification operation on the target transaction data to obtain a transaction object;
and the data processing unit is used for calling a preset transaction type identification model to process the transaction data of the transaction object in the target transaction data to obtain the transaction type of the transaction object.
9. A data processing apparatus, characterized by comprising: a memory and a processor, the memory having stored therein computer readable instructions which, when executed by the processor, cause the processor to perform the steps in the data processing method of any one of claims 1 to 6.
10. A storage medium storing computer readable instructions which, when executed by one or more processors, cause the one or more processors to perform the steps in the data processing method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210959739.XA CN115344633A (en) | 2022-08-11 | 2022-08-11 | Data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210959739.XA CN115344633A (en) | 2022-08-11 | 2022-08-11 | Data processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115344633A true CN115344633A (en) | 2022-11-15 |
Family
ID=83952138
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210959739.XA Pending CN115344633A (en) | 2022-08-11 | 2022-08-11 | Data processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115344633A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116959694A (en) * | 2023-05-30 | 2023-10-27 | 厦门大学附属中山医院 | Portable mobile nursing recording system |
-
2022
- 2022-08-11 CN CN202210959739.XA patent/CN115344633A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116959694A (en) * | 2023-05-30 | 2023-10-27 | 厦门大学附属中山医院 | Portable mobile nursing recording system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10339038B1 (en) | Method and system for generating production data pattern driven test data | |
| CN105447046A (en) | Distributed system data consistency processing method, device and system | |
| US11394794B2 (en) | Fast ingestion of records in a database using data locality and queuing | |
| US10455264B2 (en) | Bulk data extraction system | |
| CN108509392B (en) | Multi-mechanism account checking method, system, computer equipment and storage medium | |
| EP2171614A2 (en) | Transporting table valued parameter over tabular data stream protocol | |
| CN102929961A (en) | Data processing method and device thereof based on building quick data staging channel | |
| CN111897863A (en) | Multi-source heterogeneous data fusion and convergence method | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| US11297105B2 (en) | Dynamically determining a trust level of an end-to-end link | |
| KR101357135B1 (en) | Apparatus for Collecting Log Information | |
| US20120323924A1 (en) | Method and system for a multiple database repository | |
| CN107133231A (en) | A kind of data capture method and device | |
| CN112328708A (en) | Mixed data warehouse technology for real-time aggregation of multiple data sources | |
| CN115344633A (en) | Data processing method, device, equipment and storage medium | |
| CN107480283A (en) | Realize the method, apparatus and storage system of big data quick storage | |
| CN113836237A (en) | Method and device for auditing data operation of database | |
| CN110083501A (en) | Interface calls method of counting and device | |
| CN110309206B (en) | Order information acquisition method and system | |
| CN112363838A (en) | Data processing method and device, storage medium and electronic device | |
| CN102855297A (en) | Method for controlling data transmission, and connector | |
| CN111401819B (en) | Intersystem data pushing method and system | |
| CN112749197B (en) | Data fragment refreshing method, device, equipment and storage medium | |
| CN106557530B (en) | Operation system, data recovery method and device | |
| CN112232911A (en) | Asynchronous processing method and system for equity ordering |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |