CN115629848A - Business processing method and device, electronic equipment and storage medium - Google Patents
Business processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115629848A CN115629848A CN202211268135.7A CN202211268135A CN115629848A CN 115629848 A CN115629848 A CN 115629848A CN 202211268135 A CN202211268135 A CN 202211268135A CN 115629848 A CN115629848 A CN 115629848A
- Authority
- CN
- China
- Prior art keywords
- data
- preset
- service
- processing
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/466—Transaction processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a business processing method and a device thereof, electronic equipment and a storage medium, relating to the field of financial science and technology or other related fields, wherein the processing method comprises the following steps: receiving change message data of a target service sent by a preset server, starting a target thread according to a database configuration file, identifying data in a transaction of the same transaction in the change message data by adopting the target thread, storing the data in the transaction into a preset memory array, and processing the data in all the transactions in the preset memory array to obtain a processing result. The invention solves the technical problems that the service processing needs to be carried out by depending on the message middleware, the processing efficiency is lower and the data inconsistency is easily caused in the related technology.
Description
Technical Field
The invention relates to the field of financial science and technology, in particular to a business processing method and device, electronic equipment and a storage medium.
Background
At present, with the access of a large number of users to e-commerce applications, the services of e-commerce applications are required to have the capability of supporting the characteristics of high concurrency, high response and the like. For example, in a merchandise system, hundreds of thousands of active merchandise are simultaneously handled at 0 second from the start of sales promotion, and the merchandise is quickly modified in batch. If the state of a large amount of commodity information can be modified within the millisecond level of response time, functions (such as monitoring, logging, statistics, cache synchronization and the like) in the business logic are decoupled from the business main logic.
In the related technology, the service decoupling scheme mainly uses message middleware, and achieves service peak clipping decoupling by adding a message sending mode in service interface service and then asynchronously processing messages. Taking online payment as an example, firstly, deducting money on a user account, and secondly, notifying the user of successful payment through a short message. In order to improve the processing speed of the payment function interface, the message queue mode can be used for decoupling, and only the deduction is processed in the payment interface, and the result is returned when the deduction is successful. The function of processing the short message notification is modified into the function of sending messages to the message queue, and the short message notification is asynchronously processed by a message subscription program, so that the aim of improving the processing speed of the payment function interface is fulfilled.
However, by performing service decoupling in the above manner, the system becomes more complex, problems such as message repeat processing, message sequence, etc. need to be considered, and because of relying on the message middleware, once the message middleware fails, the system may crash, and the availability of the system is reduced. Moreover, if only part of the messages processed asynchronously are successful, the page still returns success, which finally results in data inconsistency and causes inconsistency problem.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
Embodiments of the present invention provide a service processing method and apparatus, an electronic device, and a storage medium, so as to solve at least the technical problems that in the related art, a message middleware is required to perform service processing, processing efficiency is low, and data inconsistency is easily caused.
According to an aspect of the embodiments of the present invention, there is provided a service processing method applied to a preset consumer, including: receiving change message data of a target service sent by a preset server, wherein the change message data comprises: a database configuration file; starting a target thread according to the database configuration file; adopting the target thread to identify data in the same transaction in the change message data, and storing the data in the transaction into a preset memory array; and processing the data in all the transactions in the preset memory array to obtain a processing result.
Optionally, before receiving change message data of a target service sent by a preset server, the method further includes: configuring database configuration information and message filtering rule configuration information for each database through the preset server, wherein the databases are located in different or same data sources; and generating the database configuration file for each database through the preset server based on the database configuration information and the message filtering rule configuration information.
Optionally, after generating the database configuration file for each database, the method further includes: monitoring a database log through the preset server; acquiring site information through the preset server according to the database log, wherein the site information at least comprises: a data update time point; and accessing a preset service registration center through the preset service terminal, and acquiring the change message data from the preset service registration center according to the site information.
Optionally, the step of accessing the preset service registry through the preset service end includes: configuring a database address and a table address of service data corresponding to the target service through the preset server according to the database configuration information; configuring a preset service address through the preset service terminal based on the database address and the table address; and accessing the preset service registration center through the preset service end according to the preset service address, wherein the preset service registration center is used for recording the site information of a database log, and the preset service registration center acquires database data corresponding to the site information.
Optionally, the step of identifying, by the target thread, data in a transaction that changes a same transaction in the message data includes: and identifying the operation data of the same transaction in the changed message data by adopting the target thread according to the message body type, wherein the message body type comprises the following steps: transaction start, transaction in, transaction end; and acquiring data in the transaction of the same transaction based on the operation data.
Optionally, the step of processing data in all the transactions in the preset memory array to obtain a processing result includes: analyzing data in all the transactions in the preset memory array by adopting a preset application to obtain analyzed data; according to the business rule of the target business, filtering the analysis data to obtain the processing result, wherein the processing result comprises: processing data, the processing data comprising: the data before and after the target service is updated, the operation type of the target service, the database identifier and the table identifier of the operation.
Optionally, after processing data in all the transactions in the preset memory array to obtain a processing result, the method further includes: according to the service requirement of the target service, sending a table identifier related to asynchronous task processing in the processing data to a processing system corresponding to the service requirement, wherein the processing system performs service logic processing on a target table indicated by the table identifier and a field in the target table according to the service requirement to obtain an asynchronous task processing result; assembling the asynchronous task processing results of different processing systems to obtain processing result data; and storing the processing result data to a preset database.
According to another aspect of the embodiments of the present invention, there is also provided a service processing apparatus, applied to a preset consuming side, including: a receiving unit, configured to receive change message data of a target service sent by a preset server, where the change message data includes: a database configuration file; the starting unit is used for starting a target thread according to the database configuration file; the identification unit is used for identifying the data in the transaction of the same transaction in the change message data by adopting the target thread and storing the data in the transaction into a preset memory array; and the processing unit is used for processing the data in all the transactions in the preset memory array to obtain a processing result.
Optionally, the service processing apparatus further includes: the system comprises a first configuration module, a second configuration module and a third configuration module, wherein the first configuration module is used for configuring database configuration information and message filtering rule configuration information for each database through a preset server before receiving change message data of target service sent by the preset server, and the databases are located in different or same data sources; and the first generating module is used for generating the database configuration file for each database according to the database configuration information and the message filtering rule configuration information of the preset server.
Optionally, the service processing apparatus further includes: the first monitoring module is used for monitoring database logs through the preset server after the database configuration file is generated for each database; a first obtaining module, configured to obtain, by the preset server, location information according to the database log, where the location information at least includes: a data update time point; and the first access module is used for accessing a preset service registration center through the preset service terminal and acquiring the change message data from the preset service registration center according to the site information.
Optionally, the first access module comprises: the first configuration submodule is used for configuring a database address and a table address of the service data corresponding to the target service through the preset server according to the database configuration information; the second configuration submodule is used for configuring a preset service address through the preset service end on the basis of the database address and the table address; the first access sub-module is used for accessing the preset service registration center through the preset service end according to the preset service address, wherein the preset service registration center is used for recording the site information of a database log, and the preset service registration center acquires database data corresponding to the site information.
Optionally, the identification unit comprises: a first identifying module, configured to identify, by using the target thread, operation data that is the same transaction in the change message data according to a message body type, where the message body type includes: transaction start, transaction in, transaction end; and the second acquisition module is used for acquiring the data in the transaction of the same transaction based on the operation data.
Optionally, the processing unit comprises: the first analysis module is used for analyzing the data in all the transactions in the preset memory array by adopting a preset application to obtain analyzed data; a first filtering module, configured to filter the analysis data according to a service rule of the target service to obtain the processing result, where the processing result includes: processing data, the processing data comprising: the data before and after the target service is updated, the operation type of the target service, the database identifier and the table identifier of the operation.
Optionally, the service processing apparatus further includes: the first sending module is used for sending a table identifier related to asynchronous task processing in the processing data to a processing system corresponding to the service requirement according to the service requirement of the target service after processing data in all the transactions in the preset memory array to obtain a processing result, wherein the processing system carries out service logic processing on a target table indicated by the table identifier and fields in the target table according to the service requirement to obtain an asynchronous task processing result; the first assembly module is used for assembling the asynchronous task processing results of different processing systems to obtain processing result data; and the first storage module is used for storing the processing result data to a preset database.
According to another aspect of the embodiments of the present invention, a computer-readable storage medium is further provided, where the computer-readable storage medium includes a stored computer program, and when the computer program runs, a device where the computer-readable storage medium is located is controlled to execute the service processing method.
According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the above-mentioned service processing method.
In the disclosure, change message data of a target service sent by a preset server is received, a target thread is started according to a database configuration file, the target thread is adopted to identify data in a transaction of the same transaction in the change message data, the data in the transaction is stored into a preset memory array, and the data in all transactions in the preset memory array are processed to obtain a processing result. In the method, the change message data of the target service collected by the preset server from the database can be received through the preset consumption end, then the target thread is started according to the corresponding database configuration file to identify the data in the transaction of the same transaction in the change message data, and the data in all the transactions in the transaction are stored into the preset memory array, so that the purpose of uniformly processing the same transaction message is achieved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention and do not constitute a limitation of the invention. In the drawings:
fig. 1 is a flow chart of an alternative method of service processing according to an embodiment of the present invention;
FIG. 2 is a flow chart of an alternative method for obtaining change message data according to an embodiment of the invention;
FIG. 3 is a schematic diagram of an alternative business processing system architecture in accordance with an embodiment of the present invention;
fig. 4 is a schematic diagram of an alternative service processing apparatus according to an embodiment of the present invention;
fig. 5 is a block diagram of a hardware structure of an electronic device (or a mobile device) for a service processing method according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It should be noted that the business processing method and the device thereof in the present disclosure may be used in the case of business processing in the field of financial technology, and may also be used in the case of business processing in any field other than the field of financial technology.
It should be noted that relevant information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for presentation, analyzed data, etc.) referred to in the present disclosure are information and data that are authorized by the user or sufficiently authorized by various parties. For example, an interface is provided between the system and the relevant user or organization, before obtaining the relevant information, an obtaining request needs to be sent to the user or organization through the interface, and after receiving the consent information fed back by the user or organization, the relevant information is obtained.
The embodiments of the invention described below may be applied to various systems/applications/devices that perform business processes. The invention is suitable for non-invasive service decoupling processing in a high concurrency scene, can perform asynchronous task processing on hundreds of millions of mass data under the condition of not invading service system codes, and comprises the following steps: asynchronous task processing such as cache data synchronization, compliance data monitoring, log processing, data statistical analysis, data consistency check and the like can realize function decoupling in a non-invasive asynchronous mode and realize a near-real-time response service function.
The present invention will be described in detail with reference to examples.
Example one
In accordance with an embodiment of the present invention, there is provided an embodiment of a business process method, it should be noted that the steps illustrated in the flowchart of the figure may be performed in a computer system such as a set of computer executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
Fig. 1 is a flowchart of an alternative service processing method according to an embodiment of the present invention, and as shown in fig. 1, the method includes the following steps:
step S101, receiving change message data of a target service sent by a preset server, wherein the change message data comprises: a database configuration file.
And step S102, starting the target thread according to the database configuration file.
Step S103, the data in the transaction of the same transaction in the change message data is identified by adopting the target thread, and the data in the transaction is stored into a preset memory array.
Step S104, processing the data in all the transactions in the preset memory array to obtain a processing result.
Through the steps, the change message data of the target service sent by the preset server can be received, the target thread is started according to the database configuration file, the target thread is adopted to identify the data in the transaction of the same transaction in the change message data, the data in the transaction is stored into the preset memory array, and the data in all the transactions in the preset memory array are processed to obtain the processing result. In the embodiment of the invention, the change message data of the target service collected by the preset server from the database can be received through the preset consumption end, then the target thread is started according to the corresponding database configuration file to identify the data in the transaction of the same transaction in the change message data, and the data in all the transactions in the transaction are stored into the preset memory array so as to achieve the purpose of uniformly processing the same transaction message.
The following will explain the embodiments of the present invention in detail with reference to the above steps. The embodiment of the invention can be applied to a preset consumption end.
An optional embodiment, before receiving change message data of a target service sent by a preset server, further includes: configuring database configuration information and message filtering rule configuration information for each database through a preset server, wherein the databases are located in different or same data sources; and generating a database configuration file for each database through the preset service terminal based on the database configuration information and the message filtering rule configuration information.
In the embodiment of the present invention, a preset server (e.g., a CanalServer service, that is, an architecture service, which can provide a data acquisition function, and achieve data acquisition from a database (e.g., a MySQL database) and send the data to a preset consumer (e.g., a CanalClient service, which is an architecture service for consuming data)), is introduced, the preset server can configure database configuration information and message filtering rule configuration information for each database (the databases are located in different or the same data sources), and then can generate a database configuration file for each database according to the database configuration information and the message filtering rule configuration information (that is, the database configuration information and the message filtering rule configuration information can be split into a plurality of configuration files according to the number of the databases at the preset server, each database corresponds to one configuration file), so that a system can flexibly process message distribution in a database table according to a business rule after commissioning, and a message filtering rule can be configured by an expression.
Optionally, after generating the database configuration file for each database, the method further includes: monitoring a database log through a preset server; obtaining the site information through a preset server according to the database log, wherein the site information at least comprises: a data update time point; and accessing the preset service registration center through the preset service terminal, and acquiring the change message data from the preset service registration center according to the site information.
In the embodiment of the invention, the preset server can simulate the slave database of the database and monitor the database log (such as binlog log) of the master database of the database, so that the data can be synchronized in near real time without code intrusion, and meanwhile, even if the preset server faces massive data, the preset server can not cause pressure on the database and can not influence on online services.
In the embodiment of the invention, the preset service end can monitor the database log, and then acquire the site information (the site information at least comprises a data updating time point) according to the database log, can access a preset service registration center (such as a zookeeper registration center which is used for recording the operation site information of the database log, namely the data updating time point of the current database operation, can monitor the data processing condition, and can judge whether delay exists or not through the site information), and can acquire the change message data from the preset service registration center according to the site information (namely, the same site information can be matched from the preset service registration center according to the site information, so as to acquire the number of the change messages from the database through the site information matched from the preset service registration center).
Fig. 2 is a flowchart of an alternative method for acquiring change message data according to an embodiment of the present invention, and as shown in fig. 2, the method includes the following steps:
step S201, monitoring a database log through a preset server;
step S202, obtaining the site information through a preset server according to the database log, wherein the site information at least comprises: a data update time point;
step S203, accessing the preset service registration center through the preset service end, and obtaining the change message data from the preset service registration center according to the site information.
Optionally, the step of accessing the preset service registration center through the preset service includes: configuring a database address and a table address of service data corresponding to a target service through a preset server according to database configuration information; configuring a preset service address through a preset service terminal based on a database address and a table address; and accessing a preset service registration center through a preset service end according to a preset service address, wherein the preset service registration center is used for recording the site information of the database log, and the preset service registration center acquires database data corresponding to the site information.
In the embodiment of the present invention, the preset service end may configure a database address and a table address of service data corresponding to a target service according to database configuration information, then configure a preset service address according to the database address and the table address, and then access a preset service registry according to the preset service address (that is, the preset service end may configure a database address and a corresponding table to be acquired according to service characteristics, and configure a preset service address to access the preset service registry), where the preset service registry is configured to record location information of a database log to monitor a data processing condition, and the preset service registry can obtain database data corresponding to the location information.
Step S101, receiving change message data of a target service sent by a preset server, wherein the change message data comprises: a database configuration file.
In the embodiment of the invention, after the preset server acquires the change message data of the target service, the change message data can be sent to the preset consumption end, and after the preset consumption end receives the change message data, the change message data can be processed according to the database configuration file carried in the change message data, so that the target service can be modified in near real time, and all modified data are recorded.
And step S102, starting a target thread according to the database configuration file.
In the embodiment of the invention, each database corresponds to a database configuration file of a preset server, and the preset consumption end can start an independent processing thread to change message data according to the database configuration file of the preset server, so that the changed message data in a plurality of databases cannot be confused, and the changed message data in a single thread is also sequential, therefore, the preset consumption end can start a corresponding target thread according to the database configuration file.
Step S103, recognizing and changing data in the same transaction in the message data by adopting the target thread, and storing the data in the transaction into a preset memory array.
Optionally, the step of identifying, by the target thread, data in a transaction that changes a same transaction in the message data includes: according to the type of the message body, identifying and changing the operation data of the same transaction in the message data by adopting a target thread, wherein the type of the message body comprises the following steps: transaction start, transaction in, transaction end; based on the operation data, data in a transaction of the same transaction is obtained.
In the embodiment of the present invention, in a processing thread of a preset consuming side, whether the processing thread is the same transaction operation or not may be distinguished according to a message body type, where the message body type includes: the method comprises the steps of transaction start transaction, transaction ROWDATA and transaction end transaction (namely, according to message body types, operation data of the same transaction in changed message data are identified by adopting a target thread), and then, collecting ROWDATA type messages in the transaction between the transaction start type message TRANSACTIONBEGIN and the transaction end type message TRANSACTIONNEND into a List memory array (namely, data in the transaction of the same transaction are obtained based on the operation data, and the data in the transaction are stored into a preset memory array) so as to achieve the purpose of processing the messages of the same transaction together.
Step S104, processing the data in all the transactions in the preset memory array to obtain a processing result.
Optionally, the step of processing data in all transactions in the preset memory array to obtain a processing result includes: analyzing data in all transactions in a preset memory array by adopting a preset application to obtain analyzed data; according to the business rule of the target business, filtering and analyzing the data to obtain a processing result, wherein the processing result comprises the following steps: processing data, the processing data comprising: the data before and after the target service is updated, the operation type of the target service, the database identifier and the table identifier of the operation.
In this embodiment of the present invention, a preset application (e.g., a CanalAPI) may be used to analyze data in all transactions in a preset memory array to obtain analyzed data, and then, according to a service rule of a target service, the analyzed data is filtered to obtain a processing result, where the processing result includes: processing data, the processing data comprising: data before and after updating the target service, operation type (such as update, delete, insert, etc.) for operating the target service, database identification and table identification of the operation, and the like.
Optionally, after processing data in all transactions in the preset memory array and obtaining a processing result, the method further includes: according to the service requirement of the target service, sending a table identifier related to asynchronous task processing in the processing data to a processing system corresponding to the service requirement, wherein the processing system performs service logic processing on a target table indicated by the table identifier and a field in the target table according to the service requirement to obtain an asynchronous task processing result; assembling asynchronous task processing results of different processing systems to obtain processing result data; and storing the processing result data into a preset database.
In the embodiment of the present invention, a table identifier related to asynchronous task processing in processing data may be sent to a processing system corresponding to a service requirement according to the service requirement of a target service, and then the processing system may perform service logic processing on a target table indicated by the table identifier and fields in the target table according to the service requirement to obtain asynchronous task processing results (i.e., the table requiring asynchronous task processing and the fields in the table are subjected to service logic processing according to the service requirement), then assemble asynchronous task processing results of different processing systems to obtain processing result data, and then store the processing result data in a preset database (i.e., the assembled asynchronous task processing result data is stored in a preset database (e.g., a task processing result database, or other file systems or non-relational databases may be used to store the processing result data).
For example, after the data change message is received by the CanalClient service, the data of the commodity category system can be asynchronously updated, and the final data consistency in the distributed system is achieved. For another example, after the canalclean service receives the data change message, the canalclean service can asynchronously modify and synchronize the price data in the price system, and the original system is not changed.
This is described in detail below in connection with an alternative embodiment.
Fig. 3 is a schematic diagram of an alternative service processing system architecture according to an embodiment of the present invention, as shown in fig. 3, including: a plurality of databases (such as database a, database B, database C, etc.), a default service end, a default consumption end, and a default database, wherein,
the preset server side provides a data acquisition function, acquires data from the database and sends the data to the preset consumption side.
The preset consumption end carries out a series of processing treatments such as analysis and filtration on the collected data according to the service characteristics, and then stores the data into a preset database.
In this embodiment, the service processing system may implement a backend service by using a programming language (such as Java), may be transversely extended to other applications, and is applicable to all scenes in which database operations need to be recorded.
The embodiment can adopt the open source architecture CanalServer service to collect data from the database. Canal has the following characteristics: the Canal can simulate the monitoring of the database logs of the database master library by the database slave library, code intrusion is avoided, near real-time synchronization of data can be achieved, and meanwhile, even if massive data are encountered, pressure cannot be caused on the database, and online services cannot be influenced.
The following describes the processing flows of the preset server and the preset consumer in detail:
(1) The preset service end configures database addresses and corresponding tables to be acquired according to service characteristics, and configures the preset service addresses to access a preset service registration center (the preset service registration center is used for recording operation site information of database logs, namely a data updating time point of current database operation, can monitor data processing conditions, and can judge whether delay exists or not through the site information). Because the database of each service system adopts a structure deployment mode of database and table division, the configuration information of the database and the configuration information of the message filtering rules can be split into a plurality of configuration files at the preset server side according to the number of the databases, so that the message distribution in the database table can be flexibly processed according to the service rules after each service system is put into production, the message filtering rules can be configured through regular expressions, and finally, the collected change message data is sent to the preset consumption side.
(2) The preset consumption end receives data sent by the preset server end, and each database corresponds to one database configuration file of the preset server end, so that the preset consumption end can start an independent processing thread to process messages according to the database configuration files of the preset server end, messages in a plurality of databases cannot be disordered, and messages in a single thread are sequential. In the processing thread of the preset consumption end, whether the operation is the same transaction or not can be distinguished according to the message body type, wherein the message body type comprises the following steps: transaction start type message transaction end type message transaction collect a List memory array to achieve the purpose of processing the messages of a transaction together.
And analyzing and filtering the data in the List memory array in one transaction through preset application, so as to obtain processing data such as data before and after updating, operation types (update, delete and insert), database names and table names of operations and the like. And finally, storing the assembled asynchronous task processing result data into a preset database.
In the embodiment of the invention, by introducing the preset service end and the preset consumption end, service decoupling processing of non-intrusive service system codes can be carried out on the service systems of the sub-base and sub-table in a high concurrency scene, the orderliness of messages can be ensured, and near-real-time data asynchronous processing is realized, so that the aim of non-intrusive asynchronous decoupling processing of services is fulfilled. Meanwhile, the embodiment has no code intrusion, does not affect an online service system, supports the near-real-time asynchronous task processing of hundreds of millions of data every day, can quickly integrate an asynchronous task system for a system adopting a MySQL database, and reduces the system access cost.
The following is a detailed description of another embodiment.
Example two
The service processing apparatus provided in this embodiment includes a plurality of implementation units, and each implementation unit corresponds to each implementation step in the first embodiment.
Fig. 4 is a schematic diagram of an alternative service processing apparatus according to an embodiment of the present invention, and as shown in fig. 4, the processing apparatus may include: a receiving unit 40, an initiating unit 41, a recognizing unit 42, a processing unit 43, wherein,
a receiving unit 40, configured to receive change message data of a target service sent by a preset server, where the change message data includes: a database configuration file;
a starting unit 41, configured to start a target thread according to the database configuration file;
the identifying unit 42 is configured to identify data in a transaction of the same transaction in the change message data by using a target thread, and store the data in the transaction into a preset memory array;
the processing unit 43 is configured to process data in all transactions in the preset memory array to obtain a processing result.
The processing apparatus may receive change message data of a target service sent by a preset server through the receiving unit 40, start a target thread through the starting unit 41 according to the database configuration file, identify data in a transaction of a same transaction in the change message data by using the target thread through the identifying unit 42, store the data in the transaction into a preset memory array, and process data in all transactions in the preset memory array through the processing unit 43 to obtain a processing result. In the embodiment of the invention, the change message data of the target service collected by the preset server from the database can be received through the preset consumption end, then the target thread is started according to the corresponding database configuration file to identify the data in the transaction of the same transaction in the change message data, and the data in all the transactions in the transaction are stored into the preset memory array so as to achieve the purpose of uniformly processing the same transaction message.
Optionally, the service processing apparatus further includes: the first configuration module is used for configuring database configuration information and message filtering rule configuration information for each database through a preset server before receiving change message data of a target service sent by the preset server, wherein the databases are located in different or same data sources; and the first generation module is used for generating a database configuration file for each database through the preset server based on the database configuration information and the message filtering rule configuration information.
Optionally, the service processing apparatus further includes: the first monitoring module is used for monitoring database logs through a preset server after a database configuration file is generated for each database; the first acquisition module is used for acquiring site information through a preset server according to a database log, wherein the site information at least comprises: a data update time point; and the first access module is used for accessing the preset service registration center through the preset service terminal and acquiring the change message data from the preset service registration center according to the site information.
Optionally, the first access module comprises: the first configuration submodule is used for configuring a database address and a table address of service data corresponding to the target service through a preset server according to the database configuration information; the second configuration submodule is used for configuring a preset service address through a preset service end on the basis of the database address and the table address; the first access sub-module is used for accessing a preset service registration center through a preset service end according to a preset service address, wherein the preset service registration center is used for recording the site information of a database log, and the preset service registration center acquires database data corresponding to the site information.
Optionally, the identification unit comprises: the first identification module is used for identifying and changing the operation data of the same transaction in the message data by adopting a target thread according to the message body type, wherein the message body type comprises: transaction start, transaction in, transaction end; and the second acquisition module is used for acquiring data in the same transaction based on the operation data.
Optionally, the processing unit comprises: the first analysis module is used for analyzing data in all transactions in a preset memory array by adopting a preset application to obtain analyzed data; the first filtering module is used for filtering and analyzing the data according to the business rule of the target business to obtain a processing result, wherein the processing result comprises: processing data, the processing data comprising: the data before and after the target service is updated, the operation type of the target service, the database identifier and the table identifier of the operation.
Optionally, the service processing apparatus further includes: the first sending module is used for sending a table identifier related to asynchronous task processing in the processing data to a processing system corresponding to a service requirement according to the service requirement of a target service after processing data in all transactions in a preset memory array to obtain a processing result, wherein the processing system carries out service logic processing on a target table indicated by the table identifier and fields in the target table according to the service requirement to obtain an asynchronous task processing result; the first assembly module is used for assembling asynchronous task processing results of different processing systems to obtain processing result data; and the first storage module is used for storing the processing result data to a preset database.
The processing device may further include a processor and a memory, and the receiving unit 40, the starting unit 41, the identifying unit 42, the processing unit 43, and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions.
The processor comprises a kernel, and the kernel calls a corresponding program unit from the memory. The kernel can be set to be one or more, and data in all transactions in the preset memory array is processed by adjusting kernel parameters to obtain a processing result.
The memory may include volatile memory in a computer readable medium, random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.
The present application further provides a computer program product adapted to perform a program for initializing the following method steps when executed on a data processing device: receiving change message data of a target service sent by a preset server, starting a target thread according to a database configuration file, identifying data in a transaction of the same transaction in the change message data by adopting the target thread, storing the data in the transaction into a preset memory array, processing the data in all the transactions in the preset memory array, and obtaining a processing result.
According to another aspect of the embodiments of the present invention, a computer-readable storage medium is further provided, where the computer-readable storage medium includes a stored computer program, and when the computer program runs, a device on which the computer-readable storage medium is located is controlled to execute the service processing method described above.
According to another aspect of embodiments of the present invention, there is also provided an electronic device, including one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the service processing method described above.
Fig. 5 is a block diagram of a hardware structure of an electronic device (or a mobile device) for a service processing method according to an embodiment of the present invention. As shown in fig. 5, the electronic device may include one or more (shown as 502a, 502b, \8230;, 502 n) processors 502 (processors 502 may include, but are not limited to, processing devices such as microprocessor MCUs or programmable logic devices FPGAs), memory 504 for storing data. Besides, the method can also comprise the following steps: a display, an input/output interface (I/O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the I/O interface), a network interface, a keyboard, a power supply, and/or a camera. It will be understood by those skilled in the art that the structure shown in fig. 5 is only an illustration and is not intended to limit the structure of the electronic device. For example, the electronic device may also include more or fewer components than shown in FIG. 5, or have a different configuration than shown in FIG. 5.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described apparatus embodiments are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or may not be executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in the form of hardware, or may also be implemented in the form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. A service processing method is applied to a preset consumer side, and comprises the following steps:
receiving change message data of a target service sent by a preset server, wherein the change message data comprises: a database configuration file;
starting a target thread according to the database configuration file;
adopting the target thread to identify data in the transaction of the same transaction in the change message data, and storing the data in the transaction into a preset memory array;
and processing the data in all the transactions in the preset memory array to obtain a processing result.
2. The service processing method according to claim 1, before receiving the change message data of the target service sent by the preset server, further comprising:
configuring database configuration information and message filtering rule configuration information for each database through the preset server, wherein the databases are located in different or same data sources;
and generating the database configuration file for each database through the preset server based on the database configuration information and the message filtering rule configuration information.
3. The transaction processing method of claim 2, further comprising, after generating the database configuration file for each of the databases:
monitoring a database log through the preset server;
acquiring site information through the preset server according to the database log, wherein the site information at least comprises: a data update time point;
and accessing a preset service registration center through the preset service terminal, and acquiring the change message data from the preset service registration center according to the site information.
4. The traffic processing method according to claim 3, wherein the step of accessing the predetermined service registry through the predetermined service terminal comprises:
configuring a database address and a table address of service data corresponding to the target service through the preset server according to the database configuration information;
configuring a preset service address through the preset service terminal based on the database address and the table address;
and accessing the preset service registration center through the preset service end according to the preset service address, wherein the preset service registration center is used for recording the site information of a database log, and the preset service registration center acquires database data corresponding to the site information.
5. The transaction processing method according to claim 1, wherein the step of identifying data in a transaction of the same transaction in the change message data using the target thread comprises:
and identifying the operation data of the same transaction in the changed message data by adopting the target thread according to the message body type, wherein the message body type comprises the following steps: transaction start, transaction in, transaction end;
and acquiring data in the transaction of the same transaction based on the operation data.
6. The service processing method according to claim 1, wherein the step of processing data in all the transactions in the preset memory array to obtain a processing result comprises:
analyzing data in all the transactions in the preset memory array by adopting a preset application to obtain analyzed data;
according to the business rule of the target business, filtering the analysis data to obtain the processing result, wherein the processing result comprises: processing data, the processing data comprising: the data before and after the target service is updated, the operation type of the target service, the database identifier and the table identifier of the operation.
7. The service processing method according to claim 6, after processing data in all the transactions in the preset memory array to obtain a processing result, further comprising:
according to the service requirement of the target service, sending a table identifier related to asynchronous task processing in the processing data to a processing system corresponding to the service requirement, wherein the processing system performs service logic processing on a target table indicated by the table identifier and a field in the target table according to the service requirement to obtain an asynchronous task processing result;
assembling the asynchronous task processing results of different processing systems to obtain processing result data;
and storing the processing result data to a preset database.
8. A service processing device, applied to a preset consumer side, includes:
a receiving unit, configured to receive change message data of a target service sent by a preset server, where the change message data includes: a database configuration file;
the starting unit is used for starting a target thread according to the database configuration file;
the identification unit is used for identifying data in the same transaction in the change message data by adopting the target thread and storing the data in the transaction into a preset memory array;
and the processing unit is used for processing the data in all the transactions in the preset memory array to obtain a processing result.
9. A computer-readable storage medium, comprising a stored computer program, wherein when the computer program is run, the computer-readable storage medium is controlled by a device to execute the service processing method according to any one of claims 1 to 7.
10. An electronic device, comprising one or more processors and memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the business processing method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211268135.7A CN115629848A (en) | 2022-10-17 | 2022-10-17 | Business processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211268135.7A CN115629848A (en) | 2022-10-17 | 2022-10-17 | Business processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115629848A true CN115629848A (en) | 2023-01-20 |
Family
ID=84903818
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211268135.7A Pending CN115629848A (en) | 2022-10-17 | 2022-10-17 | Business processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115629848A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115801788A (en) * | 2023-02-01 | 2023-03-14 | 深圳市思为软件技术有限公司 | Event delivery method and device, electronic equipment and storage medium |
-
2022
- 2022-10-17 CN CN202211268135.7A patent/CN115629848A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115801788A (en) * | 2023-02-01 | 2023-03-14 | 深圳市思为软件技术有限公司 | Event delivery method and device, electronic equipment and storage medium |
| CN115801788B (en) * | 2023-02-01 | 2023-05-16 | 深圳市思为软件技术有限公司 | Event delivery method, device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108153670B (en) | Interface testing method and device and electronic equipment | |
| CN110909063B (en) | User behavior analysis method and device, application server and storage medium | |
| CN108734028B (en) | Data management method based on block chain, block chain link point and storage medium | |
| CN111355802B (en) | Information pushing method and device | |
| CN113553310B (en) | Data acquisition method and device, storage medium and electronic equipment | |
| CN109445768B (en) | Database script generation method and device, computer equipment and storage medium | |
| CN112017007A (en) | User behavior data processing method and device, computer equipment and storage medium | |
| CN111611249A (en) | Data management method, device, equipment and storage medium | |
| CN115629848A (en) | Business processing method and device, electronic equipment and storage medium | |
| CN112434243A (en) | Method, device and computer readable storage medium for synchronizing data | |
| CN111008207A (en) | Resource data processing method, device and equipment | |
| CN114448972A (en) | Distributed storage log compression downloading method, system, terminal and storage medium | |
| CN114461705A (en) | Data visualization generation method and device, electronic equipment and storage medium | |
| CN113434742A (en) | Account screening method and device, storage medium and electronic device | |
| CN110309206B (en) | Order information acquisition method and system | |
| CN110362540B (en) | Data storage and visitor number acquisition method and device | |
| CN116957813A (en) | Wind control strategy testing method and device, electronic equipment and readable storage medium | |
| CN111506564A (en) | Remote data management method and device based on CS (circuit switched) architecture, computer equipment and storage medium | |
| CN114895879B (en) | Management system design scheme determining method, device, equipment and storage medium | |
| CN114611039B (en) | Analysis method and device of asynchronous loading rule, storage medium and electronic equipment | |
| US20160117689A1 (en) | Process and apparatus for assigning a match confidence metric for inferred match modeling | |
| CN114742547A (en) | Internet online collection control method, device, medium and electronic equipment | |
| CN114490865A (en) | Database synchronization method, device, equipment and computer storage medium | |
| CN114528010A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN112860456A (en) | Log processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |