CN111625535A - Method, device and storage medium for realizing business data association - Google Patents

Method, device and storage medium for realizing business data association Download PDF

Info

Publication number
CN111625535A
CN111625535A CN202010306366.7A CN202010306366A CN111625535A CN 111625535 A CN111625535 A CN 111625535A CN 202010306366 A CN202010306366 A CN 202010306366A CN 111625535 A CN111625535 A CN 111625535A
Authority
CN
China
Prior art keywords
service
database
business
message
log information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010306366.7A
Other languages
Chinese (zh)
Other versions
CN111625535B (en
Inventor
陈宏亮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Seashell Housing Beijing Technology Co Ltd
Original Assignee
Beike Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beike Technology Co Ltd filed Critical Beike Technology Co Ltd
Priority to CN202010306366.7A priority Critical patent/CN111625535B/en
Publication of CN111625535A publication Critical patent/CN111625535A/en
Application granted granted Critical
Publication of CN111625535B publication Critical patent/CN111625535B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2358—Change logging, detection, and notification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2379—Updates performed during online database operations; commit processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application discloses a method, a device and a storage medium for realizing business data association, wherein the method is applied to a business data association system, the business data association system and the business system are decoupled, and the business system comprises a business subsystem; the business data association system carries out the following steps: respectively monitoring the service database log information of the service databases corresponding to the service subsystems and respectively storing the generated service data in the service databases corresponding to the service subsystems; the message producer packages the log information of each monitored service database into service messages and pushes the service messages to corresponding message queues; when business data association is carried out, the business data is used as a message consumer to pull the business message from each message queue and obtain the log information of the business database; and judging whether the log information of the service database contains service data needing to be associated according to a preset association condition.

Description

Method, device and storage medium for realizing business data association
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method, an apparatus, and a storage medium for implementing service data association.
Background
With the development of computer and internet technologies, various industries increasingly use the internet to perform online operations or transactions. The system provided for satisfying the user requirement is called a service system, and a complex service system may also comprise a plurality of independent service subsystems. After a user logs in a service system, a large amount of service data can be generated in the process of man-machine interaction. To better meet the user's needs, the business system will typically store the relevant business data in a database, such as the MYSQL database. And then, extracting the associated service data from the database to provide services for the user.
The existing service system usually performs synchronous processing when service data are generated, which not only focuses on the problem of service processing itself, but also focuses on the problem of correlation between service data in a plurality of service subsystems, and the coupling of the service processing and the service data correlation is very strong, which causes the problem of the reduction of the stability performance of the service system itself.
Disclosure of Invention
Aiming at the prior art, the embodiment of the invention discloses a method for realizing business data association, which can overcome the coupling problem of two logics of business processing and business data association without a business subsystem concerning data association and increase the stability of a business system.
The method for realizing the business data association is applied to a business data association system, the business data association system and the business system are decoupled, and the business system comprises a business subsystem; the business data association system carries out the following steps:
respectively monitoring service database log information of service databases corresponding to the service subsystems, wherein the service subsystems are systems independently realizing services without performing service data association, and respectively storing the generated service data in the service databases corresponding to the service subsystems; the message producer packages the log information of each monitored service database into service messages and pushes the service messages to corresponding message queues; when business data association is carried out, the business data is used as a message consumer to pull the business message from each message queue and obtain the log information of the business database; judging whether the log information of the service database contains service data needing to be associated according to a preset association condition, and if so, storing the service data needing to be associated in an association database established in advance; and if not, discarding the service message.
Further, the step of monitoring the service database log information of the service database corresponding to each service subsystem includes: and taking the service database corresponding to each service subsystem as a master side, taking the monitoring party as a slave side, and monitoring the log information of the service database according to a binlog master-slave synchronization protocol.
Further, the step of serving as a message producer to encapsulate the monitored log information of each service database into service messages respectively and push the service messages to the corresponding message queues includes: analyzing the monitored log information of the service database into a statement format of the service database from a binary file format; packaging the service database log information in the statement format of the service database into service database log information in a json format; and the business database log information in the json format is used as a business message and pushed to a message queue corresponding to the business subsystem as a message producer.
Further, the step of judging whether the log information of the service database contains service data to be associated according to a preset association condition includes: analyzing a database name from the journal information of the service database in the json format, wherein the database name belongs to the association condition; judging whether the analyzed database name is the database name needing to be associated, if so, continuing to execute, and if not, discarding the service message; analyzing a database table name from the journal information of the business database in the json format, wherein the database table name belongs to the association condition; judging whether the analyzed database table name is the database table name needing to be associated, if so, continuing to execute, and if not, discarding the service message; analyzing a database operation type from the service database log information in the json format, wherein the database operation type belongs to the association condition; judging whether the analyzed database operation type is the database operation type needing to be associated or not, if so, continuing to execute, and if not, discarding the service message; analyzing a database table attribute column from the journal information of the business database in the json format, wherein the database table attribute column belongs to the association condition; judging whether the analyzed database table attribute column is the database table attribute column needing to be associated, if so, taking the value in the analyzed database table attribute column as the service data needing to be associated, and if not, discarding the service message.
Further, the step of storing the service data to be associated in a pre-established association database includes: judging whether business data needing to be correlated is stored in the correlation database, if not, storing the business data in a corresponding database table attribute column in the correlation database; if the data is stored, updating the value of the corresponding database table attribute column according to the service data; and the association database is provided with corresponding database table attribute columns in advance according to the business data needing to be associated of each business subsystem.
The embodiment of the application also discloses a device for realizing the business data association, which does not need the business subsystem to pay attention to the data association, can overcome the coupling problem of two logics of business processing and business data association, and increases the stability of the business system.
A device for realizing business data association is applied to a business data association system, the business data association system and the business system are decoupled, and the business system comprises a business subsystem; the device comprises: the monitoring module is used for respectively monitoring the service database log information of the service databases corresponding to the service subsystems, wherein the service subsystems are systems which independently realize services and do not perform service data association, and respectively store the generated service data in the service databases corresponding to the service subsystems; the message producer packages the log information of each monitored service database into service messages and pushes the service messages to corresponding message queues; the message queue module is used for storing the message queue; the filtering module is used for pulling the service message from each message queue as a message consumer and acquiring the log information of the service database when service data association is carried out; judging whether the log information of the service database contains service data needing to be associated according to a preset association condition, and if so, storing the service data needing to be associated in an association database established in advance; and if not, discarding the service message.
Further, the listening module comprises: the monitoring execution module is used for respectively monitoring the service database log information of the service databases corresponding to the service subsystems, wherein the service subsystems are systems for independently realizing services, and the generated service data are respectively stored in the service databases corresponding to the service subsystems; and the message pushing module is used as a message producer to respectively package the log information of each monitored service database into service messages and push the service messages to the corresponding message queues.
Furthermore, the monitoring execution module monitors the log information of the service database according to a binlog master-slave synchronization protocol by taking the service database corresponding to each service subsystem as a master side and taking a monitor as a slave side.
Further, the message pushing module comprises: the first analysis module is used for analyzing the monitored log information of the business database from a binary file format into a business database statement format; the packaging module is used for packaging the business database log information in the business database statement format into business database log information in a json format; and the pushing execution module is used as a message producer to push the business database log information in the json format to a message queue corresponding to the business subsystem as a business message.
Further, the filtration module comprises: the system comprises a message pulling module, a filtering execution module and an association execution module; the message pulling module is used for pulling the service message from each message queue as a message consumer and obtaining the log information of the service database when service data association is carried out; the filtering execution module is used for judging whether the log information of the service database contains service data needing to be associated according to a preset association condition, and if so, triggering the association execution module; if not, discarding the service message; and the association execution module is used for storing the service data needing to be associated in an association database established in advance.
Further, the filtering execution module comprises a first filtering module, a second filtering module, a third filtering module and a fourth filtering module; the first filtering module is configured to analyze a database name from the service database log information in the json format, where the database name belongs to the association condition; judging whether the analyzed database name is the database name needing to be associated, if so, continuing to execute, and if not, discarding the service message; the second filtering module is used for analyzing a database table name from the business database log information in the json format, wherein the database table name belongs to the association condition; judging whether the analyzed database table name is the database table name needing to be associated, if so, continuing to execute, and if not, discarding the service message; the third filtering module is configured to analyze a database operation type from the service database log information in the json format, where the database operation type belongs to the association condition; judging whether the analyzed database operation type is the database operation type needing to be associated or not, if so, continuing to execute, and if not, discarding the service message; the fourth filtering module is configured to analyze a database table attribute column from the json-formatted service database log information, where the database table attribute column belongs to the association condition; judging whether the analyzed database table attribute column is a database table attribute column needing to be associated, if so, taking the value in the analyzed database table attribute column as service data needing to be associated, and triggering the association execution module; and if the attribute column is not the database table attribute column to be associated, discarding the service message.
Further, the association executing module includes: the judging module is used for judging whether the business data needing to be associated is stored in the association database or not, and if not, the storing module is triggered; if the data is saved, the updating module is triggered; the association database is provided with corresponding database table attribute columns in advance according to the business data needing to be associated of each business subsystem; the storage module is used for storing the business data in a corresponding database table attribute column in the associated database; and the updating module is used for updating the values of the corresponding database table attribute columns according to the service data.
The embodiment of the present application further discloses a computer-readable storage medium, on which computer instructions are stored, and the instructions, when executed by a processor, can implement the steps in any of the above-mentioned methods for implementing business data association.
The embodiment of the present application further discloses an electronic device for implementing service data association, where the electronic device includes: a processor; a memory for storing the processor-executable instructions; the processor is configured to read the executable instructions from the memory, and execute the instructions to implement any of the steps in the method for implementing business data association described above.
To sum up, the embodiment of the present application discloses a method, an apparatus, and a storage medium for implementing service data association, which monitor service database log information of a service database corresponding to each service subsystem, determine service data to be associated according to association conditions, and store the service data to be associated in an association database established in advance. The business system only executes the business of the business system and associates the business data outside the business system, thereby overcoming the problem that the business system not only carries out business processing logic, but also carries out strong coupling of business data association logic and enhancing the stability of the business system.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive labor.
Fig. 1 is a schematic view of an application scenario for implementing service data association according to an embodiment of the present application.
Fig. 2 is a flowchart of a first embodiment of a method for implementing service data association by the service data association system S2.
Fig. 3 is a schematic view of an application scenario for implementing service data association according to a second embodiment of the present application.
Fig. 4 and fig. 5 are flowcharts of a second embodiment of the method for implementing service data association by the service data association system S2.
Fig. 6 is a schematic view of an application scenario of the third embodiment of the method of the present application.
Fig. 7 is a flowchart illustrating a service message generation part for the same subsystem H1 according to a third embodiment of the present invention.
Fig. 8 is a flowchart illustrating a service message consumption part for the same subsystem H1 according to a third embodiment of the present invention.
Fig. 9 is a flow chart of a business message generation part for the performance subsystem Y1 in the third embodiment of the method of the present application.
Fig. 10 is a flow chart of the business message consumption part for the performance subsystem Y1 in the third embodiment of the method of the present application.
Fig. 11 is a schematic structural diagram of a first embodiment of the apparatus of the present application.
Fig. 12 is a schematic structural diagram of a second embodiment of the apparatus of the present application.
Fig. 13 is a schematic internal structural diagram of the message pushing module M12 in the second embodiment of the application apparatus.
Fig. 14 is a schematic diagram of the internal structure of the filtering execution module M32 in the second embodiment of the apparatus according to the present application.
Fig. 15 is a schematic internal structural diagram of an associated execution module M33 in the second embodiment of the apparatus according to the present application.
Fig. 16 is a schematic structural diagram of an electronic device for implementing service data association according to the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprising" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, article, or apparatus.
The technical solution of the present invention will be described in detail with specific examples. Several of the following embodiments may be combined with each other and some details of the same or similar concepts or processes may not be repeated in some embodiments.
In the embodiment of the application, the service system does not concern the association problem of the service data, and only plays a role of independently realizing the service, and the special role of realizing the association of the service data concerns the association problem of the service data. On one hand, each service subsystem in the service system is respectively provided with a corresponding service database, and when each service subsystem independently realizes the service, the generated service data is respectively stored in the corresponding service database and is recorded through the log information of the service database; on the other hand, the special role for realizing the business data association can monitor the log information of each business database, determine the business data needing to be associated from the log information, and store the business data in the associated database, thereby achieving the purpose of associating the business data of each business subsystem.
Fig. 1 is a schematic view of an application scenario for implementing service data association according to an embodiment of the present application. As shown in fig. 1, the scenario includes a business system S1 and a business data association system S2. The service system S1 comprises a plurality of service subsystems S11 and corresponding service databases S12; the business data association system S2 includes an association performing subsystem S21 and an association database S22. Each service subsystem S11 realizes the service independently and stores the service data generated in the realization process in the corresponding service database S12. The association execution subsystem S21 is responsible for monitoring the service databases S12, associating the service data that needs to be associated in the service subsystems, and storing the associated service data in the association database S22.
Fig. 2 is a flowchart of a first embodiment of a method for implementing service data association by the service data association system S2. As shown in fig. 2, the method is applied to a service data association system, where the service data association system and the service system are decoupled, and the service system includes a service subsystem; specifically, the method comprises the following steps:
step 201: and respectively monitoring the service database log information of the service database S12 corresponding to each service subsystem S11, wherein the service subsystem S11 is a system for independently realizing services, and respectively storing the generated service data in the service database S12 corresponding to the service subsystem S3526.
As described above in fig. 1, each service subsystem S11 stores the generated service data in the corresponding service database S12, and the information operated on the service database S12 forms service database log information. For example, a service database is MYSQL database, and information operated on the database forms binlog database log. The service data association system S2 in this step needs to monitor the service database log information in each service database S12, so as to obtain the service database log information.
Step 202: and the message producer packages the monitored log information of each service database into service messages and pushes the service messages to corresponding message queues.
Step 203: and when the business data association is carried out, the business data is used as a message consumer to pull the business message from each message queue and obtain the log information of the business database.
The above steps 202 and 203 are methods for processing the monitored log information of each service database. Because each service subsystem S11 independently completes each service, the time for generating service data is different, the time for generating service database log information is different, and the time for filtering the service data to be correlated is different, the embodiment of the application adopts an asynchronous message queue mode, thereby overcoming the problem of inconsistent generation and processing speed. Therefore, the service database log information generated by each service database S12 can be monitored in time, and a large amount of service database log information cannot impact the association execution subsystem S21.
Step 204: judging whether the log information of the service database contains service data needing to be associated according to a preset association condition, and if so, executing step 205; if not, step 206 is performed.
Step 205: and storing the service data needing to be associated in a pre-established association database S22.
Step 206: and discarding the service message.
The above steps 204 to 206 are methods for processing the log information of the service database, and the service data to be associated needs to be selected from a large amount of log information of the service database according to the association condition, and associated to the association database S22. In practical applications, since each service subsystem S11 generates a relatively large amount of service data, only a small amount of specific service data is usually required for association. Of course, which service data needs to be associated is determined by actual user needs, that is, the above-mentioned association condition is set reasonably.
In practical applications, the service data association system S2 will continuously push the monitored database log information to the message queue, and will also continuously obtain the service database log information from the message queue, and continuously store the service data that needs to be associated in the association database S22 according to the association condition. Therefore, in the embodiment of the present application, the steps 201 to 202 may be regarded as a service message generation part, and the steps 203 to 206 may be regarded as a service message consumption part, and the two parts may be executed repeatedly and circularly, and may be executed asynchronously and independently, and the above-mentioned step sequence is only convenient for description, and cannot have a limitation on the actual execution sequence.
By applying the scheme of the embodiment of the application, the service system S1 does not concern about the association problem of the service data, and only needs to execute the service itself. However, since the embodiment of the present application enables the function of the service database log, any record of the operation of the service database S12 is stored in the service database log information, which necessarily includes the service data to be associated. Therefore, as long as the log information of the service database is monitored and processed outside the service system S1, the association of the service data is asynchronously implemented, and the service system S1 does not need to synchronously concern the data association when processing the service, thereby solving the problem of strong coupling between the service processing logic and the service data association logic and enhancing the stability of the service system S1.
For a better understanding of the present invention, reference will now be made in detail to the preferred embodiments. Fig. 3 is a schematic view of an application scenario of the second embodiment of the method of the present application. The scenario includes a business system S1 and a business data association system S2. The service system S1 includes several service subsystems S11 and corresponding service databases S12; the business data association system S2 includes an association performing subsystem S21 and an association database S22. Several groups of functions of monitoring, message queuing and filtering are included in the associated execution subsystem S21, and each group independently corresponds to different service subsystems S11 and service databases S12. Here, the embodiment of the present application assumes that the service database S12 and the association database S22 are MYSQL databases.
Fig. 4 and fig. 5 are flowcharts of a second embodiment of the method for implementing service data association by the service data association system S2. The method at least comprises two parts of service message generation and service message consumption, wherein the service message generation part is shown in figure 4 and comprises steps 401-404; the service message consumption part is shown in FIG. 5 and comprises steps 501-514. The following describes in detail the implementation of the second embodiment of the method.
As shown in fig. 4, the service message generation part of the method of the embodiment of the present application includes:
step 401: and taking the service database corresponding to each service subsystem as a master side, taking the monitoring party as a slave side, and monitoring the log information of the service database according to a binlog master-slave synchronization protocol.
The service database S12 in the embodiment of the present application is a MYSQL database, and the binlog log function of the service database may be started. As known to those skilled in the art, binlog is used to record information for MYSQL database operations, and is typically stored in a binary file format. In addition, in order to avoid data loss, the working MYSQL database can be set as a master side, another MYSQL database is equipped as a slave side, and the business database log information of the master side is transmitted to the slave side through a binlog master-slave synchronization protocol to serve as a backup.
The embodiment of the application monitors the MYSQL database by using a binlog master-slave synchronization protocol to obtain the log information of the service database. It should be noted that, in the embodiment of the present application, whether a standby database is set for a working MYSQL database is not concerned, but a binlog master-slave synchronization protocol is used to complete monitoring of log information of a service database. That is, the service database corresponding to the service subsystem is used as the master side, the monitoring party service data association system S2 is used as the slave side, and the master side transmits the service database log information to the slave side according to the binlog master-slave synchronization protocol, so that the service data association system S2 monitors the service database log information. In practical application, if there are a plurality of service subsystems S11 and service databases S12, the binlog log function of each service database S12 can be turned on and monitored respectively. The embodiment of the application realizes monitoring by using the binlog master-slave synchronization protocol, does not need to change the original function of the service system, obtains the log information of the service database under the condition that the service system is not sensitive, and ensures the stability of the service system.
Step 402: and analyzing the monitored log information of the service database into a statement format of the service database from a binary file format.
In the embodiment of the present application, since the service database S12 is a MYSQL database, the monitored log information of the service database is in a binary file format, and needs to be parsed into a format of MYSQL database statements.
Step 403: and packaging the service database log information in the statement format of the service database into the service database log information in the json format.
In practical application, because the readability of the log information of the business database in the statement format of the business database is poor, the subsequent application of the log information of the business database is not facilitated, and in order to enhance the identifiability of the log information of the business database, the log information of the business database is packaged into a json format.
Step 404: and as a message producer, pushing the service database log information in the json format as a service message to a message queue corresponding to the service subsystem, and returning to the step 401.
To this end, the service data association system S2 completes a process of generating a service message, and pushes the generated service message to a message queue, where each service message is service database log information in json format.
After the service message is pushed to the message queue, on one hand, the service message generation part can return to the step 401 to continue monitoring the log information of the service database, and repeatedly execute the steps 401-404 to generate a new service message; on the other hand, the service message consumption part may perform the following steps 501 to 514 to filter the service messages in the message queue to determine the service data that needs to be associated.
When filtering the service message, the embodiment of the present application sets the association condition in advance, which may include "database name", "database table name", "database operation type", and "database table attribute column". Wherein, the "database name" represents the identifier of each service database; a plurality of tables, namely database tables, are stored in the business database, and the name of each database table represents the identifier of each database table; each database table includes several columns for recording information, these columns are also commonly referred to as attributes, and thus are referred to as "database table attribute columns" in the embodiments of the present application; the "database operation type" indicates the operation type of the service subsystem S11 on the service database S12 in the process of generating service data, such as "add", "delete", "change", "check", and so on.
Although there are several service databases S12, and each service database S12 will generate its own service data, in practical applications, it may be only necessary to associate some service data in some service databases S12, and the rest of service data are useless. Therefore, the embodiment of the present application sets the association condition, and the specific manner thereof may be set according to the actual situation, and is not limited by the embodiment of the present application.
As shown in fig. 5, the service message consumption part includes:
step 501: and when the business data association is carried out, the business data is used as a message consumer to pull the business message from each message queue and obtain the log information of the business database.
Step 502: and analyzing a database name from the service database log information in the json format, wherein the database name belongs to the association condition.
Step 503: judging whether the analyzed database name is the database name needing to be associated, if so, continuing to execute the step 504; if not, step 514 is executed.
The steps 502-503 are performed for the first filtering according to the database name.
Step 504: analyzing a database table name from the journal information of the business database in the json format, wherein the database table name belongs to the association condition.
Step 505: judging whether the analyzed database table name is the database table name needing to be associated, if so, continuing to execute the step 506; if not, step 514 is performed.
The steps 504-505 are performed for the second filtering according to the database table name.
Step 506: analyzing a database operation type from the service database log information in the json format, wherein the database operation type belongs to the correlation condition.
Step 507: judging whether the analyzed database operation type is the database operation type needing to be associated, if so, continuing to execute the step 508; if not, step 514 is executed.
The steps 506-507 are to perform the third filtering according to the operation type of the database.
Step 508: analyzing a database table attribute column from the service database log information in the json format, wherein the database table attribute column belongs to the association condition.
Step 509: judging whether the analyzed database table attribute column is the database table attribute column needing to be associated, if so, executing step 510; if not, step 514 is performed.
The steps 508-509 are performed by performing a fourth filtering according to the database table attribute column.
Step 510: and taking the value in the attribute column of the analyzed database table as the service data needing to be associated.
Through the four times of filtering of the database name, the database table name, the database operation type and the database table attribute column, the service data meeting the association condition can be accurately extracted from the service message.
Step 511: judging whether the business data needing to be associated is stored in the association database, if not, executing step 512; if so, step 513 is performed.
Step 512: and storing the service data in a corresponding database table attribute column in the association database, and returning to step 501.
Step 513: and updating the value of the corresponding database table attribute column according to the service data, and returning to the step 501.
The steps 511 to 513 are processes of storing the filtered business data into the association database S22 in association. In this embodiment of the application, the association database S22 may also be a MYSQL database, where a plurality of database table attribute columns are set to represent the association of service data in different service subsystems. Assuming that three database table attribute columns needing to be associated are set in the association database in advance, and a1 and a2 meeting the association condition are filtered from the business data from the business database a, a1 and a2 may be first stored in the association database S22, that is, the values of the three database table attribute columns are: (A1, A2, empty). Then, if B1 meeting the association condition is filtered out from the business data from the business database B, the values of the three database table attribute columns are changed to: (A1, A2, B1) to achieve the purpose of associating the service data in different service subsystems.
Step 514: the service message is discarded and returns to step 501.
At this point, the service data association system S2 completes a service message consumption process, and then repeatedly executes steps 501 to 514 to consume the next new service message, and stores the service data to be associated in the association database S22 in this way.
The method and the device open the binlog function of the MYSQL database, monitor log information of the business database by using a binlog master-slave synchronization protocol, transmit business messages by using an asynchronous message queue, filter out business data needing to be associated by using set association conditions, and store the business data in the association database to realize association. Therefore, the scheme of the embodiment of the application not only decouples the service processing logic and the service data association logic, but also can monitor and accurately filter the service data under the condition that a service system is not sensible.
Fig. 6 is a schematic view of an application scenario of the third embodiment of the method of the present application. Assume that the scenario includes a business system T1 and a business data association system T2. The business system T1 is a system for processing a certain transaction, including processing a transaction contract and processing transaction performance, which are two business subsystems that can be operated independently. Thus, business system T1 includes contract subsystem H1 and performance subsystem Y1, where contract subsystem H1 corresponds to contract database H2 and business data generated by contract subsystem H1 will be stored in contract database H2; performance subsystem Y1 corresponds to performance database Y2 and business data generated by performance subsystem Y1 will be stored in performance database Y2. The contract database H2 mainly records information relating to contracts, such as contract ID numbers, contract amounts, and the like. The performance database Y2 primarily records information relating to contract performance, such as audit status and the like. The business data association system T2 includes an association execution subsystem T21 and an association database T22, wherein the association execution subsystem T21 includes functions of listening, message queuing, and filtering, and the association database T22 is used for storing associated business data. Like the above-described embodiment, the contract database H2, the performance database Y2, and the association database T22 of the embodiment of the present application are MYSQL databases. Of course, the actual application may not be MYSQL database, as long as the business data can be saved and the log information of the business database can be monitored. For convenience of description, the third embodiment of the present application only lists two service subsystems, and in practical applications, more service subsystems may be included, and the method is the same as that in the third embodiment of the present application and is not limited by the number of the service subsystems.
Specifically, it is assumed that the contract database H2 of the embodiment of the present application is named "db _ extract", and includes a database table named "t _ extract", where the database table includes two database table attribute columns "extract _ id" and "extract _ amount"; the performance database Y2 is named "db _ perf" and includes a database table named "t _ perf" that includes two database table attribute columns, "extract _ id" and "perf _ status". Business data (contract ID number and contract amount) generated by the contract subsystem H1 will be stored in the two database table attribute columns "contract _ ID" and "contract _ account" of the contract database H2, respectively, and business data (contract ID number and audit status) generated by the performance subsystem Y1 will be stored in the two database table attribute columns "contract _ ID" and "perf _ status" of the performance database Y2, respectively. In practical applications, the contract subsystem H1 may further include other business databases, several other database tables may be included in the contract database H2, and other database table attribute columns may be included in the database table "t _ contact". Similarly, the performance subsystem Y1 may also include other business databases, several other database tables may be included in the performance database Y2, and other database table attribute columns may be included in the database table "t _ perf". In addition, the database operation type "changedType" may be "insert", "update", "delete", "select".
In this embodiment, it is assumed that the business data generated by the contract subsystem H1 and the business data generated by the performance subsystem Y1 need to be associated with each other, and the association conditions need to be set in advance:
1) the association condition of the business data for the contract subsystem H1 may be set as: the database name "db _ extract" + database table name "t _ extract" + database operation type is "insert" or "update" + database table attribute columns are "extract _ id" and "extract _ amount".
2) The conditions of association of the business data for performance subsystem Y1 may be set to: the database name "db _ perf" + database table name "t _ perf" + database operation type "insert" or "update" + database table attribute columns are "contact _ id" and "perf _ status".
In the third embodiment of the present application, a flowchart of a method for implementing business data association for the same subsystem H1 and a flowchart of a method for implementing business data association for the performance subsystem Y1 are respectively described below. Wherein fig. 7 shows the flow of the service message generation section for the contract subsystem H1 and fig. 8 shows the flow of the service message consumption section for the contract subsystem H1. Fig. 9 shows the flow of the business message generation section for the performance subsystem Y1, and fig. 10 shows the flow of the business message consumption section for the performance subsystem Y1.
As shown in fig. 7, the flow of the service message generation part for the contract subsystem H1 includes:
step 701: and monitoring the service database log information of the contract database H2 according to a binlog master-slave synchronization protocol by taking the contract database H2 corresponding to the contract subsystem H1 as a master side and taking the monitoring party as a slave side. This step is the same as the step 401 in the second embodiment, and the binlog log function of the contract database H2 is started, so that the binlog master-slave synchronization protocol is used to monitor the contract database H2, so as to obtain the service database log information of the contract database H2. Step 702: and analyzing the monitored log information of the service database into a statement format of the service database from a binary file format. This step is the same as step 402 in the second embodiment. Step 703: and packaging the service database log information in the statement format of the service database into the service database log information in the json format. This step is the same as step 403 in the second embodiment. Assuming that the contract subsystem H1 newly establishes a contract according to the user's requirement, and the contract ID number and the contract amount are recorded in the contract database H2, the service database log information of the contract database H2 needs to record the contract ID number, the contract amount and the database operation type, which can be expressed as follows in json format:
Figure BDA0002455925080000111
wherein "contract _ ID" represents a contract ID number, the value of which is 1; "contract _ account" represents the contract amount, which has a value of 200000 dollars; "changedDb" represents a database name, and its value is "db _ extract" (i.e., the database name of contract database H2 is "db _ extract"); "table" represents a database table name with a value of "t _ extract" (i.e., the name of the table operating on the affinity database H2 is "t _ extract"); "changedType" represents the database operation type, with a value of "insert". The above-mentioned service database log information may indicate that a new contract is added in the "t _ contract" table of the "db _ contract" database by the operation of "insert", and the contract ID number of the new contract is 1 and the contract amount is 200000 yuan. Table one below may represent the added information in the contracts database H2:
contract ID number Amount of contract
… …
1 200000
Watch 1
Step 704: and as a message producer, pushing the transaction database log information in the json format as a transaction message to a contract message queue corresponding to the contract subsystem H1, and returning to step 701. This step is the same as step 404 of the second embodiment of the method described above. Similar to the method embodiment, the steps 701 to 704 may be repeatedly executed for the service message generation part of the contract subsystem H1 to package the monitored log information of other service databases into service messages and push the service messages to the corresponding contract message queues.
As shown in fig. 8, the flow of the service message consumption part for the contract subsystem H1 includes:
step 801: when the business data association is carried out, the business data association server is used as a message consumer to pull the business message from the contract message queue and obtain the log information of the business database. The step is the same as step 501 of the second method embodiment, where the message queue is specifically a contract message queue corresponding to the contract subsystem H1, the service database log information is database log information generated by the contract database H2, and the obtained information is the service database log information in json format in step 703:
Figure BDA0002455925080000121
step 802: and analyzing a database name from the service database log information in the json format, wherein the database name belongs to the association condition. This step is the same as step 502 of method embodiment two. According to the specific service database log information in step 801 of the embodiment of the method, the name of the database analyzed here should be "db _ extract", and the specific analysis method is as follows:
String changedDb＝JSON.parseObject(json).get("changedDb")；
step 803: judging whether the analyzed database name is the database name needing to be associated, if so, continuing to execute the step 804; if not, go to step 814. The steps 802 to 803 are the first filtering according to the database name, which is the same as the steps 502 to 503 in the second embodiment of the method. According to the association condition set in the embodiment of the method, the database name "db _ extract" analyzed in step 802 is the same as the database name "db _ extract" in the association condition, and it can be determined that the analyzed service database log information is from the database that needs to be associated in this embodiment. If it is not from the database that the embodiment needs to associate with, the pulled service message may be discarded and the next service message may be pulled from the contract message queue via step 814. Step 804: analyzing a database table name from the journal information of the business database in the json format, wherein the database table name belongs to the association condition. This step is the same as step 804 of method embodiment two. According to the specific service database log information in step 801 of the embodiment of the method, the name of the database table that is analyzed here should be "t _ contact", and the specific analysis method is as follows:
String table＝JSON.parseObject(json).get("table")；
step 805: judging whether the analyzed database table name is the database table name needing to be associated, if so, continuing to execute step 806; if not, step 814 is performed. The steps 804 to 805 are performed for the second filtering according to the database table name, which is the same as the steps 504 to 505 in the second embodiment of the method. According to the association condition set in the embodiment of the method, the database table name "t _ extract" analyzed in step 804 is the same as the database table name "t _ extract" in the association condition, and it can be determined that the analyzed service database log information is from the database table that needs to be associated in this embodiment. If it is not from the database table that needs to be associated in this embodiment, the pulled service message may be discarded and the next service message may be pulled from the contract message queue via step 814.
Step 806: analyzing a database operation type from the service database log information in the json format, wherein the database operation type belongs to the correlation condition. This step is the same as step 506 of method embodiment two. According to the specific service database log information in step 801 of the embodiment of the method, the operation type of the analyzed database should be "insert", and the specific analysis method is as follows:
String changedType＝JSON.parseObject(json).get("changedType")；
step 807: judging whether the analyzed database operation type is the database operation type needing to be associated, if so, continuing to execute the step 808; if not, go to step 814. The steps 806-807 are performed for the third filtering according to the database operation type, which is the same as the steps 506-507 in the second embodiment of the method. According to the association condition set in this embodiment of the method, the database operation type "insert" analyzed in step 806 is the same as the database operation type "insert" in the association condition, and it may be determined that the analyzed service database log information is information obtained through the database operation type that needs to be associated in this embodiment. If the type of database operation to be associated is not the one from the present embodiment, the pulled service message may be discarded and the next service message may be pulled from the contract message queue via step 814. Step 808: analyzing a database table attribute column from the service database log information in the json format, wherein the database table attribute column belongs to the association condition. This step is the same as step 508 of method embodiment two. According to the specific service database log information of step 801 of the embodiment of the method, the attribute columns of the database table should be "extract _ id" and "extract _ amount" as parsed out here. Step 809: judging whether the analyzed database table attribute column is the database table attribute column needing to be associated, if so, executing step 810; if not, step 814 is performed. The steps 808-809 are performed for the fourth filtering according to the database table attribute column, which is the same as the steps 508-509 in the second embodiment of the method. According to the association condition set by the embodiment of the method, the database table attribute columns ("extract _ id" and "extract _ amount") parsed in step 808 are the same as the database table attribute columns ("extract _ id" and "extract _ amount") in the association condition, and it can be determined that the parsed service database log information includes the database table attribute column that needs to be associated. If the database table attribute column that needs to be associated is not included, the pulled service message may be discarded and the next service message pulled from the contract message queue via step 814. Step 810: and taking the value in the attribute column of the analyzed database table as the service data needing to be associated. Through the four filtering of the database name "db _ extract", the database table name "t _ extract", the database operation type "insert", the database table attribute column "extract _ id" and the "extract _ amount", the service data meeting the association condition can be accurately extracted from the service message. Here, it is assumed that the value of "extract _ id" is "1" and the value of "extract _ amount" is "200000". Step 811: judging whether the business data needing to be associated is stored in the association database, if not, executing a step 812; if so, step 813 is performed. Step 812: and storing the service data in the corresponding database table attribute column in the association database, and returning to the step 801. Step 813: and updating the values of the attribute columns of the corresponding database table according to the service data, and returning to the step 801. The above steps 811 to 813 are the same as the steps 511 to 513 in the second embodiment of the method. Since the embodiment of the present method does not store the related service data in advance, the values of "extract _ id" and "extract _ amount" should be stored in the association database T22. Assuming that the association database T22 has set the database table attribute column to be associated, the stored contents are as shown in the following table two:
contract_id contract_amount perf_status
… … …
1 200000 ——
watch two
It can be seen that the method embodiment saves the values of "extract _ id" and "extract _ amount" to the association database T22 through step 812, but the value of "perf _ status" is empty. Step 814: the traffic message is discarded and returns to step 801. To this end, the service data association system T2 completes a service message consumption process. In practical application, the steps 801 to 814 can be repeatedly executed to consume the next new service message in the contract message queue, and the service data needing to be associated is stored in the association database T22 in the same manner.
As shown in fig. 9, the flow of the business message generation portion for performance subsystem Y1 includes:
step 901: and monitoring the service database log information of the performance database Y2 according to the binlog master-slave synchronization protocol by taking the performance database Y2 corresponding to the performance subsystem Y1 as a master side and taking the monitor as a slave side. This step is the same as the step 401 in the second embodiment, and the binlog log function of the performance database Y2 is turned on, so that the binlog master-slave synchronization protocol is used to monitor the performance database Y2 to obtain the service database log information of the performance database Y2. Step 902: and analyzing the monitored log information of the service database into a statement format of the service database from a binary file format. This step is the same as step 402 in the second embodiment. Step 903: and packaging the service database log information in the statement format of the service database into the service database log information in the json format. This step is the same as step 403 in the second embodiment. Assuming that the performance subsystem Y1 newly establishes a contract audit according to the transaction situation, and the achievement database Y2 records the contract ID number and the audit state, the business database log information of the performance database Y2 needs to record the contract ID number and the audit state, and can be encapsulated into the business database log information in json format according to the method of step 703. Table three below may indicate information added to the performance database Y2, where a median value of the audit state may be set in advance to "10" to indicate a contract audit initialization state, "20" to indicate that the contract audit is passed, "30" to indicate that the contract audit is rejected, and so on.
Contract ID number Audit status
… …
1 10
Watch III
Step 904: the message producer pushes the transaction database log information in the json format as a transaction message to a performance message queue corresponding to the performance subsystem Y1, and returns to step 901. This step is the same as step 404 of the second embodiment of the method described above. Similar to the embodiment of the method, the business message generation part of the performance subsystem Y1 may further repeatedly execute the steps 901-904 to package the monitored log information of other business databases into business messages and push the business messages to the corresponding performance message queues
As shown in fig. 10, the flow of the business message consumption portion to the performance subsystem Y1 includes:
step 1001: when the business data correlation is carried out, the business data correlation is used as a message consumer to pull the business message from the performance message queue and obtain the log information of the business database. The step is the same as the step 501 of the second embodiment of the method, the message queue is specifically a performance message queue corresponding to the performance subsystem Y1, the business database log information is database log information generated by the performance database Y2, and the obtained information is the business database log information in json format in the step 903. Step 1002: and analyzing a database name from the service database log information in the json format, wherein the database name belongs to the association condition. This step is the same as step 502 of method embodiment two. Step 1003: judging whether the analyzed database name is the database name needing to be associated, if so, continuing to execute the step 804; if not, go to step 1014. The steps 1002 to 1003 are the first filtering according to the database name, and are the same as the steps 502 to 503 in the second embodiment of the method. Step 1004: analyzing a database table name from the journal information of the business database in the json format, wherein the database table name belongs to the association condition. This step is the same as step 804 of method embodiment two. Step 1005: judging whether the analyzed database table name is the database table name needing to be associated, if so, continuing to execute the step 1006; if not, step 1014 is performed. The steps 1004-1005 are performed for the second filtering according to the database table name, which is the same as the steps 504-505 in the second embodiment of the method. Step 1006: analyzing a database operation type from the service database log information in the json format, wherein the database operation type belongs to the correlation condition. This step is the same as step 506 of method embodiment two. Step 1007: judging whether the analyzed database operation type is the database operation type needing to be associated, if so, continuing to execute the step 1008; if not, then step 1014 is executed. The steps 1006-1007 are performed by performing a third filtering according to the operation type of the database, which is the same as the steps 506-507 in the second embodiment of the method. Step 1008: analyzing a database table attribute column from the service database log information in the json format, wherein the database table attribute column belongs to the association condition. This step is the same as step 508 of method embodiment two. According to the service database log information in step 1001 of the embodiment of the method, it is assumed that the database table attribute columns are "extract _ id" and "perf _ status" are analyzed here. Step 1009: judging whether the analyzed database table attribute column is a database table attribute column needing to be associated, if so, executing step 1010; if not, then step 1014 is performed. The steps 1008 to 1009 are the fourth filtering according to the database table attribute column, which is the same as the steps 508 to 509 in the second embodiment of the method. According to the association condition set by the embodiment of the method, the database table attribute columns ("extract _ id" and "perf _ status") parsed in step 1008 are the same as the database table attribute columns ("extract _ id" and "perf _ status") in the association condition, and it can be determined that the parsed service database log information contains the database table attribute columns that need to be associated. If the database table attribute column that needs to be associated is not included, the pulled service message can be discarded and the next service message pulled from the contract message queue via step 1014. Step 1010: and taking the value in the attribute column of the analyzed database table as the service data needing to be associated. Through the four times of filtering of the database name db _ perf, the database table name t _ perf, the database operation type insert, the database table attribute column contact _ id and the perf _ status, the service data meeting the correlation condition can be accurately extracted from the service message. Here, it is assumed that the value of "extract _ id" is "1" and the value of "perf _ status" is "10". Step 1011: judging whether the business data needing to be associated is stored in the association database, if not, executing step 1012; if so, step 1013 is performed. Step 1012: and storing the service data in a corresponding database table attribute column in the association database, and returning to the step 1001. Step 1013: and updating the value of the corresponding database table attribute column according to the service data, and returning to the step 1001. The steps 1011 to 1013 are the same as the steps 511 to 513 in the second embodiment of the method. Since the embodiment of the present method already stores the relevant service data in step 812, the association database T22 should be updated according to the values of "extract _ id" and "perf _ status" in step 1013, and the updated content is shown in table four below:
contract_id contract_amount perf_status
… … …
1 200000 10
watch four
It can be seen that the method embodiment updates "perf _ status" that was originally empty to "10" via step 1013. In this way, the business data ("contract _ id" and "contract _ amount") in the contract subsystem H1 is associated with the business data ("contract _ id" and "perf _ status") in the performance subsystem Y1. Step 1014: the traffic message is discarded and the process returns to step 1001. To this end, the service data association system T2 completes a service message consumption process. In practical application, the steps 1001 to 1014 can be repeatedly executed to consume the next new service message in the contract message queue, and the service data needing to be associated is stored in the association database T22 in the same way.
In the embodiment of the application, the contract subsystem H1 only focuses on the logic related to its own service, and does not participate in the service data association logic, and the association performing subsystem T21 stores the service data to be associated with the contract subsystem H1 in the association database T22 by the method shown in fig. 7 and 8. The performance subsystem Y1 also only focuses on the logic related to its own business, and does not participate in the business data association logic, and the association performing subsystem T21 stores the business data to be associated with the performance subsystem Y1 in the association database T22 by the method shown in fig. 9 and 10. Therefore, under the condition of ensuring that the two service subsystems are stable respectively, the service data of different service subsystems are accurately associated, so that the service is better provided for the user.
Based on the method in the foregoing embodiment, the present application further provides a device for implementing service data association, that is, an association execution subsystem S21, where the device is applied to a service data association system, the service data association system and a service system are decoupled, and the service system includes a service subsystem. Fig. 11 is a schematic structural diagram of a first embodiment of the apparatus of the present application. As shown in fig. 11, the apparatus includes: a monitoring module M1, a message queue module M2 and a filtering module M3. Wherein:
a monitoring module M1, configured to monitor service database log information of service databases corresponding to the service subsystems, respectively, where the service subsystems are systems that independently implement services and do not perform service data association, and store the generated service data in the service databases corresponding to the service subsystems respectively; and the message producer packages the monitored log information of each service database into service messages and pushes the service messages to corresponding message queues. A message queue module M2, configured to store the message queue. The filtering module M3 is configured to, when performing service data association, serve as a message consumer to pull the service message from each message queue and obtain the service database log information; judging whether the log information of the service database contains service data needing to be associated according to a preset association condition, and if so, storing the service data needing to be associated in an association database established in advance; and if not, discarding the service message. That is, the monitoring module M1 monitors the service database log information of the service database corresponding to each service subsystem respectively; the monitoring module M1, as a message producer, packages the log information of each monitored service database into service messages and pushes the service messages to corresponding message queues; when the filtering module M3 associates service data, it is used as a message consumer to pull the service message from each message queue and obtain the log information of the service database; judging whether the log information of the service database contains service data needing to be associated according to a preset association condition, and if so, storing the service data needing to be associated in an association database established in advance; and if not, discarding the service message.
By applying the scheme of the embodiment of the application, the service system only needs to execute the service of the service system, and the problem of service data association does not need to be concerned. The execution subsystem S21 monitors and processes the log information of the service database, and asynchronously realizes the association of the service data, thereby solving the problem of strong coupling between the service processing logic and the service data association logic and enhancing the stability of the service system.
Fig. 12 is a schematic structural diagram of a second embodiment of the apparatus of the present application. As shown in fig. 12, the apparatus still includes a listening module M1, a message queue module M2, and a filtering module M3. The listening module M1 includes a listening execution module M11 and a message pushing module M12. The filtering module M3 comprises a message pulling module M31, a filtering execution module M32 and an association execution module M33. Specifically, the method comprises the following steps:
and the monitoring execution module M11 is configured to monitor service database log information of the service databases corresponding to the service subsystems, where the service subsystems are systems that independently implement services, and store the generated service data in the service databases corresponding to the service subsystems. Specifically, the monitoring execution module M11 is configured to monitor the log information of the service database according to the binlog master-slave synchronization protocol, with the service database corresponding to each service subsystem as the master side and the monitor itself as the slave side. And the message pushing module M12 is configured to serve as a message producer to encapsulate the monitored log information of each service database into service messages respectively and push the service messages to corresponding message queues. And the message pulling module M31 is configured to, when performing service data association, serve as a message consumer to pull the service message from each message queue and obtain the service database log information. A filtering execution module M32, configured to determine, according to a preset association condition, whether the log information of the service database includes service data that needs to be associated, and if so, trigger the association execution module; and if not, discarding the service message. And the association execution module M33 is configured to store the service data to be associated in an association database established in advance.
Fig. 13 is a schematic internal structural diagram of the message pushing module M12 in the second embodiment of the application apparatus. As shown in fig. 13, the message pushing module M12 includes a first parsing module M121, an encapsulating module M122, and a pushing execution module M123, where:
the first parsing module M121 is configured to parse the monitored service database log information from a binary file format into a service database statement format. And the encapsulating module M122 is configured to encapsulate the service database log information in the service database statement format into service database log information in a json format. And the pushing execution module M123, as a message producer, pushes the json-formatted service database log information as a service message to a message queue corresponding to the service subsystem.
Fig. 14 is a schematic diagram of the internal structure of the filtering execution module M32 in the second embodiment of the apparatus according to the present application. As shown in fig. 14, the filtering execution module M32 includes: a first filtering module M321, a second filtering module M322, a third filtering module M323, a fourth filtering module M324, wherein:
a first filtering module M321, configured to analyze a database name from the service database log information in the json format, where the database name belongs to the association condition; and judging whether the analyzed database name is the database name needing to be associated, if so, continuing to execute, and if not, discarding the service message. A second filtering module M322, configured to analyze a database table name from the json-formatted service database log information, where the database table name belongs to the association condition; and judging whether the analyzed database table name is the database table name needing to be associated, if so, continuing to execute, and if not, discarding the service message. A third filtering module M323, configured to analyze a database operation type from the service database log information in the json format, where the database operation type belongs to the association condition; judging whether the analyzed database operation type is the database operation type needing to be associated or not, if so, continuing to execute, and if not, discarding the service message. A fourth filtering module M324, configured to parse a database table attribute column from the json-formatted service database log information, where the database table attribute column belongs to the association condition; judging whether the analyzed database table attribute column is a database table attribute column needing to be associated, if so, taking the value in the analyzed database table attribute column as service data needing to be associated, and triggering the association execution module; and if the attribute column is not the database table attribute column to be associated, discarding the service message.
Fig. 15 is a schematic internal structural diagram of an associated execution module M33 in the second embodiment of the apparatus according to the present application. As shown in fig. 15, the association execution module M33 includes: a judging module M331, a saving module M332 and an updating module M333, wherein:
a judging module M331, configured to judge whether the service data to be associated is already stored in the association database, and if not, trigger the storing module M332; if so, triggering the update module M333; and the association database is provided with corresponding database table attribute columns in advance according to the business data needing to be associated of each business subsystem. A storing module M332, configured to store the service data in a corresponding database table attribute column in the association database. And the updating module M333 is configured to update the value of the corresponding database table attribute column according to the service data.
That is, the monitor execution module M11 monitors the service database log information of the service database corresponding to each service subsystem. The message pushing module M12 is configured to serve as a message producer to encapsulate the monitored log information of each service database into service messages respectively, and push the service messages to corresponding message queues. Specifically, the first parsing module M121 parses the monitored log information of the service database from a binary file format into a statement format of the service database; the packaging module M122 packages the service database log information in the service database statement format into service database log information in a json format; and the pushing execution module M123 serves as a message producer to push the json-format service database log information serving as a service message to a message queue corresponding to the service subsystem. Thereafter, the message pull module M31 acts as a message consumer to pull the service message from each message queue and obtain the service database log information. The first filtering module M321, the second filtering module M322, the third filtering module M323, and the fourth filtering module M324 filter the service data to be associated from the service database log information. The judging module M331 judges whether the business data to be associated has been stored in the association database, and if not, the storing module M332 is triggered to store the business data in the corresponding database table attribute column in the association database; if the data is saved, the triggering updating module M333 updates the value of the corresponding database table attribute column according to the service data.
By applying the scheme of the second embodiment of the application device, the service system does not need to pay attention to the problem of service data association, and the service execution subsystem S21 automatically monitors the service data to be associated in each service subsystem and stores the association in the association database, so that the problem of strong coupling between service processing logic and service data association logic is solved, and the stability of the service system is enhanced.
Embodiments of the present application further provide a computer-readable storage medium, which stores instructions that, when executed by a processor, may perform the steps in the method for implementing service data association as described above. In practical applications, the computer readable medium may be included in the apparatus/device/system described in the above embodiments, or may exist alone without being assembled into the apparatus/device/system. According to embodiments disclosed herein, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example and without limitation: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing, without limiting the scope of the present disclosure. In the embodiments disclosed herein, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
As shown in fig. 16, an embodiment of the present invention further provides an electronic device for implementing service data association. As shown in fig. 16, a schematic structural diagram of an electronic device according to an embodiment of the present invention is shown, specifically: the electronic device may include a processor 1601 of one or more processing cores, memory 1602 of one or more computer-readable storage media, and a computer program stored on the memory and executable on the processor. The method of service data association may be implemented when executing the program of the memory 1602. Specifically, in practical applications, the electronic device may further include a power source 1603, an input/output unit 1604, and the like. Those skilled in the art will appreciate that the configuration of the electronic device shown in fig. 16 is not intended to be limiting of the electronic device and may include more or fewer components than shown, or some components in combination, or a different arrangement of components. Wherein: the processor 1601 is a control center of the electronic apparatus, connects various parts of the whole electronic apparatus by various interfaces and lines, and performs various functions of the server and processes data by running or executing software programs and/or modules stored in the memory 1602 and calling data stored in the memory 1602, thereby performing overall monitoring of the electronic apparatus. The memory 1602 may be used to store software programs and modules, i.e., the computer-readable storage media described above. The processor 1601 executes various functional applications and data processing by executing software programs and modules stored in the memory 1602. The memory 1602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the server, and the like. Further, the memory 1602 may include high-speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 1602 may also include a memory controller to provide the processor 1601 access to the memory 1602. The electronic device further includes a power source 1603 for supplying power to each component, and the power source may be logically connected to the processor 1601 by a power management system, so that functions of managing charging, discharging, power consumption management and the like are realized by the power management system. The power supply 1603 may further include any component of one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device may also include an input-output unit 1604, the input-output unit 1604 being operable to receive entered numeric or character information and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control. The input unit output 1604 may also be used to display information entered by or provided to the user as well as various graphical user interfaces that may be composed of graphics, text, icons, video, and any combination thereof.
The flowchart and block diagrams in the figures of the present application illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments disclosed herein. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not explicitly recited in the present application. In particular, the features recited in the various embodiments and/or claims of the present application may be combined and/or coupled in various ways, all of which fall within the scope of the present disclosure, without departing from the spirit and teachings of the present application. The principles and embodiments of the present invention are explained herein using specific examples, which are provided only to help understanding the method and the core idea of the present invention, and are not intended to limit the present application. It will be appreciated by those skilled in the art that changes may be made in this embodiment and its broader aspects and without departing from the principles, spirit and scope of the invention, and that all such modifications, equivalents, improvements and equivalents as may be included within the scope of the invention are intended to be protected by the claims.

Claims (10)

1. A method for realizing business data association is characterized in that the method is applied to a business data association system, the business data association system and the business system are decoupled, and the business system comprises a business subsystem; the business data association system carries out the following steps:
respectively monitoring service database log information of service databases corresponding to the service subsystems, wherein the service subsystems are systems independently realizing services without performing service data association, and respectively storing the generated service data in the service databases corresponding to the service subsystems;
the message producer packages the log information of each monitored service database into service messages and pushes the service messages to corresponding message queues;
when business data association is carried out, the business data is used as a message consumer to pull the business message from each message queue and obtain the log information of the business database;
judging whether the log information of the service database contains service data needing to be associated according to a preset association condition, and if so, storing the service data needing to be associated in an association database established in advance; and if not, discarding the service message.
2. The method of claim 1, wherein the step of monitoring the service database log information of the service database corresponding to each service subsystem comprises:
and taking the service database corresponding to each service subsystem as a master side, taking the monitoring party as a slave side, and monitoring the log information of the service database according to a binlog master-slave synchronization protocol.
3. The method according to claim 1 or 2, wherein the step of serving as a message producer to encapsulate the monitored log information of each service database into service messages respectively and push the service messages to the corresponding message queues comprises:
analyzing the monitored log information of the service database into a statement format of the service database from a binary file format;
packaging the service database log information in the statement format of the service database into service database log information in a json format;
and the business database log information in the json format is used as a business message and pushed to a message queue corresponding to the business subsystem as a message producer.
4. The method according to claim 3, wherein the step of determining whether the service database log information contains service data to be associated according to a preset association condition comprises:
analyzing a database name from the journal information of the service database in the json format, wherein the database name belongs to the association condition;
judging whether the analyzed database name is the database name needing to be associated, if so, continuing to execute, and if not, discarding the service message;
analyzing a database table name from the journal information of the business database in the json format, wherein the database table name belongs to the association condition;
judging whether the analyzed database table name is the database table name needing to be associated, if so, continuing to execute, and if not, discarding the service message;
analyzing a database operation type from the service database log information in the json format, wherein the database operation type belongs to the association condition;
judging whether the analyzed database operation type is the database operation type needing to be associated or not, if so, continuing to execute, and if not, discarding the service message;
analyzing a database table attribute column from the journal information of the business database in the json format, wherein the database table attribute column belongs to the association condition;
judging whether the analyzed database table attribute column is the database table attribute column needing to be associated, if so, taking the value in the analyzed database table attribute column as the service data needing to be associated, and if not, discarding the service message.
5. The method of claim 4, wherein the step of storing the service data to be associated in a pre-established association database comprises:
judging whether business data needing to be correlated is stored in the correlation database, if not, storing the business data in a corresponding database table attribute column in the correlation database; if the data is stored, updating the value of the corresponding database table attribute column according to the service data; and the association database is provided with corresponding database table attribute columns in advance according to the business data needing to be associated of each business subsystem.
6. A device for realizing business data association is characterized in that the device is applied to a business data association system, the business data association system and the business system are decoupled, and the business system comprises a business subsystem; the device comprises:
the monitoring module is used for respectively monitoring the service database log information of the service databases corresponding to the service subsystems, wherein the service subsystems are systems which independently realize services and do not perform service data association, and respectively store the generated service data in the service databases corresponding to the service subsystems; the message producer packages the log information of each monitored service database into service messages and pushes the service messages to corresponding message queues;
the message queue module is used for storing the message queue;
the filtering module is used for pulling the service message from each message queue as a message consumer and acquiring the log information of the service database when service data association is carried out; judging whether the log information of the service database contains service data needing to be associated according to a preset association condition, and if so, storing the service data needing to be associated in an association database established in advance; and if not, discarding the service message.
7. The apparatus of claim 6, wherein the listening module comprises:
the monitoring execution module is used for respectively monitoring the service database log information of the service databases corresponding to the service subsystems, wherein the service subsystems are systems for independently realizing services, and the generated service data are respectively stored in the service databases corresponding to the service subsystems;
and the message pushing module is used as a message producer to respectively package the log information of each monitored service database into service messages and push the service messages to the corresponding message queues.
8. The apparatus of claim 7, wherein the snooping execution module snoops the traffic database log information according to a binlog master-slave synchronization protocol with the traffic database corresponding to each of the traffic subsystems as a master side and the snooper itself as a slave side.
9. A computer readable storage medium, having stored thereon computer instructions, wherein the instructions, when executed by a processor, can implement the steps of the method for implementing business data association according to any one of claims 1 to 5.
10. An electronic device for implementing service data association, the electronic device comprising:
a processor;
a memory for storing the processor-executable instructions;
the processor is used for reading the executable instructions from the memory and executing the instructions to realize the steps in the method for realizing the business data association as claimed in any one of the claims 1 to 5.
CN202010306366.7A 2020-04-17 2020-04-17 Method, device and storage medium for realizing business data association Active CN111625535B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010306366.7A CN111625535B (en) 2020-04-17 2020-04-17 Method, device and storage medium for realizing business data association

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010306366.7A CN111625535B (en) 2020-04-17 2020-04-17 Method, device and storage medium for realizing business data association

Publications (2)

Publication Number Publication Date
CN111625535A true CN111625535A (en) 2020-09-04
CN111625535B CN111625535B (en) 2021-07-30

Family

ID=72260950

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010306366.7A Active CN111625535B (en) 2020-04-17 2020-04-17 Method, device and storage medium for realizing business data association

Country Status (1)

Country Link
CN (1) CN111625535B (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112311883A (en) * 2020-10-30 2021-02-02 杭州安恒信息安全技术有限公司 Data synchronization method and system
CN112395157A (en) * 2020-11-13 2021-02-23 广州至真信息科技有限公司 Audit log obtaining method and device, computer equipment and storage medium
CN112988422A (en) * 2021-03-18 2021-06-18 北京京东乾石科技有限公司 Asynchronous message processing method and device, electronic equipment and storage medium
CN116501376A (en) * 2023-06-26 2023-07-28 和创（北京）科技股份有限公司 Decoupling method, system and device based on configuration instant trigger task
CN112988422B (en) * 2021-03-18 2024-05-17 北京京东乾石科技有限公司 Asynchronous message processing method and device, electronic equipment and storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080046230A1 (en) * 2006-07-14 2008-02-21 Fujitsu Limited Reception support system and program therefor
CN106681882A (en) * 2015-11-06 2017-05-17 上海瑞致软件有限公司 IT-service concentrated monitoring and managing system based on Apriori algorithm
CN108076098A (en) * 2016-11-16 2018-05-25 北京京东尚科信息技术有限公司 A kind of method for processing business and system
CN108765141A (en) * 2018-06-20 2018-11-06 中国银行股份有限公司 A kind of data tracking management method and device
CN108804237A (en) * 2017-05-05 2018-11-13 北京京东尚科信息技术有限公司 Data real-time statistical method, device, storage medium and electronic equipment
CN109344199A (en) * 2018-09-20 2019-02-15 普元信息技术股份有限公司 The system and method for big data quantity distributed treatment is realized in cloud computing platform

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109639755B (en) * 2018-10-23 2022-04-12 平安科技（深圳）有限公司 Associated system server decoupling method, device, medium and electronic equipment

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080046230A1 (en) * 2006-07-14 2008-02-21 Fujitsu Limited Reception support system and program therefor
CN106681882A (en) * 2015-11-06 2017-05-17 上海瑞致软件有限公司 IT-service concentrated monitoring and managing system based on Apriori algorithm
CN108076098A (en) * 2016-11-16 2018-05-25 北京京东尚科信息技术有限公司 A kind of method for processing business and system
CN108804237A (en) * 2017-05-05 2018-11-13 北京京东尚科信息技术有限公司 Data real-time statistical method, device, storage medium and electronic equipment
CN108765141A (en) * 2018-06-20 2018-11-06 中国银行股份有限公司 A kind of data tracking management method and device
CN109344199A (en) * 2018-09-20 2019-02-15 普元信息技术股份有限公司 The system and method for big data quantity distributed treatment is realized in cloud computing platform

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112311883A (en) * 2020-10-30 2021-02-02 杭州安恒信息安全技术有限公司 Data synchronization method and system
CN112311883B (en) * 2020-10-30 2023-04-07 杭州安恒信息安全技术有限公司 Data synchronization method and system
CN112395157A (en) * 2020-11-13 2021-02-23 广州至真信息科技有限公司 Audit log obtaining method and device, computer equipment and storage medium
CN112395157B (en) * 2020-11-13 2023-08-08 广州至真信息科技有限公司 Audit log acquisition method and device, computer equipment and storage medium
CN112988422A (en) * 2021-03-18 2021-06-18 北京京东乾石科技有限公司 Asynchronous message processing method and device, electronic equipment and storage medium
CN112988422B (en) * 2021-03-18 2024-05-17 北京京东乾石科技有限公司 Asynchronous message processing method and device, electronic equipment and storage medium
CN116501376A (en) * 2023-06-26 2023-07-28 和创（北京）科技股份有限公司 Decoupling method, system and device based on configuration instant trigger task
CN116501376B (en) * 2023-06-26 2023-09-12 和创（北京）科技股份有限公司 Decoupling method, system and device based on configuration instant trigger task

Also Published As

Publication number Publication date
CN111625535B (en) 2021-07-30

Similar Documents

Publication Publication Date Title
CN111625535B (en) Method, device and storage medium for realizing business data association
CN108536761B (en) Report data query method and server
CN111752799A (en) Service link tracking method, device, equipment and storage medium
CN108076098A (en) A kind of method for processing business and system
US20080065400A1 (en) System and Method for Producing Audit Trails
CN111339073A (en) Real-time data processing method and device, electronic equipment and readable storage medium
JP7254975B2 (en) Key-based logging for processing structured data items with executable logic
WO2012155643A1 (en) Method for implementing configuration console based on database, configuration console, and system
CN109840159A (en) Abnormality processing project management method, apparatus, computer installation and storage medium
CN111339118A (en) Kubernetes-based resource change history recording method and device
CN111355802A (en) Information pushing method and device
CN115168400A (en) External data management system and method
CN112363902A (en) Business process monitoring method and device, computer equipment and storage medium
CN111324480B (en) Large-scale host transaction fault positioning system and method
CN113094166A (en) Link tracking method, device, medium and computing equipment
CN108984156A (en) A kind of software auto generating method and system for exempting from code development based on template
KR101888131B1 (en) Method for Performing Real-Time Changed Data Publish Service of DDS-DBMS Integration Tool
CN110740153A (en) monitoring data acquisition method, system and device
CN111274255B (en) Service data monitoring method and system, monitoring architecture, equipment and storage medium
CN116136801A (en) Cloud platform data processing method and device, electronic equipment and storage medium
CN115599781A (en) Calling logic display method, service system and electronic equipment
CN113297516A (en) Customer interaction interface generation method and device and electronic equipment
CN117033522A (en) Data processing method and device based on data lake, electronic equipment and medium
CN117055977A (en) Method and device for linking data between code-free applications
CN116415914A (en) Service data processing method, device, computer equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20201027

Address after: 100085 Floor 102-1, Building No. 35, West Second Banner Road, Haidian District, Beijing

Applicant after: Seashell Housing (Beijing) Technology Co.,Ltd.

Address before: 300 280 Tianjin Economic and Technological Development Zone Nangang Industrial Zone Office Building C Block 1, Room 112, Unit 05

Applicant before: BEIKE TECHNOLOGY Co.,Ltd.

GR01 Patent grant
GR01 Patent grant