CN115455059A - Method, device and related medium for analyzing user behavior based on underlying data - Google Patents
Method, device and related medium for analyzing user behavior based on underlying data Download PDFInfo
- Publication number
- CN115455059A CN115455059A CN202211230953.8A CN202211230953A CN115455059A CN 115455059 A CN115455059 A CN 115455059A CN 202211230953 A CN202211230953 A CN 202211230953A CN 115455059 A CN115455059 A CN 115455059A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- log
- time sequence
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Abstract
The invention discloses a method, a device and a related medium for analyzing user behaviors based on bottom data, wherein the method comprises the following steps: generating a data record during transaction execution, marking the data record, and acquiring a global transaction ID in a mysql database; storing the data record and global transaction ID into a temporal database; monitoring a log file in a mysql database, generating corresponding log events and object information for the log file through a data flow service, and then storing the log events and the object information into a time sequence database; the stored data in the time sequence database is read regularly through a data analysis service, and operation log data are obtained through analysis from the stored data according to a rule engine; and storing the operation log data into a time sequence database, and inquiring and displaying the operation log data in the time sequence database. The invention realizes the analysis of the user behavior and reduces the service code coupling, thereby ensuring the performance of the service system.
Description
Technical Field
The invention relates to the technical field of computer software, in particular to a method and a device for analyzing user behaviors based on bottom data and a related medium.
Background
In the prior art, there are the following two general methods for collecting behavior operation logs of a real user operation service system:
1. adding a large number of interceptors in a routing layer, intercepting preRequest and postRequest of most interfaces, and then analyzing into a user operation log;
2. and adding an interceptor in the model layer to analyze the operations of adding and deleting.
However, both of these approaches are more intrusive to the original service code and result in overall performance degradation.
Disclosure of Invention
The embodiment of the invention provides a method, a device, computer equipment and a storage medium for analyzing user behaviors based on underlying data, and aims to reduce service code coupling while analyzing the user behaviors and ensure performance of a service system.
In a first aspect, an embodiment of the present invention provides a method for analyzing a user behavior based on underlying data, including:
responding to the operation of a user on a business system, generating a data record when the transaction of the mysql database is executed, and marking the data record;
acquiring a global transaction ID in the mysql database after the transaction is executed;
storing the data record and the global transaction ID into a time sequence database;
monitoring a log file in a mysql database, generating corresponding log events and object information for the log file through a data flow service, and then storing the log events and the object information into the time sequence database;
the stored data in the time sequence database is read regularly through a data analysis service, and operation log data of a user on a service system are obtained through analysis from the stored data according to a rule engine; wherein the stored data comprises the data record, a global transaction ID, a log event, and object information;
storing the operation log data into the time sequence database;
and responding to the operation of the user on the service system, and inquiring and displaying the operation log data in the time sequence database.
In a second aspect, an embodiment of the present invention provides an apparatus for analyzing a user behavior based on underlying data, where the apparatus includes:
the response unit is used for responding to the operation of a user on the business system, generating a data record when the transaction of the mysql database is executed, and marking the data record;
the transaction acquisition unit is used for acquiring a global transaction ID in the mysql database after the transaction is executed;
a first storage unit, configured to store the data record and the global transaction ID in a time series database;
the monitoring unit is used for monitoring the log file in the mysql database, generating corresponding log event and object information for the log file through a data flow service, and then storing the log event and the object information into the time sequence database;
the analysis unit is used for reading the stored data in the time sequence database at regular time through a data analysis service and analyzing the stored data according to a rule engine to obtain operation log data of a user on a service system; wherein the stored data comprises the data record, a global transaction ID, a log event, and object information;
the second storage unit is used for storing the operation log data into a time sequence database;
and the query unit is used for querying and displaying the operation log data in the time sequence database when responding to the operation of the user on the service system.
In a third aspect, an embodiment of the present invention provides a computer device, including a memory and a processor, where the memory stores a computer program operable on the processor, and the processor, when executing the computer program, implements the method for analyzing user behavior based on underlying data according to the first aspect.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the method for analyzing user behavior based on underlying data according to the first aspect.
The embodiment of the invention provides a method, a device, computer equipment and a storage medium for analyzing user behaviors based on bottom data, wherein the method comprises the following steps: responding to the operation of a user on a business system, generating a data record when the transaction of the mysql database is executed, and marking the data record; acquiring a global transaction ID in the mysql database after the transaction is executed; storing the data record and the global transaction ID into a time sequence database; monitoring a log file in a mysql database, generating corresponding log events and object information for the log file through a data flow service, and then storing the log events and the object information into the time sequence database; the stored data in the time sequence database is read regularly through a data analysis service, and operation log data of a user on a service system are obtained through analysis from the stored data according to a rule engine; wherein the stored data comprises the data record, a global transaction ID, a log event, and object information; storing the operation log data into the time sequence database; and responding to the operation of the user on the service system, and inquiring and displaying the operation log data in the time sequence database. According to the embodiment of the invention, the operations of analyzing, storing and the like are carried out on the bottom data of the mysql database, so that the analysis of the user behavior is realized, the service code coupling is reduced, and the performance of the service system is ensured.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.
Fig. 1 is a schematic flowchart of a method for analyzing user behavior based on underlying data according to an embodiment of the present invention;
fig. 2 is a network flow chart of a method for analyzing user behavior based on underlying data according to an embodiment of the present invention;
fig. 3 is a schematic block diagram of an apparatus for analyzing user behavior based on underlying data according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items and includes such combinations.
Referring to fig. 1, fig. 1 is a schematic flow chart of a method for analyzing a user behavior based on underlying data according to an embodiment of the present invention, where the method is applied to a server or a service terminal, and specifically includes: steps S101 to S105.
S101, responding to the operation of a user on a service system, generating a data record when the transaction of the mysql database is executed, and marking the data record;
s102, acquiring a global transaction ID in the mysql database after the transaction is executed;
s103, storing the data records and the global transaction ID into a time sequence database;
s104, monitoring a log file in a mysql database, generating corresponding log events and object information for the log file through a data flow service, and then storing the log events and the object information into the time sequence database;
s105, reading the stored data in the time sequence database at regular time through data analysis service, and analyzing the stored data according to a rule engine to obtain operation log data of a user on a service system; wherein the stored data comprises the data record, a global transaction ID, a log event, and object information;
s106, storing the operation log data into the time sequence database;
s107, responding to the operation of the user on the service system, and inquiring and displaying the operation log data in the time sequence database.
In this embodiment, with reference to fig. 2, when the service system is operated, the mysql database generates a data record, records the data record, obtains the global transaction ID after the transaction of the mysql database is executed, and then writes the data record and the global transaction ID into the time sequence database. And then, monitoring and circulating the log file in the mysql database through a data stream transfer service to generate corresponding log events and object information, and storing the log events and the object information into a time sequence database. And then, reading the data stored in the time sequence database at regular time through the data analysis service, and analyzing the data according to the rule engine to obtain the operation log data of the user. When the subsequent user operates the service system, the service system can inquire the operation log data corresponding to the user from the time sequence database and display the operation log data to the time sequence database.
According to the embodiment, the operations such as analysis, storage and the like are performed on the bottom-layer data of the mysql database, so that the business code coupling is reduced while the user behavior is analyzed, and the performance of the business system is ensured.
In an embodiment, when a transaction of the mysql database is executed, a data record is generated, and the data record is marked, a transaction identifier may be generated by the mysql database, and when the transaction is executed, a mapping relationship with the data record is established by using the transaction identifier, so as to mark the data record.
In an embodiment, the step S102 includes:
after the transaction is executed, the global transaction ID is obtained from an ok package in the mysql database by using a reflection technology.
In this embodiment, in the service system, when a user operates the service system, data records are generated from the mysql database most of the time, and the data records are specifically generated through transactions of the mysql database. When the transaction is executed, marking the transaction, wherein the marking content is as follows: organization/team, user, tagged data records can accurately identify in subsequent data parsing services which operations were generated by which user of which team. The GTID (Global Transaction ID) in the mysql database is obtained after the Transaction execution. Since there is no way to directly acquire GTID in mysql driver in go language, the code of mysql driver is adjusted and GTID is acquired from ok package (ok package) in mysql database by using reflection technology. The GTID is obtained through reflection in the transaction, so that the understanding of the developer to the bottom layer can be reduced, and only the service layer needs to be concerned.
After the transaction is performed, the data (i.e., data record and global transaction ID) will be written into the mysql database and the operator information (including organization/team, user, time, etc.) and GTID of the transaction will be recorded into the pgsql database (i.e., the timing database). Pgsql (PostgreSQL) is an object relational database management system (ordms), and is mainly used as a time series database in the present embodiment. It should be further noted that, in the process of interacting with the mysql database, the mysql database returns an ok packet when a certain command is completed.
In a specific embodiment, after the transaction is executed, the step of obtaining the global transaction ID from an ok packet in the mysql database by using a reflection technique includes:
establishing a mapping relation of global transaction IDs among different systems in the mysql database based on an elastic configuration mode;
adopting a formatting tool written by a getter method and a setter method, and utilizing the formatting tool to format the global transaction ID in the mysql database so as to enable the global transaction ID to meet the requirements of the getter method and the setter method calling;
automatically identifying and automatically converting the global transaction ID;
and according to the mapping relation, extracting the global transaction ID from the mysql database by adopting a reflection technology.
In one embodiment, the step S104 includes:
simulating a master-slave synchronization mode of the mysql database by using canal technology to transfer binlog data streams in the mysql database to data stream service; wherein the binlog data is the log file.
Further, a master-slave synchronization mode of the mysql database is simulated through the canal technology, so that binlog data streams in the mysql database are transferred to a data stream service; wherein, the binlog data is the log file, including:
and transferring the binlog data stream into a data stream service in a stream mode.
In this embodiment, in the data transfer service binlog-event-sync, the binlog data in the mysql database is monitored by the canal technology, and the monitoring process actually simulates a master-slave synchronization mode of the mysql database, so that the binlog data in the mysql database is transferred to the target in a binlog _ format = ROW mode.
The service developed by the data streaming service binlog-event-sync based on the canal technology in the embodiment is used for streaming the binlog data of the mysql database to the pgsql database, and besides common streaming data, the service additionally provides preprocessing data generated according to the configuration file.
In addition, the canal is a technology for transferring binlog data streams in the mysql database to a target in a streaming mode, and the main purpose is to provide incremental data subscription and consumption based on mysql database incremental log analysis. Binlog data is a log file in mysql database binary format. Binlog is used for recording the changes to the database (only the modification operation to the data) in mysql, and is mainly used for master-slave copy and increment recovery of the database.
In an embodiment, the simulating the mysql database master-slave synchronization by the canal technology to transfer the binlog data stream in the mysql database to the data stream transfer service further includes:
acquiring binlog data in a mysql database in real time through a canal cluster;
constructing a target table by using a table association strategy pre-stored in the canal cluster;
and writing the target table into a data flow service.
Wherein, the table association policy specifically comprises: if two data tables in the mysql database are modified in the same transaction, newly added binlog data in the two data tables are correlated in the canal cluster, and a target table is formed and written into the data flow service; if one newly added binlog data in the data table exists in the mysql database, the fields related to the binlog data are obtained through the data circulation service, and a target table is formed to be written into the data circulation service. In addition, the canal cluster adopts a key-value mode, takes the data of the associated fields as keys, and matches with the corresponding value values.
In an embodiment, the step S104 further includes:
carrying out pre-analysis processing on the binlog data through a configuration file in a data stream transfer service to generate a corresponding log event;
and judging the data type of the binlog data, determining whether to directly serialize the binlog data into a data structure of the time sequence database according to a judgment result, and synchronizing the data structure to the time sequence database.
In this embodiment, after the binlog-event-sync service receives the binlog data, two flows, namely dump and stream, may be used to process the received data, so as to ensure that the transaction operation source data is not lost. The specific process is as follows:
dump: when the binlog-event-sync service is just started and actually receives dump mysql full-volume data, the data is preprocessed according to a configuration file defined in the data flow service at the moment, and data of corresponding < uuid: name > and data of some attributes < alias: name > are generated, and the information is used for associating some corresponding names when log data are analyzed.
For example, obj _ mapping and field _ setting are defined in the service, and the format is similar:
the corresponding < uuid: name > and < alias: name > data can be generated according to the above format.
stream: the type of binlog data is judged. When the type of the binlog data is judged not to be GTID and XID, the data can be regarded as mysql data generated by a user using a service system at the moment, and at the moment, the binlog data can be directly serialized into a well-defined data structure in a pgsql database and synchronized into the pgsql database, so that the data cannot be lost.
In one embodiment, the step S105 includes:
compiling a log according to the characteristics of the underlying data structure to generate a rule engine method set;
and dividing rules for the stored data by using the rule engine method set, and generating corresponding operation log data.
Further, the writing of the log according to the characteristics of the underlying data structure to generate a rule engine method set includes:
setting an Action interface to record or access the global transaction ID;
setting a DbHint interface and a TablesHint interface to monitor the mysql database and/or the time sequence database;
setting a Verify interface to Verify each data in the analysis process;
and setting a BuildLogs interface to generate storage data with a set structure according to verification data provided by the Verify interface, and storing the storage data into a time sequence database for a business system to inquire.
In this embodiment, the data parsing service may read binlog data, pre-parsed data (e.g., the log event and the object information) and user operation information in the pgsql database at regular time, and then parse the operation log data of the real user when operating the service system through the rule engine, so that a new service rule may be quickly added without affecting the generation of the original log.
The rule engine described in this embodiment writes a log generation method set according to the characteristics of the underlying data structure, and after a data stream enters, which rule is entered is determined by a database, a data table, and characteristic verification, and after entering the rule, log data is generated in a uniform manner within the rule.
Writing a rule requires implementing the following interfaces:
wherein, action (): an ID representing an operation, such as a global transaction ID;
DbHint (): and the database which needs to be monitored is shown, so that the database is checked in advance and the analysis speed is accelerated. For example: a certain operation of the user a is operated in the service system B, where the database B of the service system needs to be filled in.
Tablenext (): data tables which show that monitoring needs to be concerned, such as mysql database and time sequence database, are checked in advance to accelerate the resolution speed. For example, user a invites user B to join the team, and this action can add a piece of data to the go-to-user table, so the user table needs to be filled in here.
Verify (): showing a specific verification rule method. If relying only on the previous DbHint, tableshift is not sufficient to achieve the corresponding result, since different operations may operate on the same table, more specific fields are needed to determine. In addition, this step also generates some necessary data, such as user UUID.
Buildlgs (): and the representation directly generates data with a set structure according to the data provided by Verify, and stores the data into a pgsql database for being inquired by a business system and showing the data to a user.
Fig. 3 is a schematic block diagram of an apparatus 300 for analyzing user behavior based on underlying data according to an embodiment of the present invention, where the apparatus 300 includes:
a response unit 301, configured to, in response to an operation of a user on a business system, generate a data record when a transaction of the mysql database is executed, and mark the data record;
a transaction obtaining unit 302, configured to obtain a global transaction ID in the mysql database after the transaction is executed; a first storage unit 303, configured to store the data record and the global transaction ID in a time sequence database;
the monitoring unit 304 is configured to monitor a log file in the mysql database, generate a corresponding log event and object information for the log file through a data transfer service, and store the log event and the object information in the time sequence database;
the analysis unit 305 is used for regularly reading the stored data in the time sequence database through a data analysis service, and analyzing the stored data according to a rule engine to obtain operation log data of a user on a business system; wherein the stored data comprises the data record, a global transaction ID, a log event, and object information;
a second storage unit 306, configured to store the operation log data in a time-series database;
the query unit 307 is configured to query and display the operation log data in the time sequence database when responding to the operation of the user on the service system.
In this embodiment, with reference to fig. 2, when the service system is operated, the mysql database generates a data record, records the data record, obtains the global transaction ID after the transaction of the mysql database is executed, and then writes the data record and the global transaction ID into the time sequence database. And then, monitoring and circulating the log file in the mysql database through a data stream transfer service to generate corresponding log events and object information, and storing the log events and the object information into a time sequence database. And then, reading the data stored in the time sequence database at regular time through the data analysis service, and analyzing the data according to the rule engine to obtain the operation log data of the user. When the subsequent user operates the service system, the service system can inquire the operation log data corresponding to the user from the time sequence database and display the operation log data to the time sequence database.
According to the embodiment, the operations such as analysis and storage of the bottom data of the mysql database are performed, so that the user behavior is analyzed, meanwhile, the service code coupling is reduced, and the performance of the service system is ensured.
In an embodiment, when a transaction of the mysql database is executed, a data record is generated, and the data record is marked, a transaction identifier may be generated by the mysql database, and when the transaction is executed, a mapping relationship with the data record is established by using the transaction identifier, so as to mark the data record.
In an embodiment, the transaction obtaining unit 302 includes:
and the reflection acquisition unit is used for acquiring the global transaction ID from an ok packet in the mysql database by using a reflection technology after the transaction is executed.
In this embodiment, in the service system, when a user operates the service system, data records are generated from the mysql database most of the time, and the data records are specifically generated through transactions of the mysql database. When the transaction is executed, marking the transaction, wherein the marking content is as follows: organization/team, user, tagged data records can accurately identify in subsequent data parsing services which operations were generated by which user of which team. The GTID (Global Transaction ID) in the mysql database is obtained after the Transaction execution. Since there is no way to directly obtain GTID in mysql driver of go language, the code of mysql driver is adjusted, and GTID is obtained from ok packet of mysql database by using reflection technology. The GTID is obtained through reflection in the transaction, so that the understanding of the developer to the bottom layer can be reduced, and only the service layer needs to be concerned.
After the transaction is performed, the data (i.e., data record and global transaction ID) will be written into the mysql database and the operator information (including organization/team, user, time, etc.) and GTID of the transaction will be recorded into the pgsql database (i.e., the timing database). Pgsql (PostgreSQL) is an object relational database management system (ordms), and is mainly used as a time series database in the present embodiment.
In a specific embodiment, after the transaction is executed, the step of obtaining the global transaction ID from an ok package in the mysql database by using a reflection technology includes:
establishing a mapping relation of global transaction IDs among different systems in the mysql database based on an elastic configuration mode;
adopting a formatting tool written by a getter method and a setter method, and utilizing the formatting tool to format the global transaction ID in the mysql database so as to enable the global transaction ID to meet the requirements of the getter method and the setter method calling;
automatically identifying and automatically converting the global transaction ID;
and according to the mapping relation, extracting the global transaction ID from the mysql database by adopting a reflection technology.
In one embodiment, the listening unit 304 comprises:
the first data flow unit is used for simulating a master-slave synchronization mode of the mysql database through the canal technology so as to transfer the binlog data flow in the mysql database to the data flow service; wherein the binlog data is the log file.
In one embodiment, the first data stream unit includes:
and the second data flow unit is used for transferring the binlog data flow to a data flow service in a flow mode.
In this embodiment, in the binlog-event-sync of the data transfer service, the binlog data in the mysql database is monitored by the canal technology, and the monitoring process actually simulates a master-slave synchronization mode of the mysql database, so that the binlog data in the mysql database is transferred to the target in a mode of binlog _ format = ROW.
The service developed by the data streaming service binlog-event-sync based on the canal technology in the embodiment is used for streaming the binlog data of the mysql database to the pgsql database, and besides common streaming data, the service additionally provides preprocessing data generated according to the configuration file.
In addition, the canal is a technology for transferring binlog data streams in the mysql database to a target in a streaming mode, and the main purpose is to provide incremental data subscription and consumption based on mysql database incremental log analysis. Binlog data is a log file in binary format for mysql database. Binlog is used for recording the change of mysql to the database (only the modification operation to the data is recorded), and is mainly used for master-slave copy and increment recovery of the database.
In an embodiment, the simulating the mysql database master-slave synchronization by the canal technology to transfer the binlog data stream in the mysql database to the data stream transfer service further includes:
acquiring binlog data in a mysql database in real time through a canal cluster;
constructing a target table by using a table association strategy pre-stored in the canal cluster;
and writing the target table into a data flow service.
Wherein, the table association policy specifically comprises: if two data tables in the mysql database are modified in the same transaction, the binlog data newly added in the two data tables are correlated in the canal cluster, and a target table is formed and written into the data transfer service; if the mysql database has newly added binlog data in a data table, acquiring fields related to the binlog data through the data circulation service, and forming a target table write data circulation service. In addition, the canal cluster adopts a key-value mode, takes the data of the associated fields as keys, and matches with the corresponding value values.
In an embodiment, the listening unit 304 further comprises:
the analysis processing unit is used for carrying out pre-analysis processing on the binlog data through a configuration file in data stream transfer service so as to generate a corresponding log event;
and the type judging unit is used for judging the data type of the binlog data, confirming whether the binlog data is directly serialized into the data structure of the time sequence database according to the judging result, and synchronizing the data structure to the time sequence database.
In this embodiment, after the binlog-event-sync service receives the binlog data, there are two flows, namely dump and stream, to process the received data, so as to ensure that the transaction operation source data is not lost. The specific flow is as follows:
dump: when the binlog-event-sync service is just started and actually receives dump mysql full-volume data, the data is preprocessed according to a configuration file defined in the data flow service at the moment, and data of corresponding < uuid: name > and data of some attributes < alias: name > are generated, and the information is used for associating some corresponding names when log data are analyzed.
For example, obj _ mapping and field _ setting are defined in the service, and the format is similar:
corresponding < uuid: name > and < alias: name > data can be generated according to the format.
stream: the type of binlog data is judged. When the type of the binlog data is judged not to be GTID and XID, the data can be regarded as mysql data generated by a user using a service system at the moment, and at the moment, the binlog data can be directly serialized into a well-defined data structure in a pgsql database and synchronized into the pgsql database, so that the data cannot be lost.
In an embodiment, the parsing unit 305 includes:
the log writing unit is used for writing logs according to the characteristics of the bottom data structure so as to generate a rule engine method set;
and the rule dividing unit is used for dividing the storage data into rules by using the rule engine method set and generating corresponding operation log data.
In an embodiment, the log writing unit includes:
the first interface setting unit is used for setting an Action interface so as to record or access the global transaction ID;
the second interface setting unit is used for setting a DbHint interface and a TableHint interface so as to monitor the mysql database and/or the time sequence database;
the third interface setting unit is used for setting a Verify interface so as to Verify each data in the analysis process;
and the fourth interface setting unit is used for setting the BuildLogs interface so as to generate storage data with a set structure according to the verification data provided by the Verify interface, and the storage data is stored in the time sequence database for the query of the business system.
In this embodiment, the data parsing service may read binlog data, pre-parsed data (e.g., the log event and the object information) and user operation information in the pgsql database at regular time, and then parse the operation log data of the real user when operating the service system through the rule engine, so that a new service rule may be quickly added without affecting the generation of the original log.
The rule engine described in this embodiment writes a log generation method set according to the characteristics of the underlying data structure, and after a data stream comes in, it determines which rule to enter through a database, a data table, and characteristic verification, and after entering the rule, generates log data in a uniform manner within the rule.
Writing a rule requires implementing the following interfaces:
wherein, action (): an ID representing an operation, such as a global transaction ID;
DbHint (): the monitored database needs to be concerned, the monitoring is carried out in advance, and the analysis speed is accelerated. For example, the following steps: a certain operation of the user a is operated in the service system B, where the database B of the service system needs to be filled in.
Tablenext (): data tables which show that monitoring needs to be concerned, such as mysql database and time sequence database, are checked in advance to accelerate the resolution speed. For example, user a invites user B to join the team, and this action can add a piece of data to the go-to-user table, so the user table needs to be filled in here.
Verify (): showing a specific check rule method. If relying only on the previous DbHint, tableshift is not sufficient to achieve the corresponding result, since different operations may operate on the same table, more specific fields are needed to determine. In addition, this step also generates some necessary data, such as user UUID.
Buildlgs (): and the representation directly generates data with a set structure according to the data provided by Verify, and stores the data into a pgsql database for being inquired by a business system and showing the data to a user.
Since the embodiment of the apparatus portion and the embodiment of the method portion correspond to each other, please refer to the description of the embodiment of the method portion for the embodiment of the apparatus portion, and details are not repeated here.
Embodiments of the present invention also provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the steps provided by the above embodiments can be implemented. The storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The method for analyzing the user behavior based on the bottom data provided by the embodiment of the invention comprises the following steps:
responding to the operation of a user on a business system, generating a data record when the transaction of the mysql database is executed, and marking the data record;
acquiring a global transaction ID in a mysql database after the transaction is executed;
storing the data record and the global transaction ID into a temporal database;
monitoring a log file in a mysql database, generating corresponding log events and object information for the log file through a data flow service, and then storing the log events and the object information into the time sequence database;
the stored data in the time sequence database is read regularly through a data analysis service, and operation log data of a user on a service system are obtained through analysis from the stored data according to a rule engine; wherein the stored data comprises the data record, a global transaction ID, a log event, and object information;
storing the operation log data into the time sequence database;
and responding to the operation of the user on the service system, and inquiring and displaying the operation log data in the time sequence database.
The embodiment of the present invention further provides a computer device, which may include a memory and a processor, where the memory stores a computer program, and the processor may implement the steps provided in the above embodiments when calling the computer program in the memory. Of course, the computer device may also include various network interfaces, power supplies, and the like.
The method for analyzing the user behavior based on the bottom data provided by the embodiment of the invention comprises the following steps:
responding to the operation of a user on a business system, generating a data record when the transaction of the mysql database is executed, and marking the data record;
acquiring a global transaction ID in the mysql database after the transaction is executed;
storing the data record and the global transaction ID into a time sequence database;
monitoring a log file in a mysql database, generating corresponding log events and object information for the log file through a data flow service, and then storing the log events and the object information into the time sequence database;
reading the stored data in the time sequence database at regular time through a data analysis service, and analyzing the stored data according to a rule engine to obtain operation log data of a user on a service system; wherein the storage data comprises the data record, a global transaction ID, a log event, and object information;
storing the operation log data into the time sequence database;
and responding to the operation of the user on the service system, and inquiring and displaying the operation log data in the time sequence database.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. For the system disclosed by the embodiment, the description is relatively simple because the system corresponds to the method disclosed by the embodiment, and the relevant points can be referred to the description of the method part. It should be noted that, for those skilled in the art, without departing from the principle of the present application, the present application can also make several improvements and modifications, and those improvements and modifications also fall into the protection scope of the claims of the present application.
It is further noted that, in the present specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising one of 8230; \8230;" 8230; "does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.
Claims (10)
1. A method for analyzing user behavior based on underlying data, comprising:
responding to the operation of a user on a business system, generating a data record when the transaction of the mysql database is executed, and marking the data record;
acquiring a global transaction ID in the mysql database after the transaction is executed;
storing the data record and the global transaction ID into a temporal database;
monitoring a log file in a mysql database, generating corresponding log events and object information for the log file through a data flow service, and then storing the log events and the object information into the time sequence database;
the stored data in the time sequence database is read regularly through a data analysis service, and operation log data of a user on a service system are obtained through analysis from the stored data according to a rule engine; wherein the stored data comprises the data record, a global transaction ID, a log event, and object information;
storing the operation log data into the time sequence database;
and responding to the operation of the user on the service system, and inquiring and displaying the operation log data in the time sequence database.
2. The method of claim 1, wherein obtaining a global transaction ID in a database after a transaction is performed comprises:
after the transaction is executed, the global transaction ID is obtained from an ok packet in the mysql database using a reflection technique.
3. The method for analyzing user behavior based on underlying data of claim 1, wherein the monitoring a log file in a mysql database, generating corresponding log events and object information for the log file through a data flow service, and then storing the log events and object information in the time sequence database comprises:
simulating a master-slave synchronization mode of the mysql database by using canal technology to transfer binlog data streams in the mysql database to data stream service; wherein the binlog data is the log file.
4. The method for analyzing user behavior based on underlying data as claimed in claim 3, wherein the mysql database is simulated by canal technique in a master-slave synchronization manner to transfer binlog data stream in mysql database to data stream service; wherein the binlog data is the log file, including:
and transferring the binlog data stream into a data stream service in a stream mode.
5. The method of claim 3, wherein the monitoring a log file in a mysql database, generating a corresponding log event and object information for the log file through a data flow service, and then storing the log event and object information in the time sequence database, further comprises:
carrying out pre-analysis processing on the binlog data through a configuration file in a data stream transfer service to generate a corresponding log event;
and judging the data type of the binlog data, determining whether to directly serialize the binlog data into a data structure of the time sequence database according to a judgment result, and synchronizing the data structure to the time sequence database.
6. The method for analyzing user behavior based on the underlying data as claimed in claim 1, wherein the reading of the stored data in the time sequence database by the data analysis service at regular time and the analysis of the operation log data about the user to the business system from the stored data according to the rule engine comprise:
writing a log according to the characteristics of a bottom data structure to generate a rule engine method set;
and dividing rules for the stored data by using the rule engine method set, and generating corresponding operation log data.
7. The method for parsing user behavior based on underlying data as recited in claim 6, wherein said writing a log according to characteristics of an underlying data structure to generate a set of rule engine methods comprises:
setting an Action interface to record or access the global transaction ID;
setting a DbHint interface and a TableHint interface to monitor the mysql database and/or the time sequence database;
setting a Verify interface to Verify each data in the analysis process;
and setting a BuildLogs interface to generate storage data with a set structure according to verification data provided by the Verify interface, and storing the storage data into a time sequence database for a service system to inquire.
8. An apparatus for resolving user behavior based on underlying data, comprising:
the response unit is used for responding to the operation of a user on the business system, generating a data record when the transaction of the mysql database is executed, and marking the data record;
the transaction acquisition unit is used for acquiring a global transaction ID in the mysql database after the transaction is executed;
a first storage unit, configured to store the data record and the global transaction ID in a time series database;
the monitoring unit is used for monitoring the log file in the mysql database, generating corresponding log event and object information for the log file through a data flow service, and then storing the log event and the object information into the time sequence database;
the analysis unit is used for regularly reading the stored data in the time sequence database through a data analysis service and analyzing the stored data according to a rule engine to obtain operation log data of a user on a service system; wherein the stored data comprises the data record, a global transaction ID, a log event, and object information;
the second storage unit is used for storing the operation log data into a time sequence database;
and the query unit is used for querying and displaying the operation log data in the time sequence database when responding to the operation of the user on the service system.
9. A computer device comprising a memory, a processor, a computer program stored on the memory and executable on the processor, the processor implementing a method of resolving user behavior based on underlying data as claimed in any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the method of resolving user behavior based on underlying data according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211230953.8A CN115455059A (en) | 2022-09-30 | 2022-09-30 | Method, device and related medium for analyzing user behavior based on underlying data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211230953.8A CN115455059A (en) | 2022-09-30 | 2022-09-30 | Method, device and related medium for analyzing user behavior based on underlying data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115455059A true CN115455059A (en) | 2022-12-09 |
Family
ID=84309027
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211230953.8A Pending CN115455059A (en) | 2022-09-30 | 2022-09-30 | Method, device and related medium for analyzing user behavior based on underlying data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115455059A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117033470A (en) * | 2023-10-08 | 2023-11-10 | 天津市天河计算机技术有限公司 | Data generation method, device, equipment and medium |
-
2022
- 2022-09-30 CN CN202211230953.8A patent/CN115455059A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117033470A (en) * | 2023-10-08 | 2023-11-10 | 天津市天河计算机技术有限公司 | Data generation method, device, equipment and medium |
| CN117033470B (en) * | 2023-10-08 | 2024-01-30 | 天津市天河计算机技术有限公司 | Data generation method, device, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021088724A1 (en) | Testing method and apparatus | |
| US7979850B2 (en) | Method and system for generating a common trace data format | |
| CN108647357B (en) | Data query method and device | |
| CN109710439B (en) | Fault processing method and device | |
| CN111400288A (en) | Data quality inspection method and system | |
| US20130263156A1 (en) | Operation log collection method and device | |
| CN113704790A (en) | Abnormal log information summarizing method and computer equipment | |
| CN116204438A (en) | Test case generation method, automatic test method and related device | |
| CN115455059A (en) | Method, device and related medium for analyzing user behavior based on underlying data | |
| CN109101412B (en) | Test file generation method, test file generation device, test file testing method, test file testing device, storage medium and computer equipment | |
| CN112965912B (en) | Interface test case generation method and device and electronic equipment | |
| CN116881145B (en) | Interface testing method, device, equipment and storage medium based on flow playback | |
| CN116126808A (en) | Behavior log recording method, device, computer equipment and storage medium | |
| CN115293685A (en) | Logistics order state tracking method, device, equipment and storage medium | |
| CN113392005A (en) | Large file processing test method and system | |
| CN112597119A (en) | Method and device for generating processing log and storage medium | |
| CN113760696A (en) | Program problem positioning method and device, electronic equipment and storage medium | |
| CN112632211A (en) | Semantic information processing method and equipment for mobile robot | |
| JP2010152707A (en) | Backup method of database and database system | |
| US8321844B2 (en) | Providing registration of a communication | |
| CN117573564B (en) | Method for automatically identifying differences based on gitlab code submitted log | |
| CN114327377B (en) | Method and device for generating demand tracking matrix, computer equipment and storage medium | |
| CN110837536B (en) | Information processing method, device and storage medium | |
| CN115776508A (en) | Buried point checking method and system | |
| CN117724947A (en) | Method, device, equipment and storage medium for analyzing performance of functional interface |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |