CN112764888B - Distributed transaction checking and judging method and system based on log analysis - Google Patents
Distributed transaction checking and judging method and system based on log analysis Download PDFInfo
- Publication number
- CN112764888B CN112764888B CN202110084412.8A CN202110084412A CN112764888B CN 112764888 B CN112764888 B CN 112764888B CN 202110084412 A CN202110084412 A CN 202110084412A CN 112764888 B CN112764888 B CN 112764888B
- Authority
- CN
- China
- Prior art keywords
- transaction
- log
- distributed
- distributed transaction
- obtaining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/465—Distributed object oriented systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/466—Transaction processing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a distributed transaction checking and judging method and a system based on log analysis, which are used for collecting all logs generated in the running of a distributed database in real time to obtain cluster logs; obtaining a distributed transaction cluster log according to the cluster log; obtaining a first distributed transaction log according to the distributed transaction cluster log; obtaining a first component transaction packet according to the first distributed transaction log; acquiring two or more pieces of corresponding fragment information according to the first component transaction packet; judging whether the log abstracts in the fragment information and the transaction logs in the component transaction packet meet a first preset condition or not; and when the consistency of the distributed transaction is met, obtaining a first checking judgment result, wherein the first checking judgment result is that the distributed transaction passes consistency check. The method solves the technical problem that whether the data of the distributed transaction on the plurality of fragments are consistent or not is judged in the prior art.
Description
Technical Field
The invention relates to the field of transaction checking and judging, in particular to a distributed transaction checking and judging method and system based on log analysis.
Background
At present, customers increasingly pay more attention to the reliability and stability of services, how to ensure the consistency of service data by means of a database, and improving the quality of service becomes the central importance of the service field. In the prior art, whether a distributed transaction is consistent or not is judged only by means of existing components of a distributed database, and the consistency of the distributed transaction cannot be judged when an abnormality occurs in the components, that is, whether the data of the distributed transaction on a plurality of fragments are consistent or not cannot be confirmed.
However, in the process of implementing the technical solution of the invention in the embodiments of the present application, the inventors of the present application find that the above-mentioned technology has at least the following technical problems:
the prior art has the technical problem that whether the data of the distributed transaction on a plurality of fragments are consistent or not is judged.
Disclosure of Invention
The embodiment of the application provides a method and a system for checking and judging distributed transactions based on log analysis, solves the technical problem that whether data of the distributed transactions on a plurality of fragments are consistent or not in the prior art, achieves the purpose of judging the consistency of the distributed transactions based on the distributed transaction logs generated in the operation of the components, overcomes the defect that whether the transactions are consistent or not only judged by the interior of the components, judges the consistency of the transactions by the generated transaction logs, and ensures the technical effect of normal operation of the components.
In view of the foregoing problems, the embodiments of the present application provide a method and a system for distributed transaction checking and determining based on log analysis.
In a first aspect, an embodiment of the present application provides a method for determining distributed transaction checking based on log analysis, where the method includes: collecting all logs generated in the operation of a distributed database in real time to obtain cluster logs; obtaining a distributed transaction cluster log according to the cluster log; obtaining a first distributed transaction log according to the distributed transaction cluster log; obtaining a first component transaction packet according to the first distributed transaction log, wherein the component transaction packet comprises two or more transaction logs; acquiring two or more pieces of corresponding fragment information according to the first component transaction packet, wherein each piece of fragment information comprises a plurality of log abstracts; judging whether the log abstracts in the fragment information and the transaction logs in the component transaction packet meet a first preset condition or not; and when the consistency of the distributed transaction is met, obtaining a first checking judgment result, wherein the first checking judgment result is that the distributed transaction passes consistency check.
In another aspect, the present application further provides a system for checking and determining distributed transactions based on log analysis, where the system includes: the first acquisition unit is used for acquiring all logs generated in the operation of the distributed database in real time to acquire cluster logs; a second obtaining unit, configured to obtain a distributed transaction cluster log according to the cluster log; a third obtaining unit, configured to obtain a first distributed transaction log according to the distributed transaction cluster log; a fourth obtaining unit, configured to obtain a first component transaction packet according to the first distributed transaction log, where the component transaction packet includes two or more transaction logs; a fifth obtaining unit, configured to obtain two or more pieces of corresponding fragmentation information according to the first component transaction packet, where each piece of fragmentation information includes multiple log digests; the first judging unit is used for judging whether the log abstracts in the fragment information and the transaction logs in the component transaction packet meet a first preset condition or not; a sixth obtaining unit, configured to, when the first check result is satisfied, obtain a first check judgment result, where the first check judgment result is that the distributed transaction passes consistency check.
In a third aspect, the present invention provides a distributed transaction checking and determining system based on log analysis, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to the first aspect when executing the program.
One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:
the method comprises the steps of acquiring all logs generated in the operation of a database in real time to obtain cluster logs, obtaining distributed transaction cluster logs according to the cluster logs, obtaining a first component transaction packet of a first distributed transaction log in the cluster, obtaining a log abstract of fragment information corresponding to the component transaction packet, and judging whether the transaction logs and the log abstract meet a first preset condition, so that the purposes of judging the consistency of the distributed transactions based on the distributed transaction logs generated in the operation of the components, making up the defect of judging whether the transactions are consistent only by depending on the interior of the components, judging the consistency of the transactions by using the generated transaction logs and ensuring the technical effect of normal operation of the components are achieved.
The foregoing description is only an overview of the technical solutions of the present application, and the present application can be implemented according to the content of the description in order to make the technical means of the present application more clearly understood, and the following detailed description of the present application is given in order to make the above and other objects, features, and advantages of the present application more clearly understandable.
Drawings
Fig. 1 is a schematic flowchart of a distributed transaction checking and determining method based on log analysis according to an embodiment of the present application;
fig. 2 is a schematic structural diagram of a distributed transaction checking and determining system based on log analysis according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of an exemplary electronic device according to an embodiment of the present application.
Description of reference numerals: a first obtaining unit 11, a second obtaining unit 12, a third obtaining unit 13, a fourth obtaining unit 14, a fifth obtaining unit 15, a first judging unit 16, a sixth obtaining unit 17, a bus 300, a receiver 301, a processor 302, a transmitter 303, a memory 304, and a bus interface 306.
Detailed Description
The embodiment of the application provides a method and a system for checking and judging distributed transactions based on log analysis, solves the technical problem that whether data of the distributed transactions on a plurality of fragments are consistent or not in the prior art, achieves the purpose of judging the consistency of the distributed transactions based on the distributed transaction logs generated in the operation of the components, overcomes the defect that whether the transactions are consistent or not only judged by the interior of the components, judges the consistency of the transactions by the generated transaction logs, and ensures the technical effect of normal operation of the components. Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only a few embodiments of the present application, and not all embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein.
Summary of the application
At present, customers increasingly pay more attention to the reliability and stability of services, how to ensure the consistency of service data by means of a database, and improving the quality of service becomes the central importance of the service field. In the prior art, whether a distributed transaction is consistent or not is judged only by means of existing components of a distributed database, and the consistency of the distributed transaction cannot be judged when an abnormality occurs in the components, that is, whether the data of the distributed transaction on a plurality of fragments are consistent or not cannot be confirmed. However, the prior art has the technical problem that whether data of a distributed transaction on a plurality of fragments are consistent or not is judged.
In view of the above technical problems, the technical solution provided by the present application has the following general idea:
the embodiment of the application provides a distributed transaction checking and judging method based on log analysis, which comprises the following steps: collecting all logs generated in the operation of a distributed database in real time to obtain cluster logs; obtaining a distributed transaction cluster log according to the cluster log; obtaining a first distributed transaction log according to the distributed transaction cluster log; obtaining a first component transaction packet according to the first distributed transaction log, wherein the component transaction packet comprises two or more transaction logs; acquiring two or more pieces of corresponding fragment information according to the first component transaction packet, wherein each piece of fragment information comprises a plurality of log abstracts; judging whether the log abstracts in the fragment information and the transaction logs in the component transaction packet meet a first preset condition or not; and when the consistency of the distributed transaction is met, obtaining a first checking judgment result, wherein the first checking judgment result is that the distributed transaction passes consistency check.
Having thus described the general principles of the present application, various non-limiting embodiments thereof will now be described in detail with reference to the accompanying drawings.
Example one
As shown in fig. 1, an embodiment of the present application provides a distributed transaction checking and determining method based on log analysis, where the method includes:
step S100: collecting all logs generated in the operation of a distributed database in real time to obtain cluster logs;
specifically, the distributed database usually uses smaller computer systems, each computer can be individually located in one place, each computer may have a complete copy or partial copy of DBMS therein, and has its own local database, many computers located at different locations are interconnected through a network to form a complete and global logically centralized and physically distributed large database, and logs generated during the operation of the distributed database are collected to obtain cluster logs.
Step S200: obtaining a distributed transaction cluster log according to the cluster log;
for example, the distributed transaction cluster logs are obtained by collecting MySQLbrinlog logs, proxy transaction logs and dbagent logs of the distributed database in real time and filtering out the distributed transaction logs. The MySQLbrinlog records the change operation of table data in a database in a binary form, and does not include the operation of non-changed data content such as select, show and the like, and the Proxy transaction log is obtained by generating a transaction packet through a Proxy end, namely sequentially scanning a log file, and outputting when encountering a complete transaction log. The dbagent log is the output record of the rollback of the committed transaction.
Further, in the obtaining a distributed transaction cluster log according to the cluster log, step S200 in this embodiment of the present application further includes:
step S210: filtering out the distributed transaction logs according to the cluster logs, wherein each distributed transaction log is provided with a distributed transaction identifier;
step S220: grouping distributed transaction logs in the distributed transaction cluster logs according to the distributed transaction identifiers to obtain a plurality of distributed transaction combinations;
step S230: and storing the plurality of distributed transaction combinations through a transaction log storage tool to obtain the distributed transaction cluster log.
Specifically, the distributed transaction identifier is unique identification information corresponding to the identity of the distributed transaction, that is, a unique ID corresponding to a transaction, for example, the distributed transaction identifier may be a GTID, a GTID is executed only once on a server, and the distributed transaction logs are combined together according to the distributed transaction identifier to obtain the distributed transaction cluster log.
Step S300: obtaining a first distributed transaction log according to the distributed transaction cluster log;
step S400: obtaining a first component transaction packet according to the first distributed transaction log, wherein the component transaction packet comprises two or more transaction logs;
specifically, binlog logs of MySQL are collected through a canal component, proxy transaction logs and dbagent logs are collected through a flux component, and then the collected logs are written into topic of each component through kafka. And processing the collected binlog log of the MySQL and the transaction log on the proxy in real time according to a real-time stream processing tool to obtain a first component transaction log of the distributed transaction log.
Step S500: acquiring two or more pieces of corresponding fragment information according to the first component transaction packet, wherein each piece of fragment information comprises a plurality of log abstracts;
in particular, the shard is a type of database partitioning that divides a large database into smaller, faster, and more manageable portions, called data shards. And acquiring at least two pieces of fragment information corresponding to the component transaction packet, and acquiring a plurality of pieces of log summary information in the fragment information.
Step S600: judging whether the log abstracts in the fragment information and the transaction logs in the component transaction packet meet a first preset condition or not;
step S700: and when the consistency of the distributed transaction is met, obtaining a first checking and judging result, wherein the first checking and judging result is that the distributed transaction passes consistency checking.
Specifically, according to whether an inclusion relationship exists between a log summary in the fragment information and a transaction log of the component transaction packet, for example, when the component transaction packet is a Proxy end group transaction packet, it is determined that the transaction log of the Proxy end group transaction packet includes a log summary of its fragment, and further, it is determined whether the log summary of the fragment information is included in the transaction log of the Proxy end group transaction packet, and when the above conditions are satisfied, a first check determination result is obtained, and the first check determination result is that the distributed transaction passes consistency check. The technical effects that the consistency of the distributed transaction is judged based on the distributed transaction log generated in the operation of the component, the defect that whether the transaction is consistent or not is judged only by depending on the inside of the component is overcome, the consistency of the transaction is judged through the generated transaction log, and the normal operation of the component is ensured are achieved.
Further, in the obtaining the first component transaction package according to the first distributed transaction log, step S400 in this embodiment of the present application further includes:
step S410: obtaining an end instruction of the first distributed transaction log;
step S420: obtaining a first end time according to the end instruction;
step S430: and according to the first starting time and the first ending time of the first distributed transaction log, obtaining a first component transaction packet with the same distributed transaction identifier from the distributed transaction cluster log through a real-time stream processing tool.
Specifically, the end instruction detection is performed on the distributed transaction log through a real-time stream processing tool, the start instruction of the transaction is pushed back according to the end instruction, the duration of the transaction is obtained, and the first component transaction packet is obtained from the distributed transaction cluster log through the real-time stream processing tool. And according to the difference of the real-time stream processing tools, obtaining the response time of processing the transaction log by different tools, and according to the response time, adjusting the starting and ending time of the transaction to obtain the first component transaction packet. For example, the real-time stream processing tool may be a flink real-time stream processing tool, and the flink real-time stream processing tool is used for processing the collected binlog log of MySQL and the transaction log on proxy in real time to perform transaction judgment.
Further, the determining whether the log digests in the plurality of pieces of fragmentation information and the transaction logs in the component transaction package satisfy a first predetermined condition, in step S600 in this embodiment of the present application, further includes:
step S610: and when the log summary in the fragment information is contained in the transaction log in the component transaction packet, the first preset condition is met.
Specifically, the log digest of the fragment information is generated by a command of a component transaction, the first predetermined condition is to determine whether the transaction log of the component transaction is consistent with the log of the fragment information, and determine whether the log digest of the fragment information is included in the transaction log of the component transaction packet, when the log digest of the fragment information is included in the transaction log of the component transaction packet, the first predetermined condition is satisfied, that is, a first check determination result is obtained, and the first check determination result is that the distributed transaction passes consistency check. The distributed transaction consistency judgment is carried out based on the distributed transaction log generated in the operation of the component, the defect that whether the transactions are consistent or not is judged only by depending on the inside of the component is overcome, and the technical effect of ensuring the normal operation of the component is achieved by carrying out the transaction consistency judgment on the generated transaction log.
Further, the embodiment of the present application further includes:
step S810: obtaining a fragmentation link corresponding to each piece of fragmentation information according to two or more pieces of fragmentation information connected by the first component transaction packet;
step S820: judging whether the fragmentation links of each piece of fragmentation information are the same;
step S830: and when the two transactions are the same, obtaining the first checking and judging result, wherein the first checking and judging result is that the distributed transactions pass consistency check.
Specifically, for a normally executed transaction, the link must satisfy that the front-end link is not changed, that is, the front-end broken link information must be recorded together; the backend link is not changed, i.e. during a transaction: there can be only one link on each slice. For example, the change of proxy transaction to trace must be contained in binlog of DB, i.e.: the binlog log on the corresponding fragment has a corresponding transaction record. Thus, normal distributed transaction detection may be judged by: checking uniqueness of the back-end link, wherein if the uniqueness is not the only, the distributed transactions are inconsistent; and checking the analysis results of the proxy transaction log P and the transaction log B in the binlog to satisfy the inclusion relationship, namely P contains B. Whether the fragment links of the fragment information are the same or not is judged, so that whether the consistency check of the distributed transaction is more accurate is judged, and the technical effect of ensuring the normal operation of the component is further achieved.
Further, after obtaining the first checking and determining result when the first checking and determining result is satisfied, step S700 in this embodiment further includes:
step S710: and obtaining storage information according to the first checking judgment result, wherein the storage information is used for storing the first checking judgment result in a storage module.
Specifically, the result of the analysis determination is stored. For example, the storage unit may be an HBase, which is a distributed, column-oriented open database, as well as Bigtable utilizing distributed data storage provided by the Google File System (File System), which provides Bigtable-like capabilities on top of Hadoop. HBase is a sub-item of the Hadoop item of Apache. HBase is different from a general relational database, and is a database suitable for unstructured data storage.
For example, the detailed process of the consistency determination of the distributed transaction log is as follows:
tracking transaction results by parsing proxy's received sequence of statements:
1) set autocommit =0 or start transaction or begin, namely, starting a transaction and recording transaction change information;
2) An update T operation is carried out, and if 2 OK records are returned by the fragment 1, the update operation of 2 tables T is recorded in the fragment 1; if the fragment 2 returns OK records 1, recording the update operation of 1 table T in the fragment 2; if the fragment 4 returns OK records 3, recording the update operation of 3 tables T in the fragment 4;
3) Here, a savepoint point s1 is marked;
4) An insert T1, if the fragment 6 returns OK records 1, recording that the fragment 6 has 1 insert operation in the table T1; if 8 OK records are returned by the fragment 8, 8 insert operations of the table T1 are recorded in the fragment 8;
5) rolling back to savepoint s1, returning to the saving point s1, namely step 3, discarding all the changes after the change, and restarting to record the change;
6) If the partition 1 returns OK records 1 record, the partition 1 is recorded with 1 update operation of the table T2; and if the fragment 3 returns OK records 6, 6 update operations of the table T2 are recorded in the fragment 3.
7) When commit is encountered, the change is valid; otherwise, all are discarded.
Thus, the end result of a distributed transaction on proxy is:
1) On slice 1, there are 2 updates of table T and 1 update of table T2;
2) On slice 2, there are 1 update of table T;
3) On slice 4, there are 3 updates for table T;
4) On slice 3, there are 6 updates for table T2.
For a normally executed transaction, it must also be satisfied:
1) The front-end link is not changed (the front-end broken link information must be recorded together);
2) The back-end link is unchanged (i.e. during the transaction: only one link on each slice);
3) The change from proxy transaction to trace must be contained in binlog of DB, i.e.: the binlog log on the corresponding fragment has corresponding transaction records;
thus, normal distributed transaction detection may be judged in the following manner
1) Checking uniqueness of back-end link, if not, distributed transaction inconsistency
2) And checking the analysis results of the proxy transaction log P and the transaction log B in the binlog to satisfy the inclusion relationship, namely P contains B.
To sum up, the distributed transaction checking and determining method and system based on log analysis provided by the embodiment of the present application have the following technical effects:
1. the method comprises the steps of acquiring all logs generated in the operation of a database in real time to obtain cluster logs, obtaining distributed transaction cluster logs according to the cluster logs, obtaining a first component transaction packet of a first distributed transaction log in the cluster, obtaining a log abstract of fragment information corresponding to the component transaction packet, and judging whether the transaction logs and the log abstract meet a first preset condition, so that the purposes of judging the consistency of the distributed transactions based on the distributed transaction logs generated in the operation of the components, making up the defect of judging whether the transactions are consistent only by depending on the interior of the components, judging the consistency of the transactions by using the generated transaction logs and ensuring the technical effect of normal operation of the components are achieved.
2. By adopting the mode of judging whether the fragment links of the fragment information are the same or not, the consistency check of the distributed transaction is more accurately judged, and the technical effect of ensuring the normal operation of the component is further achieved.
Example two
Based on the same inventive concept as the distributed transaction checking and judging method based on log analysis in the foregoing embodiment, the present invention further provides a distributed transaction checking and judging system based on log analysis, as shown in fig. 2, the system includes:
the first obtaining unit 11 is configured to collect all logs generated during operation of the distributed database in real time to obtain cluster logs;
a second obtaining unit 12, where the second obtaining unit 12 is configured to obtain a distributed transaction cluster log according to the cluster log;
a third obtaining unit 13, where the third obtaining unit 13 is configured to obtain a first distributed transaction log according to the distributed transaction cluster log;
a fourth obtaining unit 14, where the fourth obtaining unit 14 is configured to obtain a first component transaction packet according to the first distributed transaction log, where the component transaction packet includes two or more transaction logs;
a fifth obtaining unit 15, where the fifth obtaining unit 15 is configured to obtain two or more pieces of corresponding fragment information according to the first component transaction packet, where each piece of fragment information includes a plurality of log digests;
a first judging unit 16, where the first judging unit 16 is configured to judge whether log digests in the plurality of pieces of fragmentation information and transaction logs in the component transaction packet satisfy a first predetermined condition;
a sixth obtaining unit 17, where the sixth obtaining unit 17 is configured to, when the first check result is satisfied, obtain a first check judgment result, where the first check judgment result is that the distributed transaction passes consistency check.
Further, the system further comprises:
the first filtering unit is used for filtering the distributed transaction logs according to the cluster logs, wherein each distributed transaction log is provided with a distributed transaction identifier;
a seventh obtaining unit, configured to group the distributed transaction logs in the distributed transaction cluster log according to the distributed transaction identifiers, and obtain a plurality of distributed transaction combinations;
an eighth obtaining unit, configured to store, by a transaction log storage tool, the multiple distributed transaction combinations to obtain the distributed transaction cluster log.
Further, the system further comprises:
a ninth obtaining unit to obtain an end instruction of the first distributed transaction log;
a tenth obtaining unit, configured to obtain a first end time according to the end instruction;
an eleventh obtaining unit, configured to obtain, by a real-time stream processing tool, the first component transaction packet from the distributed transaction cluster log according to the first start time and the first end time of the first distributed transaction log.
Further, the system further comprises:
a twelfth obtaining unit, configured to, when a log digest in the fragmentation information is included in a transaction log in the component transaction packet, satisfy the first predetermined condition.
Further, the system further comprises:
a thirteenth obtaining unit, configured to obtain, according to two or more pieces of fragmentation information connected by the first component transaction packet, a fragmentation link corresponding to each piece of fragmentation information;
a second judging unit, configured to judge whether the fragmentation links of each piece of fragmentation information are the same;
a fourteenth obtaining unit, configured to, when the two transactions are the same, obtain the first checking and determining result, where the first checking and determining result is that the distributed transaction passes consistency check.
Further, the system further comprises:
a fifteenth obtaining unit, configured to obtain, according to the first inspection judgment result, storage information used for storing the first inspection judgment result in a storage module.
Various changes and specific examples of the distributed transaction checking and judging method based on log analysis in the first embodiment of fig. 1 are also applicable to the distributed transaction checking and judging system based on log analysis of this embodiment, and through the foregoing detailed description of the distributed transaction checking and judging method based on log analysis, those skilled in the art can clearly know the implementation method of the distributed transaction checking and judging system based on log analysis in this embodiment, so for the brevity of the description, detailed descriptions are not repeated here.
Exemplary electronic device
The electronic device of the embodiment of the present application is described below with reference to fig. 3.
Fig. 3 illustrates a schematic structural diagram of an electronic device according to an embodiment of the present application.
Based on the inventive concept of a distributed transaction checking and judging method based on log analysis in the foregoing embodiments, the present invention further provides a distributed transaction checking and judging system based on log analysis, on which a computer program is stored, and the computer program, when being executed by a processor, implements the steps of any one of the foregoing distributed transaction checking and judging methods based on log analysis.
Where in fig. 3 a bus architecture (represented by bus 300), bus 300 may include any number of interconnected buses and bridges, bus 300 linking together various circuits including one or more processors, represented by processor 302, and memory, represented by memory 304. The bus 300 may also link together various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface 306 provides an interface between the bus 300 and the receiver 301 and transmitter 303. The receiver 301 and the transmitter 303 may be the same element, i.e., a transceiver, providing a means for communicating with various other systems over a transmission medium.
The processor 302 is responsible for managing the bus 300 and general processing, and the memory 304 may be used for storing data used by the processor 302 in performing operations.
The embodiment of the invention provides a distributed transaction checking and judging method based on log analysis, which comprises the following steps: collecting all logs generated in the operation of a distributed database in real time to obtain cluster logs; obtaining a distributed transaction cluster log according to the cluster log; obtaining a first distributed transaction log according to the distributed transaction cluster log; obtaining a first component transaction packet according to the first distributed transaction log, wherein the component transaction packet comprises two or more transaction logs; acquiring two or more pieces of corresponding fragment information according to the first component transaction packet, wherein each piece of fragment information comprises a plurality of log abstracts; judging whether the log abstracts in the fragment information and the transaction logs in the component transaction packet meet a first preset condition or not; and when the consistency of the distributed transaction is met, obtaining a first checking judgment result, wherein the first checking judgment result is that the distributed transaction passes consistency check. The technical problem that whether data of distributed transactions on a plurality of fragments are consistent or not is judged in the prior art is solved, the distributed transaction consistency judgment is carried out based on distributed transaction logs generated in the operation of the component, the defect that whether the transactions are consistent or not is judged only by means of the interior of the component is overcome, the transaction consistency judgment is carried out through the generated transaction logs, and the technical effect that the normal operation of the component is guaranteed is achieved.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a system for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction system which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the invention.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (8)
1. A distributed transaction checking and judging method based on log analysis is disclosed, wherein the method comprises the following steps:
collecting all logs generated in the operation of a distributed database in real time to obtain cluster logs;
obtaining a distributed transaction cluster log according to the cluster log;
obtaining a first distributed transaction log according to the distributed transaction cluster log;
obtaining a first component transaction packet according to the first distributed transaction log, wherein the component transaction packet comprises two or more distributed transaction logs;
acquiring two or more pieces of corresponding fragment information according to the first component transaction packet, wherein each piece of fragment information comprises a plurality of transaction related log abstracts;
judging whether the log abstracts in the fragment information and the transaction logs in the component transaction packet meet a first preset condition or not;
and when the consistency of the distributed transaction is met, obtaining a first checking judgment result, wherein the first checking judgment result is that the distributed transaction passes consistency check.
2. The method of claim 1, wherein the obtaining a distributed transaction cluster log from the cluster log comprises:
filtering the distributed transaction logs according to the cluster logs, wherein each distributed transaction log is provided with a distributed transaction identifier;
grouping distributed transaction logs in the distributed transaction cluster logs according to the distributed transaction identifiers to obtain a plurality of distributed transaction combinations;
and storing the plurality of distributed transaction combinations through a transaction log storage tool to obtain the distributed transaction cluster log.
3. The method of claim 1, wherein said obtaining a first component transaction package from said first distributed transaction log comprises:
obtaining an end instruction of the first distributed transaction log;
obtaining a first end time according to the end instruction;
and according to the first starting time and the first ending time of the first distributed transaction log, obtaining a first component transaction packet with the same distributed transaction identifier from the distributed transaction cluster log through a real-time stream processing tool.
4. The method of claim 1, wherein the determining whether the log digests in the plurality of the sharded information and the transaction logs in the component transaction package satisfy a first predetermined condition comprises:
and when the log summary in the fragment information is contained in the transaction log in the component transaction packet, the first preset condition is met.
5. The method of claim 1, wherein the method comprises:
obtaining a fragmentation link corresponding to each piece of fragmentation information according to two or more pieces of fragmentation information connected by the first component transaction packet;
judging whether the links of the information of the same transaction in each fragment are the same or not;
and when the distributed transactions are the same, obtaining a first checking and judging result, wherein the first checking and judging result is that the distributed transactions pass consistency checking.
6. The method of claim 1, wherein said obtaining, when satisfied, after obtaining a first inspection determination comprises:
and obtaining storage information according to the first checking judgment result, wherein the storage information is used for storing the first checking judgment result in a storage module.
7. A distributed transaction checking and determining system based on log analysis, wherein the system comprises:
the first acquisition unit is used for acquiring all logs generated in the operation of the distributed database in real time to acquire cluster logs;
a second obtaining unit, configured to obtain a distributed transaction cluster log according to the cluster log;
a third obtaining unit, configured to obtain a first distributed transaction log according to the distributed transaction cluster log;
a fourth obtaining unit, configured to obtain a first component transaction packet according to the first distributed transaction log, where the component transaction packet includes two or more distributed transaction logs;
a fifth obtaining unit, configured to obtain two or more pieces of corresponding fragment information according to the first component transaction packet, where each piece of fragment information includes multiple log digests;
the first judging unit is used for judging whether log abstracts in the fragment information and transaction logs in the component transaction packet meet a first preset condition or not;
a sixth obtaining unit, configured to, when the first check result is satisfied, obtain a first check judgment result, where the first check judgment result is that the distributed transaction passes consistency check.
8. A distributed transaction checking and determining system based on log analysis, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110084412.8A CN112764888B (en) | 2021-01-21 | 2021-01-21 | Distributed transaction checking and judging method and system based on log analysis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110084412.8A CN112764888B (en) | 2021-01-21 | 2021-01-21 | Distributed transaction checking and judging method and system based on log analysis |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112764888A CN112764888A (en) | 2021-05-07 |
| CN112764888B true CN112764888B (en) | 2023-03-24 |
Family
ID=75702542
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110084412.8A Active CN112764888B (en) | 2021-01-21 | 2021-01-21 | Distributed transaction checking and judging method and system based on log analysis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112764888B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114510539B (en) * | 2022-04-18 | 2022-06-24 | 北京易鲸捷信息技术有限公司 | Method for generating and applying consistency check point of distributed database |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109451078B (en) * | 2019-01-10 | 2022-05-03 | 网易（杭州）网络有限公司 | Transaction processing method and device under distributed architecture |
| CN110134735A (en) * | 2019-04-10 | 2019-08-16 | 阿里巴巴集团控股有限公司 | The storage method and device of distributed transaction log |
| CN110413687B (en) * | 2019-05-09 | 2024-01-05 | 国网冀北电力有限公司 | Distributed transaction fault processing method and related equipment based on node interaction verification |
| CN111522631B (en) * | 2020-03-23 | 2024-02-06 | 支付宝(杭州)信息技术有限公司 | Distributed transaction processing method, device, server and medium |
| CN112231144A (en) * | 2020-09-29 | 2021-01-15 | 上海艾融软件股份有限公司 | Data processing method and device and electronic equipment |
-
2021
- 2021-01-21 CN CN202110084412.8A patent/CN112764888B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112764888A (en) | 2021-05-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9996558B2 (en) | Method and system for accessing a set of data tables in a source database | |
| CN108647357B (en) | Data query method and device | |
| CN111221743A (en) | Automatic testing method and system | |
| US9632899B2 (en) | Method for analyzing request logs in advance to acquire path information for identifying problematic part during operation | |
| US11163769B2 (en) | Joining two data tables on a join attribute | |
| CN109213773B (en) | Online fault diagnosis method and device and electronic equipment | |
| CN111522728A (en) | Method for generating automatic test case, electronic device and readable storage medium | |
| CN111581234B (en) | RAC multi-node database query method, device and system | |
| CN110060139B (en) | Accounting processing method and device | |
| CN110134738B (en) | Distributed storage system resource estimation method and device | |
| CN112434043B (en) | Data synchronization method, device, electronic equipment and medium | |
| CN111444188B (en) | Preparation method and device of stock test data, storage medium and electronic equipment | |
| CN114116422B (en) | Hard disk log analysis method, hard disk log analysis device and storage medium | |
| CN108509453B (en) | Information processing method and device | |
| CN113242157A (en) | Centralized data quality monitoring method under distributed processing environment | |
| CN113220907A (en) | Business knowledge graph construction method and device, medium and electronic equipment | |
| CN112764888B (en) | Distributed transaction checking and judging method and system based on log analysis | |
| US10089350B2 (en) | Proactive query migration to prevent failures | |
| US20160357844A1 (en) | Database apparatus, search apparatus, method of constructing partial graph, and search method | |
| CN112559525B (en) | Data checking system, method, device and server | |
| CN110309206B (en) | Order information acquisition method and system | |
| CN109933798B (en) | Audit log analysis method and audit log analysis device | |
| CN115829412A (en) | Index data quantization processing method, system and medium based on business process | |
| CN116186082A (en) | Data summarizing method based on distribution, first server and electronic equipment | |
| CN115269519A (en) | Log detection method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |