CN114036167A - Method for recording data change operation log in micro service - Google Patents
Method for recording data change operation log in micro service Download PDFInfo
- Publication number
- CN114036167A CN114036167A CN202111349715.4A CN202111349715A CN114036167A CN 114036167 A CN114036167 A CN 114036167A CN 202111349715 A CN202111349715 A CN 202111349715A CN 114036167 A CN114036167 A CN 114036167A
- Authority
- CN
- China
- Prior art keywords
- service
- cluster
- operation log
- data
- micro
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention belongs to the technical field of micro-service, in particular to a micro-service system which comprises the following components: a method for recording data change operation logs in micro-service comprises a client module, a micro-service cluster data storage cluster, an intermediate cluster and an operation log service cluster, wherein the client module comprises an APP, ios and android, and the intermediate cluster comprises canal, a message cluster and an ES cluster. The invention can rapidly collect all data change records of business operation in the micro-service environment without loss, can trace the business data change, can quickly locate problems, quickly recover data for data support, initiates a delay record operation log task in an important business operation interface with complex business flow, and asynchronously collects and stores the data change in the background by the operation log service, thereby providing data support for the safe driving and the safe navigation of business operation, and rapidly recovering and rapidly rolling back the business operation.
Description
Technical Field
The invention relates to the technical field of micro services, in particular to a method for recording a data change operation log in a micro service.
Background
In the single service, generally, one service is directly connected to one or more databases, and all data changes are processed in the service operation method, so that data change records of the corresponding databases can be conveniently collected and the data change range can be directly recorded.
Therefore, we propose a method for logging data change operations in microservices to solve the above problems.
Disclosure of Invention
The invention aims to solve the defects in the prior art and provides a method for recording a data change operation log in a microservice.
In order to achieve the purpose, the invention adopts the following technical scheme: a method for recording data change operation logs in micro-service comprises a client module, a micro-service cluster data storage cluster, an intermediate distributed cluster and an operation log service cluster, wherein the client module comprises an APP, ios and android, the intermediate distributed cluster comprises canal, a message cluster and an ES cluster, and the method specifically comprises the following steps:
s1, starting to call a service platform interface to process the service;
s2, calling service after all the business processing is finished;
s3, canal monitors the data storage cluster in real time by subscribing the binlog of the data storage cluster;
s4, subscribing and processing the message generated in the canal by the operation log service cluster;
s5, the operation log service cluster receives a processing request initiated by a delay task in the micro service cluster;
and S6, inquiring the operation log and the data change record according to the business operation.
Preferably, the micro service cluster comprises a first service, a second service and a third service, interfaces of the first service, the second service and the third service are called to process services respectively during service processing, and all the interfaces called by the services are provided with globally unique link tracking ids.
Preferably, the data storage cluster comprises a first database, a second database and a third database, the operation log service cluster comprises a first operation log service and a second operation log service, a platform which is used for calling the database to store a plurality of processing services stores change data into the data storage cluster, an interface which is provided by the operation log platform and is used for recording logs at a client is called, the interface for recording the operation log is used for recording a delay task request for writing the operation log, the service name and the service description of the operation log, a service main key and a traceId are carried in the request, the delay message is called after being set for delay for 5 seconds, all the change data are completely collected, the service interface returns a calling result, and when the calling is finished.
Preferably, the canal monitors the data storage cluster in real time by subscribing to the binlog of the database, acquires data change information in the data storage cluster, and sends the change data to the message queue.
Preferably, a traceId field of the data table change data of each database involved in the business process is identified, and the data change information is temporarily stored in the redis list with the traceId field as a storage primary key.
Preferably, the traceId is extracted from the processing request parameter, the data list cached in the redis list is extracted from the redis with the traceId as a key value, an operation log object is generated, the redis is combined, the data extracted from the list is added to the operation log object, the database change information generated by the current business operation is generated, and the operation log object generated by generating the operation log object is stored in the ES cluster.
Preferably, the service name and the service main key of the service to be inquired are provided, and the operation log data is inquired from the ES cluster according to the conditions.
Compared with the prior art, the invention has the beneficial effects that:
1. the method has less code invasion to the business processing, and only a method for writing the log needs to be called;
2. the method basically does not affect the performance of service processing, calls the log writing method to do no specific service processing, and quickly returns after a delay processing message is sent;
3. the method realizes the asynchronous execution of the whole process, and automatically acquires, processes and stores data;
4. the method records the complete data change process, can quickly trace the change and quickly recover the data;
5. the lossless performance quickly collects all data change records of business operation in the micro-service environment, so that business data change can be traced, problems can be quickly positioned, and data support can be quickly recovered.
Drawings
Fig. 1 is a flowchart of a method for recording a data change operation log in a microservice according to the present invention.
In the figure: 1. a client module; 11. APP; 12. ios; 13. android; 2. a micro-service cluster; 21. a first business service; 22. a second service; 23. a third business service; 3. a data storage cluster; 31. a first database; 32. a second database; 33. a third database; 4. intermediate distribution clusters; 41. canal; 42. clustering the messages; 43. an ES cluster; 5. operating a log service cluster; 51. operation log service one; 52. operation log service 2
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Examples
Referring to fig. 1, in the present embodiment, a method for recording a data change operation log in a microservice is provided, where the method includes a client module 1, a microservice cluster 2, a data storage cluster 3, an intermediate cluster 4, and an operation log service cluster 5, where the client module 1 includes an APP11, an ios12, and an android13, and the intermediate cluster 4 includes a canal41, a message cluster 42, and an ES cluster 43, and the method specifically includes:
s1, starting to call the service platform interface to process the service;
s2, calling service after all the service processing is finished;
s3, canal41 monitors the data storage cluster 3 in real time by subscribing to the binlog of the data storage cluster 3;
s4, the operation log service cluster 5 subscribes to the message generated in canal41 and processes the message;
s5, the operation log service cluster 5 receives the processing request initiated by the delay task in the micro service cluster 2;
and S6, inquiring the operation log and the data change record according to the business operation.
In this embodiment, the micro service cluster 2 includes a first service 21, a second service 22, and a third service 23, during service processing, interfaces of the first service 21, the second service 22, and the third service 23 are invoked to process services, respectively, all interfaces invoked by the services are provided with globally unique link trace ids, the data storage cluster 3 includes a first database 31, a second database 32, and a third database 33, the operation log service cluster 5 includes a first operation log service 51 and a second operation log service 52, a platform invoking the database to store a plurality of processing services stores change data into the data storage cluster 3, an interface invoking a client record log provided by the operation log platform, the record operation log interface initiates a delay task request for writing the operation log, a service name and a service description of the operation log in the request, a service master key, and a traceId, and the delay message is called after the delay is set for 5 seconds, the service interface returns a calling result after all the change data are completely collected, and when the calling is finished.
In this embodiment, canal41 monitors the data storage cluster 3 in real time by subscribing to binlog of the database, canal41 obtains data change information in the data storage cluster 3, canal41 sends the change data to a message queue, identifies a traceId field of data table change data of each database involved in business processing, and temporarily stores the data change information in a redis list with the traceId field as a storage primary key.
In this embodiment, the traceId is taken out from the processing request parameter, the data list cached in the redis list is taken out from the redis with the traceId as a key value, an operation log object is generated, the redis is merged, the data taken out from the list is added to the operation log object, database change information generated by the current business operation is generated, the operation log object generated by the operation log object is stored in the ES cluster 43, for example, in an important business operation interface with a complex business process, 10 micro-services are involved in calling, the data of 20 data tables are changed, finally, only 20ms is spent in the business operation interface, a delay recording operation log task is initiated, the operation log service asynchronously collects and stores the data changes at the background, and data support is provided for business operation fast recovery and fast rollback.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art should be able to cover the technical scope of the present invention and the equivalent alternatives or modifications according to the technical solution and the inventive concept of the present invention within the technical scope of the present invention.
Claims (7)
1. A method for recording data change operation logs in micro-service comprises a client module (1), a micro-service cluster (2), a data storage cluster (3), an intermediate cluster (4) and an operation log service cluster (5), wherein the client module (1) comprises an APP (11), ios (12) and android (13), the intermediate cluster (4) comprises a canal (41), a message cluster (42) and an ES cluster (43), and the method specifically comprises the following steps:
s1, starting to call a service platform interface to process the service;
s2, calling service after all the business processing is finished;
s3, canal (41) monitors the data storage cluster (3) in real time by subscribing to binlog of the data storage cluster (3);
s4, subscribing and processing the message generated in the canal (41) by the operation log service cluster (5);
s5, the operation log service cluster (5) receives a processing request initiated by a delay task in the micro service cluster (2);
and S6, inquiring the operation log and the data change record according to the business operation.
2. The method for recording the data change operation log in the micro-service according to claim 1, wherein the micro-service cluster (2) comprises a first service (21), a second service (22) and a third service (23), interfaces of the first service (21), the second service (22) and the third service (23) are called to process the service during service processing, and all interfaces called by the service are provided with globally unique link trace id.
3. The method for recording the data change operation log in the microservice according to claim 1, wherein the data storage cluster (3) comprises a first database (31), a second database (32) and a third database (33), the operation log service cluster (5) comprises a first operation log service (51) and a second operation log service (52), the first operation log service and the second operation log service are called to store the change data in the data storage cluster (3) by a platform for calling a plurality of processing services, an interface for calling a client record log provided by an operation log platform is called, the record operation log interface initiates a delay task request for writing the operation log, the service name and the service description, the service main key and the traceId of the operation log are included in the request, the delay message is called after being set for 5 seconds, all the change data are collected completely, and the service interface returns the calling result, when the call ends.
4. The method for recording the data change operation log in the microservice according to claim 1, wherein the canal (41) monitors the data storage cluster (3) in real time by subscribing to binlog of the database, the canal (41) fetches the data change information in the data storage cluster (3), and the canal (41) sends the change data to the message queue.
5. The method of claim 1, wherein a traceId field of the data table change data of each database involved in the business process is identified, and the data change information is temporarily stored in the redis list with the traceId field as a storage primary key.
6. The method for recording data change operation log in microservice according to claim 1, wherein the traceId is extracted from the processing request parameter, the data list cached in the redis list is extracted with the traceId as a key value from the redis, an operation log object is generated, the redis is merged, the data extracted from the list is added to the operation log object, the database change information generated by the current business operation is generated, and the operation log object generated by the operation log object is saved in the ES cluster (43).
7. The method for recording the operation log of data change in the microservice according to claim 1, wherein the service name of the service to be queried, the service primary key, is provided, and the operation log data is queried from the ES cluster (43) according to the condition.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111349715.4A CN114036167A (en) | 2021-11-15 | 2021-11-15 | Method for recording data change operation log in micro service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111349715.4A CN114036167A (en) | 2021-11-15 | 2021-11-15 | Method for recording data change operation log in micro service |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114036167A true CN114036167A (en) | 2022-02-11 |
Family
ID=80137639
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111349715.4A Withdrawn CN114036167A (en) | 2021-11-15 | 2021-11-15 | Method for recording data change operation log in micro service |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114036167A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114780348A (en) * | 2022-04-28 | 2022-07-22 | 四川虹魔方网络科技有限公司 | Method for asynchronously monitoring client operation log based on distributed deployment environment |
| CN114896232A (en) * | 2022-04-24 | 2022-08-12 | 重庆旅游云信息科技有限公司 | Method for supporting data acquisition and cleaning of multi-type positioning hardware |
-
2021
- 2021-11-15 CN CN202111349715.4A patent/CN114036167A/en not_active Withdrawn
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114896232A (en) * | 2022-04-24 | 2022-08-12 | 重庆旅游云信息科技有限公司 | Method for supporting data acquisition and cleaning of multi-type positioning hardware |
| CN114896232B (en) * | 2022-04-24 | 2023-05-23 | 重庆旅游云信息科技有限公司 | Method for supporting multi-type positioning hardware data acquisition and cleaning |
| CN114780348A (en) * | 2022-04-28 | 2022-07-22 | 四川虹魔方网络科技有限公司 | Method for asynchronously monitoring client operation log based on distributed deployment environment |
| CN114780348B (en) * | 2022-04-28 | 2023-02-07 | 四川虹魔方网络科技有限公司 | Method for asynchronously monitoring client operation log based on distributed deployment environment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114036167A (en) | Method for recording data change operation log in micro service | |
| CN110321387B (en) | Data synchronization method, equipment and terminal equipment | |
| CN110309231B (en) | Cross-machine-room data synchronization method and system | |
| US20050278706A1 (en) | System, method, and computer program product for logging diagnostic information | |
| CN109408341A (en) | Distributed memory system method for monitoring performance, device, equipment and readable storage medium storing program for executing | |
| CN111355802A (en) | Information pushing method and device | |
| CN107402806B (en) | Task processing method and device of distributed file architecture | |
| CN111324480A (en) | Large host transaction fault positioning system and method | |
| CN116881051B (en) | Data backup and recovery method and device, electronic equipment and storage medium | |
| CN107153679B (en) | Extraction statistical method and system for semi-structured big data | |
| CN113761079A (en) | Data access method, system and storage medium | |
| CN111143304A (en) | Micro-service system abnormal log analysis method based on request link | |
| CN111737203A (en) | Database history log backtracking method, device, system, equipment and storage medium | |
| CN112579552A (en) | Log storage and calling method, device and system | |
| CN116303276A (en) | Method for realizing file export by spring batch nested script | |
| EP3082050A1 (en) | Mass data fusion storage method and system | |
| US20160085638A1 (en) | Computer system and method of identifying a failure | |
| CN114022279A (en) | Service data error correction method, device, equipment and readable storage medium | |
| CN117290163B (en) | Data disaster recovery backup system, method and medium based on relational database | |
| CN111522871B (en) | Order data aggregation method, device and storage medium | |
| CN113472469B (en) | Data synchronization method, device, equipment and storage medium | |
| CN116170321B (en) | Data collection method, device, equipment and storage medium for link tracking | |
| CN110970038B (en) | Voice decoding method and device | |
| CN115858497A (en) | Method based on TDSQL rollback | |
| JP2007226733A (en) | Normality check method for database, normality check program and normality check device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20220211 |