CN112181927A - Log auditing method under micro-service architecture - Google Patents

Abstract

A log auditing method under a micro-service architecture comprises the steps of firstly providing a public SDK packet as a public component for log output, using open zipkin to record and track a request link, generating a unique guid for each service interaction request for association, storing the obtained related guid into a service interaction log to realize full link association of the log, and outputting the log content to a file directory of the service during operation; the log filtering module consumes log contents in the cache queue at a constant speed, performs targeted filtering and conversion on logs of different topics and stores the logs into an es cluster; and finally, the display and the alarm of the log content are realized through the kibana and the operation and maintenance management platform. The invention provides the capacity of real-time collection, analysis and display of the logs through the log center, and the log center can know the system performance and the user behavior in time, thereby ensuring the high-reliability operation of the microservice.

Description

Log auditing method under micro-service architecture

Technical Field

The invention relates to the technical field of computer system development, in particular to a log auditing method based on a micro-service architecture.

Background

The service system, the equipment and the application program are generated in real time in the running process, a text for recording the running content is called a log, and each log record generally comprises information description such as date, time, user and content. The user or the operation and maintenance manager can check the reason of the system error by checking the log, count the activity record of the system, or find out the abnormal operation of the user.

As the system architecture becomes larger and more complex, a large amount of log information is generated. Many enterprises have begun to split monolithic applications into microservices. When a large application is split, a loosely coupled module needs to be established, so that the module is convenient to test and the change risk is reduced. In addition, the modules can also be independently deployed to realize the horizontal scale expansion. However, this also brings some problems that are not serious at first sight, but have a very significant effect in the long run — the typical representative of which is logging.

Under the architecture of the microservice, logs of all services are independently recorded on a server hard disk, and interaction among the services is carried out through a stateless protocol such as http, so that the logs are difficult to associate with each other, and the complexity of log investigation of operation and maintenance managers is increased. These logs need to be stored for querying and analysis, whereas traditional relational databases have limited capacity for log storage, querying, and analysis.

With the increasing of the capacity and complexity of a software system, the problem of an original log management mode exists, in the collection aspect, a http interface mode is mainly adopted, logs are actively pushed to a server (an audit platform) by clients (a service system, a server and the like), and all the clients need to be changed along with the change of the interfaces of the server due to the high coupling. Log monitoring aspect under the micro-service architecture, it has been difficult for the past form of single node sending logs to monitor all log states on one request link. Therefore, a log auditing scheme for large-capacity complex scenarios needs to be considered.

Disclosure of Invention

The invention aims to provide an efficient log auditing method under a micro-service architecture, which realizes log auditing in a self-defined protocol mode.

The technical scheme of the invention is as follows:

a log auditing method under a micro-service architecture is characterized in that:

(1) firstly, a common SDK packet is provided as a common component of log output, and the functions of the common SDK packet comprise: tracking a service link, and generating and recording a unique identifier in the service request process; log output, which automatically modifies the format of the log according to different types of logs and outputs the log to a message queue;

(2) recording and tracking a request link by using open zipkin, generating a unique guid for each service interaction request in the tracking process by the zipkin for association, and storing the obtained related guid into a service interaction log to realize the full link association of the log;

(3) after the business system integrates the log component, the log content is output to a file directory of the service during operation;

(4) the log filtering module performs targeted filtering and conversion on the log contents in the constant-speed consumption cache queue aiming at the logs of different topic and then stores the log contents in the es cluster;

(5) and finally, the display and the alarm of the log content are realized through the kibana and the operation and maintenance management platform.

The log center plays a vital role in the micro-service architecture, and is a very important entry point for micro-service monitoring. By adopting the architecture and the principle, the log center can provide real-time acquisition, analysis and display capability for the logs, and the log center can know the system performance and the user behavior in time to ensure the high-reliability operation of the micro-service.

Detailed Description

According to the generation source of the log, the log is classified into a system log and an application log, wherein the system log is mostly a log generated in the operation of middleware, an open source component and the like, the log information is mostly fixed in a form, content and position, special log analysis needs to be carried out on different components during analysis, the application log is generally an application program and a service system developed by developers in a project, and the log can generate fixed log content through a protocol in advance.

The application log can be divided into a user operation log, a service interaction log, an application operation log and a data operation log at different levels of a system architecture.

The user operation log is recorded by the client, produced by the server and mainly used for recording the operation of the user on the application program, such as events of inquiry, addition, modification, deletion and the like.

The service interaction log refers to the interaction between application programs or micro-services, and is mainly divided into two types, namely response and request according to the execution action. Different interaction forms exist according to different protocols, such as http protocol interaction, socket protocol interaction, mq protocol interaction and the like.

The application running log refers to a log record of an action actively initiated by the service system when the service system runs, for example, a log of system starting, timing task and the like.

The data operation log is a modification record of the business system to data, and is specific to different data sources, including a relational database, a non-relational database, a file system and the like, such as oracle, redis, MongoDB, elastic search, fastDFS and the like.

When multiple log types are resolved according to different situations, the business logic eventually needs to be composed of one or more components. When tracking these transactions, it is often difficult to identify them. Therefore, it is necessary to generate a unique identifier for each transaction and use it to correlate events and track transactions.

Finally, auditing the log events in the logging program should include:

1. operating System (OS) events:

starting and shutting down the system;

start and stop service;

network connection change or failure;

changes or attempts to change system security settings and controls.

2. Operating System (OS) audit logging:

login attempt (successful or unsuccessful);

functions performed after login (e.g., reading or updating key files, software installation);

account changes (e.g., account creation and deletion, account rights assignment);

success/failure uses privileged accounts.

3. Application account information:

successful and failed application verification attempts;

application account changes (e.g., account creation and deletion, account rights assignment);

application permissions are used.

4. Application operation:

starting and closing an application program;

application failure;

the main application configuration changes: application transactions, e.g., email servers, record the sender, recipient, subject name and attachment name of each email; the Web server records each URL requested and the response type provided by the server; the business application records which financial records each user accesses.

The log system comprises six modules, namely log output module, log acquisition module, buffer queue module, log filtering module, log storage module, log display module and log alarm module, and can achieve the full coverage of the function of one application program or service system from application coding to application online to application operation and maintenance in a life cycle. The specific implementation process is as follows:

firstly, a common SDK packet is provided as a common component of log output, and the functions of the common SDK packet comprise: 1. and (4) tracking a service link, and generating and recording a unique identifier in the service request process. 2. And log output, namely automatically modifying the format of the log according to different types of logs and outputting the log to a message queue. The micro-service client needs to access the public SDK, and the output and the format of the log do not need to be considered in the service coding process in an AOP (automatic object plane) intercepting mode, so that the full-automatic log intercepting and recording are achieved.

And recording and tracking the request link by using open zipkin, wherein the zipkin generates a unique guid for each service interaction request in the tracking process to be associated, and stores the obtained related guid into the service interaction log to realize the full link association of the log.

After the business system integrates the log component, the log content is output to a file directory of the service during operation.

The reason for using Kafka to transmit the log is that the log has the capacity of data caching, the data of the log can be repeatedly consumed, the Kafka has high availability and can well prevent data loss, and the number of logs generated in different time periods is greatly different according to different requirements of services, so that the log caching under the high concurrency condition can be realized through the Kafka, and the consumption of the log filtering module is delayed.

The log filtering module performs targeted filtering and conversion on the log contents in the constant-speed consumption cache queue aiming at the logs of different topic and stores the log contents in the es cluster.

And finally, the display and the alarm of the log content are realized through the kibana and the operation and maintenance management platform.

Claims (1)

1. A log auditing method under a micro-service architecture is characterized in that:

(1) firstly, a common SDK packet is provided as a common component of log output, and the functions of the common SDK packet comprise: tracking a service link, and generating and recording a unique identifier in the service request process; log output, which automatically modifies the format of the log according to different types of logs and outputs the log to a message queue;

(2) recording and tracking a request link by using open zipkin, generating a unique guid for each service interaction request in the tracking process by the zipkin for association, and storing the obtained related guid into a service interaction log to realize the full link association of the log;

(3) after the business system integrates the log component, the log content is output to a file directory of the service during operation;

(4) the log filtering module performs targeted filtering and conversion on the log contents in the constant-speed consumption cache queue aiming at the logs of different topic and then stores the log contents in the es cluster;

(5) and finally, the display and the alarm of the log content are realized through the kibana and the operation and maintenance management platform.