CN113849475A - Method for unified management of application logs in k8s environment based on ELK - Google Patents

Abstract

The invention relates to a method for summarizing application logs in an ELK-based unified management k8s environment, which comprises the following steps: s01, operating a fileteam container in a daemonset mode in k8S, wherein the fileteam is a log collection program for collecting logs, and the logs meet three specifications, namely a log content output format specification, a log file naming specification and a log file storage position specification; s02, mounting the log directory on the host to the filecut container; s03, injecting the IP address of the host machine into the environment variable of the filebeat container; s04, using the environment variable in the configuration file of fileteam. According to the invention, by unifying the log specification, the log file is named and stored according to a set format ingeniously, and then the log deleting program and the ELK are combined to realize unified and automatic management of the log, so that the operation and maintenance efficiency is improved.

Description

Method for unified management of application logs in k8s environment based on ELK

Technical Field

The invention relates to the technical field of computers, in particular to a method for uniformly managing application logs in a k8s environment.

Background

With the popularity of microservices, more and more enterprises choose to microservice traditional single-body applications and simultaneously use k8s to manage service applications, but enjoy various conveniences brought by microservices and k8s and simultaneously bear troubles brought by the microservices and the k8s, logs are one of the microservices and the k8s is a container arrangement tool and a tool for managing the full life cycle of the applications, and the microservices, the expansion and contraction of the applications and the updating of the applications are very convenient and can realize fault self-healing. Although most of the industry uses ELK to achieve unified log collection and presentation, due to the characteristics of microservice and k8s, the following problems still exist:

firstly, the application function is updated frequently, so that the corresponding log configuration needs to be newly added every time a new application is online, so that the ELK can collect the newly added log.

Secondly, the log is generated quickly, the log file is large, operation and maintenance personnel are required to pay attention to the disk space condition of the host machine at any time, and if the disk space is occupied by the log, application abnormity can be caused.

Thirdly, the logs are collected uniformly, but a plurality of instances of the same application run simultaneously, and the container of which host the logs come from cannot be distinguished when log query is performed.

The Chinese patent document with publication number CN110851396A discloses a technology of 'a unified log design method based on modeling platform micro-service architecture', the technology mentions how to perform a unified log design method under the micro-service, and when the three problems are not solved, meanwhile, the technology names the log files by using the log grade, which is undoubtedly a difficulty for users to check the logs in practical application, the regular logs, alarm logs and error logs use different log levels in one module, logs of different levels can appear in one functional service, if log files are named according to the log levels, the logs of the same service can be dispersed into different files, when an operation and maintenance person or a developer checks the log through the files, the operation and maintenance person or the developer needs to pay attention to a plurality of files at the same time, and the efficiency of checking the problem through the log is reduced.

Disclosure of Invention

The invention aims to provide a method for uniformly managing application logs in a k8s environment based on ELK, which solves the technical problems that: firstly, the application function is frequently updated, so that corresponding log configuration needs to be newly added every time a new application is on-line; secondly, the log is generated quickly, the log file is large, and operation and maintenance personnel need to pay attention to the disk space condition of the host machine at any time; thirdly, the logs are collected uniformly, but a plurality of instances of the same application run simultaneously, and the container of which host the logs come from cannot be distinguished when log query is performed.

In order to solve the above technical problems, the technical solution adopted by the present invention is stated as follows: a method for unified management of application logs in a k8s environment based on ELK comprises the following steps:

s01, operating a filebolt container in a daemonset mode in k8S, wherein the filebolt is a log collection program for collecting logs, and the logs must meet three specifications, namely a log content output format specification, a log file naming specification and a log file storage position specification, wherein the log content output format specification specifies that a unified log content output format must ensure that logstack matches and analyzes logs according to an agreed rule, structured text content can be structurally split, and the split log data content at least comprises time, log levels and log content; the log file naming specification specifies that uniform log file naming must ensure that all log files conforming to naming can be matched when log collection is performed; the log file storage position specification specifies that logs are uniformly stored in a fixed directory in a container, the directory is mounted in the fixed directory on a host, and the host is deployed with filebeat, namely a log collection program is deployed;

s02, mounting the log directory on the host to the filecut container;

s03, injecting the IP address of the host machine into the environment variable of the filebeat container;

s04, using the environment variable in the configuration file of fileteam.

Preferably, the method for deleting the log further comprises the following steps:

s05, the log deleting program reads the data file of fileteam;

s06, judging whether the data file is completely read, if so, executing the next step;

s07, judging whether the path contains log-history, if not, executing the next step;

s08, judging whether the container where the log is located is alive or not, if not, executing the next step;

and S09, deleting the log.

Preferably, the first and second electrodes are formed of a metal,

in the S06, if not, the execution of the log deletion program is terminated;

in the S07, if yes, executing deleting the log;

in S08, if yes, execution of the log deletion program is terminated.

Preferably, the first and second electrodes are formed of a metal,

in S01, the log files are divided into a file currently being written and a history file generated according to a rolling policy, where the file being written is named as "application name- # business classification # -template identification log", and the history file generated according to the rolling policy is named as "application name-yyy.mm.dd-index- # business classification # -template identification log", where the application name represents the name of an application, the business classification represents the business category to which the log belongs, the template identification represents a regular template that needs to be used when the log is parsed, the yy.mm.dd identifies the current year, month and day, and the index identification index number of the same type of log file generated according to the rolling policy, and the rolling policy is configured to roll by date and size.

Preferably, the first and second electrodes are formed of a metal,

in S01, mount/u 01/logs/$ { PROJECT _ NAME } of the host to the/app/logs directory of the CONTAINER, where $ { PROJECT _ NAME } is the NAME of the current application, specify through a yaml file when the CONTAINER is deployed, store the file currently being written to/app/logs/$ { contact _ NAME }/directory in the configuration file of the log, store the history file to/app/logs/$ { air _ NAME }/log-history/directory, where $ { contact _ NAME } is the NAME of the current CONTAINER, and inject the current CONTAINER with an environment variable at the time of CONTAINER startup.

Preferably, the first and second electrodes are formed of a metal,

in S02, mount/01/logs on the host to the app/logs directory of the container for reading all log file contents, and mount/u 01/filebolt/data to the usr/share/filebolt/data directory of the container for saving the current reading position information of each log file on the host.

Preferably, the first and second electrodes are formed of a metal,

the log deleting program is a python script, the log deleting program runs on a host machine in a DameonSet mode of k8s, a/u 01/fileboat/data of the host machine is mounted in a container to a/fileboat/data directory of the container and a/u 01/logs/to a/app/logs directory of the host machine, data.json file contents in the/fileboat/data directory are read in the python script in a timing mode, and the data.json files comprise paths of all files read by the fileboat and current read position information.

By adopting the technical scheme, the beneficial technical effects of the invention are as follows: according to the invention, unified log file naming is adopted, and the naming can ensure that all log files meeting naming specifications can be matched when log collection is carried out, so that the purpose of automatically collecting logs of subsequent newly added applications after one-time configuration is achieved, and the first problem is solved; in the invention, a research and development personnel develops a log deleting program through python, a data file of the filebot needs to be read in the program, the data file records the currently read file position and file path information of each log file monitored by the filebot, whether the log can be deleted or not is judged by comprehensively analyzing 3 information of the reading precision of the log, the directory where the log file is located and the survival state of the container where the log is located, the deleting program can be set to be executed at regular time, once the log file reaches the deleting condition, the log file can be deleted in time, the purposes of saving disk resources and ensuring the service stability are achieved, and the second problem is solved; running a filebetet container in a DaemonSet mode in k8s, mounting a log directory on a host machine into the container, injecting an IP address of the current host machine into an environment variable of the container, and using the environment variable in a configuration file of the filebetet to realize the identification of the host machine from which the log comes, thereby solving the third problem; the invention realizes the unified and automatic management of the logs and improves the operation and maintenance efficiency.

Drawings

FIG. 1 is a general physical architecture diagram of the present invention;

FIG. 2 is a directory structure diagram of the log storage of the present invention;

FIG. 3 is a flow chart of the delete log processing of the present invention.

Detailed Description

The invention will be further explained with reference to the drawings.

As shown in fig. 1, the ELK log analysis system is a collection of logstack, elasticsearch and kibana open source software, and serves as an open source scheme of the log management system, and it can perform log search, analysis and visual display from any source and any format.

elastic search (es): and (4) storing log data by building a cluster, and indexing the log data.

logstash: and processing the log, and storing the log to the es after the log is subjected to structured processing.

kibana: the log information is shown in a view form, and the method is more humanized.

The log processing steps can be performed according to the following flow: the method comprises the steps of firstly carrying out centralized management on the log, then formatting (logstack) the log and outputting the log to an elastic search, further indexing and storing (es) the formatted data, and finally displaying (kibana) the front-end data.

logstack is a powerful data processing tool that can perform data transmission, format processing, and format output, as well as data input (from business), data processing (e.g., filtering and rewriting), and data output (to an elastic search cluster).

The main components of logstash include:

shipper; the log collector is responsible for monitoring the change of the local log file and collecting the latest content of the log file in time, and usually, a remote agent end (agent) only needs to operate the component.

indexer: and the log storage person is responsible for receiving the log and writing the log into the local file.

A brooker: and the log hub is responsible for connecting a plurality of shippers and a plurality of indexers.

serrch and storage: allowing time to be searched and stored.

Web interface: a web-based presentation interface.

kibana is an open source analysis and visualization platform for the elastic search.

As shown in fig. 1, three programs are deployed on a host:

app: a business application.

log-clean: and (5) log deleting program.

And (3) filehead: and a log collection program.

after the app outputs the log to a file, the log is read by the filebeat on the host, the log content is sent to the message queue kafka after being read, the log content is consumed from the kafka by logstack to be formatted and then stored in the elastic search after being processed, and the processed log content is read from the elastic search by the kibana to be visually displayed. The log-clean program will delete the log file that has been read by fileteam regularly.

In the invention, by using an agreed log specification and combining the ELK and the log deleting program, the unified and automatic management of the application log in k8s is realized.

As shown in fig. 1 and fig. 2, the present invention provides a method for uniform management of application logs in a k8s environment based on ELK, comprising the following steps:

s01, operating a filebeat container in a daemonset mode in k8S, wherein the filebeat is a log mobile phone program for collecting logs, the logs must meet three specifications, a log content output format specification, a log file naming specification and a log file storage position specification, as shown in FIG. 2, the logs (applied) are stored in a storage directory structure on a host, all the logs are stored in a/u 01/logs directory on the host, a new directory is created according to a reference name and a container name, and a rolling log generated by a rolling strategy for a certain application is stored in a corresponding log-history directory.

The log content output format specification stipulates that a uniform log content output format must ensure that logstack matches and analyzes logs according to an agreed rule, structured text content can be structurally split, and split log data content at least comprises time, log levels and log content; the log file naming specification specifies that uniform log file naming must ensure that all log files conforming to naming can be matched when log collection is performed; the log file storage position specification specifies that the logs are uniformly stored in a fixed directory in a container, the directory is mounted in the fixed directory on a host, and the filebeat is deployed on the host, namely a log collection program is deployed.

As a specific implementation manner, the output format of the log is formulated, such as: [% d { yyyy-MM-dd HH: MM: ss.SSS } ]% -5p- - [% t ]% -40.40 trigger {40}: m% n.

And S02, mounting the log directory on the host to the filecut container.

Specifically, a-y parameter is added when the filehead container is started, and the parameter value is/u 01/log:/app/logs.

And S03, injecting the IP address of the host machine into the environment variable of the filecut container.

Specifically, $ { HOST _ IP } is used in the configuration file of filehead to use the HOST _ IP parameter value set at container start-up.

S04, use the environment variable in the configuration file of fileteam.

Specifically, a specific implementation manner is that-e HOST _ IP = IP address is added when the filehead container is started, and the IP address is an IP address of the current HOST, such as: 192.168.1.1.

as shown in fig. 3, the method for the uniform management of application logs in k8s environment based on ELK further includes a method for deleting logs, and the method for deleting logs includes the following steps:

s05, the journal deleting program reads the data file of fileteam.

Specifically, the file content is read into a program through the api of the read file of python, and then the file content is converted into a json object, wherein two key data items exist in the json object. source: indicating a path for reading the file; offset: indicating the current location of the read file.

And S06, judging whether the data file is read completely, if so, executing the next step, and if not, terminating the execution of the log deleting program.

Specifically, according to the value of source, the size of the file is obtained through the api of python for obtaining the file state, and then compared with the offset obtained in step S05, if the obtained value is consistent with the offset obtained in step S05, it means that the file is completely read, and if the offset is smaller than the file size, it means that the file is not completely read.

And S07, judging whether the path contains log-history, if not, executing the next step, and if so, executing log deletion.

Specifically, by determining the source value obtained in step S05, it is determined whether the source contains a log-history key, and if so, no new log is written into the file.

And S08, judging whether the container where the log is located is alive or not, and if not, executing the next step.

Specifically, by calling api of k8s through python, all container lists in the current k8s cluster are obtained, and the key implementation code is: ret = K8S _ V1_ api _ list _ pod _ for _ all _ namespaces (watch = False), where ret represents the container name of the current file, and by determining whether the container name is in the ret list, if so, it represents that the container is alive, and if not, it represents that the container has been deleted.

And S09, deleting the log. The key codes are: remove (source), wherein source is the source obtained in S05.

The concept of the present invention will be further described below by taking a logback log component as an example.

A. The method comprises the steps of establishing a log specification, wherein the log specification is a precondition for realizing the method, and the specification mainly comprises three parts, namely a log content output format specification, a log file naming specification and a log file storage position specification.

a. The LOG content output format specification, in the logback LOG component, may define the output format of the LOG content in the configuration FILE through a unified variable, and may refer to the variable directly in the subsequent use, so as to achieve the effects of one modification and multiple effects, such as < property name = "FILE _ LOG _ PATTERN" value = "[% d { yyyyyy-MM-dd HH: MM: ss.sss } ]% -5p- - [% t ]% -40.40logger {40}: m% n"/>, in this embodiment, the format of the LOG content output includes: time, log level, thread name, class name, and log content. Since each complete log starts in the time format, yyyy-MM-dd HH: MM: ss.SSS, the matching configuration of multiple rows of logs at filebeat collection ^ 0-9 {4} - [0-9] {2} - [0-9] {2}, is performed by the following regular expression.

b. The log file naming specification comprises log files which are divided into files currently being written and historical files generated according to a rolling strategy, wherein the files currently being written are named as application name- # business classification # -template identification log ', the historical files generated according to the rolling strategy are named as application name-yyy.MM.dd-index- # business classification # -template identification log', the application name represents the name of the application, the business classification represents the business category of the log, the logs of the same business can be output to the same files, an index taking the application name-business classification as the name is automatically created in an elasticsearch, the template identification represents a template required to be used when the logs are analyzed, and the template is represented as the current month and day, "index" refers to the index number of the same type of log file generated by a rolling policy configured to roll by date and size.

c. As shown in fig. 2, the log file storage location specification, in k8s, the application runs in the form of container, needs to mount the log directory of the host into the container, to prevent log loss problem when the container is exited, such as mounting/u 01/logs/$ { PROJECT _ NAME } of the host to the/app/logs directory of the container, wherein, $ { PROJECT _ NAME } is the NAME of the current application, and the yaml file is listened to when the container is deployed, storing the files currently being written to/app/logs/$ { contact _ NAME }/directory in the configuration file of the log, storing the history file to/app/logs/$ { contact _ NAME }/log-history/directory, wherein, $ { CONTAINER _ NAME } is the NAME of the current CONTAINER, and is injected into the current CONTAINER by means of an environment variable at the time of CONTAINER startup.

B. Collecting and parsing logs

Running a filebolt container in a daemonset mode in k8s, wherein u01/logs on a HOST are required to be mounted in an app/logs directory of the container for reading all log file contents, u 01/filebolts/data are mounted in a user/share/filebolts/data directory of the container for saving current reading position information of each log file on the HOST, and simultaneously injecting an IP address of the current HOST in an environment variable HOST _ IP of the container and using the variable in a configuration file of the filebolt to identify the HOST from which the log comes, and key configuration statements are as follows:

filebeat.inputs：

-type：log

encoding：utf-8

paths:

-/app/logs/*/*/*-#*#-*.log

multiline.pattern: ‘^\[[0-9{4}-[0-9]{2}-[0-9]{2}’

multiline.negate:true

multiline.match:after

fields:

log_src_host:${HOST_IP}

when logstack logs are analyzed, application names and container names can be matched by analyzing paths where the logs are located, meanwhile, corresponding regular expressions are selected through template identification to carry out field accompanying of log contents, and structured data which can be matched in logstack include: the method comprises the steps of host machine IP, container name, application name, service classification, time, log level, thread name, class name and log content, and then writing matched content into an application name-service classification index of the ES.

C. Log deletion (implemented through python script)

As shown in fig. 3, the log deleting program is developed by python, and runs on each host in a dameonset manner of k8s, where/u 01/fileteam/data of the host needs to be mounted in the container to a/fileteam/data directory of the container, and/uo 1/logs/app/log directory of the host to the container, and data.json file content in the data directory is read at regular time in the python script, where the file includes a path of each file read by the fileteam and current read location information.

The journal deletion program determines whether the current file is deletable by the following decision logic:

1) and whether the current reading progress is equal to the total size of the file or not, if so, entering the next step, and otherwise, skipping the file.

2) And if the current file path contains log-history keywords or the container to which the log file belongs is deleted, if the current file path meets any condition, the next step is carried out, otherwise, the file is skipped, and the specific implementation logic refers to the S07 and S08 to judge whether the container is deleted or not and can be implemented through the api of k 8S.

3) And deleting the file.

According to the invention, the log file is named and stored according to a set format, and then the unified and automatic management of the log is realized by combining the deleting program and the ELK, so that the operation and maintenance efficiency is improved.

Claims (7)

1. A method for unified management of application logs in a k8s environment based on ELK is characterized by comprising the following steps:

s01, operating a filebolt container in a daemonset mode in k8S, wherein the filebolt is a log collection program for collecting logs, and the logs meet three specifications, namely a log content output format specification, a log file naming specification and a log file storage position specification, wherein the log content output format specification specifies that a unified log content output format must ensure that logstack matches and analyzes the logs according to an agreed rule, structured text content can be structurally split, and the split log data content at least comprises time, log levels and log content; the log file naming specification specifies that uniform log file naming must ensure that all log files conforming to naming can be matched when log collection is performed; the log file storage position specification specifies that logs are uniformly stored in a fixed directory in a container, the directory is mounted in the fixed directory on a host, and the host is deployed with filebeat, namely a log collection program is deployed;

s02, mounting the log directory on the host to the filecut container;

s03, injecting the IP address of the host machine into the environment variable of the filebeat container;

s04, using the environment variable in the configuration file of fileteam.

2. The method for ELK-based unified management of application logs in a k8s environment according to claim 1, further comprising a method for deleting logs, wherein the method for deleting logs comprises the steps of:

s05, the log deleting program reads the data file of fileteam;

s06, judging whether the data file is completely read, if so, executing the next step;

s07, judging whether the path contains log-history, if not, executing the next step;

s08, judging whether the container where the log is located is alive or not, if not, executing the next step;

and S09, deleting the log.

3. The ELK-based unified method for managing application logs in a k8s environment according to claim 2,

in the S06, if not, the execution of the log deletion program is terminated;

in the S07, if yes, executing deleting the log;

in S08, if yes, execution of the log deletion program is terminated.

4. The ELK-based unified method for managing application logs in a k8s environment according to claim 1,

in S01, the log files are divided into a file currently being written and a history file generated according to a rolling policy, where the file being written is named as "application name- # business classification # -template identification log", and the history file generated according to the rolling policy is named as "application name-yyy.mm.dd-index- # business classification # -template identification log", where the application name represents the name of an application, the business classification represents the business category to which the log belongs, the template identification represents a regular template that needs to be used when the log is parsed, the yy.mm.dd identifies the current year, month and day, and the index identification index number of the same type of log file generated according to the rolling policy, and the rolling policy is configured to roll by date and size.

5. The ELK-based unified method for managing application logs in a k8s environment according to claim 1,

in S01, mount/u 01/logs/$ { PROJECT _ NAME } on the host to the CONTAINER/app/logs directory, where $ { PROJECT _ NAME } is the NAME of the current application, specify through the yaml file when deploying the CONTAINER, store the file currently being written to/app/logs/$ { contact _ NAME }/directory in the configuration file of the log, store the history file to/app/logs/$ { air _ NAME }/log-history/directory, where $ { contact _ NAME } is the NAME of the current CONTAINER, and inject the current CONTAINER with the context variable when the CONTAINER is started.

6. The ELK-based unified method for managing application logs in a k8s environment according to claim 1,

in S02, mount/01/logs on the host to the app/logs directory of the container for reading all log file contents, and mount/u 01/filebolt/data to the usr/share/filebolt/data directory of the container for saving the current reading position information of each log file on the host.

7. The ELK-based unified method for managing application logs in a k8s environment according to claim 2,

the log deleting program is a python script, the log deleting program runs on a host machine in a DameonSet mode of k8s, a/u 01/fileboat/data of the host machine is mounted in a container to a/fileboat/data directory of the container and a/u 01/logs/to a/app/logs directory of the host machine, data.json file contents in the/fileboat/data directory are read in the python script in a timing mode, and the data.json files comprise paths of all files read by the fileboat and current read position information.

Images