CN117389830A - Cluster log acquisition method and device, computer equipment and storage medium - Google Patents
Cluster log acquisition method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN117389830A CN117389830A CN202311409267.1A CN202311409267A CN117389830A CN 117389830 A CN117389830 A CN 117389830A CN 202311409267 A CN202311409267 A CN 202311409267A CN 117389830 A CN117389830 A CN 117389830A
- Authority
- CN
- China
- Prior art keywords
- log
- configuration
- cluster
- file
- container
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000012544 monitoring process Methods 0.000 claims abstract description 9
- 238000004458 analytical method Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 5
- 238000012360 testing method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000012384 transportation and delivery Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000012550 audit Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000013439 planning Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention relates to the technical field of log monitoring and discloses a cluster log acquisition method, a cluster log acquisition device, computer equipment and a storage medium, wherein the method is used for generating a log acquisition rule according to configuration option information by receiving the configuration option information; analyzing the log acquisition rule to obtain log acquisition configuration parameters, wherein the log acquisition configuration parameters comprise regional parameters, cluster parameters, container parameters and file path parameters; determining a target container node according to the regional parameters, the cluster parameters and the container parameters, and generating a log collector configuration file according to the file path parameters; and running the log collector at the target container node according to the log collector configuration file so as to collect the target log through the log collector. The cluster log acquisition method of the invention saves time and labor for configuring the log acquisition device, reduces the difficulty of managing the cluster log, improves the log acquisition efficiency and improves the user experience.
Description
Technical Field
The present invention relates to the field of log monitoring technologies, and in particular, to a method and apparatus for collecting a cluster log, a computer device, and a storage medium.
Background
Kubernetes clusters, k8s clusters for short, are open source systems for automatically deploying, expanding and managing containerized applications. Financial institutions (e.g., banks) and financial services companies may utilize Kubernetes clusters to manage financial microservice base data, rapidly deploying and expanding services. Digital medical platform companies may utilize Kubernetes clusters to deploy data processing applications to manage large-scale analysis data. The log service is a main link in the k8s cluster work, and the cluster log records detailed access requests and error information, thereby having important roles in aspects of service analysis, problem positioning and the like. The K8s cluster is used as a distributed cluster system and manages a large number of containers, and the log collection is the first step of realizing log service and relates to container logs of each container in the cluster.
Existing k8s cluster systems choose to deploy filebeans on nodes to collect container logs in the cluster. Filebean is a lightweight transport that is installed as a proxy on a server, where specified log files or locations can be monitored, log events collected, and log data forwarded. When the filebox is used for collecting logs, as the filebox cannot acquire the relation between the container log files and the containers, and the log data are scattered and large in data size, business personnel are required to write complex parameter configuration files to manage the logs of each container in the cluster, so that the configuration work of log collection is time-consuming and labor-consuming, and the accuracy and collection efficiency of log collection are further affected.
Disclosure of Invention
Based on the above, it is necessary to provide a method, an apparatus, a computer device and a storage medium for collecting a cluster log, so as to solve the problems of time and effort consuming collection configuration, poor collection accuracy and low collection efficiency during the collection of the cluster log.
A cluster log acquisition method comprises the following steps:
receiving configuration option information, and generating a log acquisition rule according to the configuration option information;
analyzing the log acquisition rule to obtain log acquisition configuration parameters, wherein the log acquisition configuration parameters comprise regional parameters, cluster parameters, container parameters and file path parameters;
determining a target container node according to the region parameter, the cluster parameter and the container parameter, and generating a log collector configuration file according to the file path parameter;
and running a log collector on the target container node according to the log collector configuration file so as to collect a target log through the log collector.
A cluster log collection device, comprising:
the acquisition rule generation module is used for receiving configuration option information and generating a log acquisition rule according to the configuration option information;
the acquisition configuration parameter acquisition module is used for analyzing the log acquisition rule to acquire log acquisition configuration parameters, wherein the log acquisition configuration parameters comprise regional parameters, cluster parameters, container parameters and file path parameters;
the collector configuration file generation module is used for determining a target container node according to the region parameter, the cluster parameter and the container parameter and generating a log collector configuration file according to the file path parameter;
and the log collector running module is used for running the log collector on the target container node according to the log collector configuration file so as to collect the target log through the log collector.
A computer device comprising a memory, a processor and computer readable instructions stored in the memory and executable on the processor, the processor implementing the cluster log collection method described above when executing the computer readable instructions.
A computer-readable storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform a cluster log collection method as described above.
In the cluster log acquisition method, the cluster log acquisition device, the computer equipment and the storage medium, the method generates a log acquisition rule according to configuration option information by receiving the configuration option information; analyzing the log acquisition rule to obtain log acquisition configuration parameters, wherein the log acquisition configuration parameters comprise regional parameters, cluster parameters, container parameters and file path parameters; determining a target container node according to the regional parameters, the cluster parameters and the container parameters, and generating a log collector configuration file according to the file path parameters; and running the log collector at the target container node according to the log collector configuration file so as to collect the target log through the log collector. The cluster log acquisition method realizes the configuration of the log acquisition parameters based on the options of the visual page, and forms the log acquisition rule, so that the configuration is time-saving and labor-saving, and the difficulty in managing the cluster log is reduced. Meanwhile, the log collector can be operated more efficiently and accurately based on the log collection rule, so that the log collection efficiency is improved, and the user experience is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments of the present invention will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of an application environment of a cluster log collection method according to an embodiment of the invention;
FIG. 2 is a flow chart of a method for collecting cluster logs according to an embodiment of the invention;
FIG. 3 is a schematic diagram of a log collection device according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a computer device in accordance with an embodiment of the invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The cluster log collection method provided by the embodiment can be applied to an application environment as shown in fig. 1, wherein a client communicates with a server. Clients include, but are not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The server may be an independent server, or may be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.
The cluster log collection method provided by the embodiment is applied to a Kubernetes cluster system, and the Kubernetes cluster provides a mechanism for application deployment, planning, updating and maintenance on a plurality of hosts in a cloud platform. Kubernetes cluster is a distributed system that facilitates deployment and management of micro-service architecture applications by forming an abstraction layer over the cluster. Financial institutions (e.g., banks) and financial services companies may utilize Kubernetes clusters to manage micro-service base data, including applications, databases, message queues, etc., to rapidly deploy and expand services so as to be able to guarantee high availability and reliability of financial services. In addition, the digital medical platform company can utilize the Kubernetes cluster to rapidly deploy the data processing application program and manage large-scale analysis result data so as to improve the data analysis efficiency and accuracy of the digital medical service.
In an embodiment, as shown in fig. 2, a cluster log collection method is provided, and the method is applied to the server in fig. 1 for illustration, and includes the following steps S10-S40.
S10, receiving configuration option information, and generating a log acquisition rule according to the configuration option information.
Understandably, kubernetes cluster is a distributed system involving log services for multiple applications. Under different business scenarios, users may generate different log collection requirements. The configuration option information is option information related to log collector settings input by a user through a client according to log collection requirements, and comprises a plurality of levels, such as regional option information at regional level, cluster option information at cluster level, container option information at container level and file path option information at file level. The client sends the configuration option information to the server, and the server generates a log acquisition rule according to the configuration option information after receiving the configuration option information. The log collection rule is a series of rule parameters corresponding to configuration option information of each level, and is used for setting the log collector to meet the log collection requirement of a user.
S20, analyzing the log acquisition rule to obtain log acquisition configuration parameters, wherein the log acquisition configuration parameters comprise region parameters, cluster parameters, container parameters and file path parameters.
It is understood that in the Kubernetes cluster system, an area refers to a geographical location area where a physical data center is located, and an area may include a plurality of clusters. In order to achieve the highest access performance, the cluster needs to be located in the same region as the cloud server instance, for example, when the data center of the Kubernetes cluster is set in the guangdong, the corresponding region is the south-china region. A cluster is a collection of nodes that is used to deploy and manage an application. Clusters are typically made up of one or more master nodes and a plurality of working nodes, providing high availability and fault tolerance through cooperation between the various nodes. In Kubernetes clusters, pod is a logical host of containers, providing a network and storage environment for the containers. A Pod may contain one or more containers that may share network and storage resources, or may communicate with each other. Containers are the basic unit of operation that runs in Pod, and containers are created and managed by a container engine such as a Docker. A node is a working node in a Kubernetes cluster, typically a virtual or physical machine. Each node runs a container engine (e.g., docker) for one container for creating and managing containers in the Pod. The file path is the position information under the file directory corresponding to the working node and is used for positioning the file under the file directory. The server analyzes the log acquisition rules to obtain log acquisition configuration parameters of each level, wherein the log acquisition configuration parameters comprise regional parameters of regional level, cluster parameters of cluster level, container parameters of container level and file path parameters of file level.
S30, determining a target container node according to the region parameter, the cluster parameter and the container parameter, and generating a log collector configuration file according to the file path parameter.
Understandably, the log collector is a tool for collecting and transferring log files, and in Kubernetes cluster, the log collector is referred to as filebean. Filebean is installed on a server as a log agent, monitors log files or locations under specified file paths, collects log events, and forwards them to an elastic search or logstack for indexing. And the server determines the container node where the corresponding container is located under the corresponding cluster of the corresponding region according to the region parameter, the cluster parameter and the container parameter which are obtained through analysis, and takes the container node as a target container node. The server side also determines the file position under the corresponding container node according to the file path parameters obtained by analysis, generates a file configuration file of the corresponding log path, and uploads the file configuration file to the file configuration directory of the container node where the file configuration file is located. In this embodiment, the server analyzes the log collection rule, the area parameter obtained by analysis is "south China", the cluster parameter is "test", the container parameter is "nginx", 2 nodes where nginx containers are located in a test cluster where a query database finds out the south China area are node a and node B respectively, a filebat configuration file is generated according to file path parameters "/data/logs/access.log", and the filebat configuration file is uploaded to a filebat configuration catalog "/etc/filebat/config/" of node a and node B.
And S40, running a log collector on the target container node according to the log collector configuration file so as to collect a target log through the log collector.
Understandably, when the server invokes the filebean configuration file at the target container node, the filebean is activated and runs, and the filebean initiates one or more inputs that will find log file data at the specified file path location. For each log file found by the filebox, the filebox will start a collector, each collector reads a single log and sends the log data to the data forwarding library libband, which aggregates the log data and sends the aggregated log data to a filebox preconfigured output, such as a log service platform.
The embodiment generates a log acquisition rule according to configuration option information by receiving the configuration option information; analyzing the log acquisition rule to obtain log acquisition configuration parameters, wherein the log acquisition configuration parameters comprise regional parameters, cluster parameters, container parameters and file path parameters; determining a target container node according to the regional parameters, the cluster parameters and the container parameters, and generating a log collector configuration file according to the file path parameters; and running the log collector at the target container node according to the log collector configuration file so as to collect the target log through the log collector. The cluster log acquisition method of the embodiment realizes the configuration of the log acquisition parameters based on the options of the visual page, and forms the log acquisition rule, so that the configuration is time-saving and labor-saving, and the difficulty in managing the cluster log is reduced. Meanwhile, the log collector can be operated more efficiently and accurately based on the log collection rule, so that the log collection efficiency is improved, and the user experience is improved.
In one embodiment, in step S10, that is, the receiving configuration option information, generating a log collection rule according to the configuration option information includes:
s101, receiving a log acquisition configuration request, and displaying an acquisition rule configuration page according to the log acquisition configuration request;
s102, receiving configuration selection instructions through the acquisition rule configuration page, wherein the configuration selection instructions comprise an area selection instruction, a cluster selection instruction, a container selection instruction and a file path selection instruction;
s103, determining regional option information according to the regional selection instruction, determining cluster option information according to the cluster selection instruction, determining container option information according to the container selection instruction, and determining file path option information according to the file path selection instruction;
s104, generating the log acquisition rule according to the region option information, the cluster option information, the container option information and the file path option information.
Understandably, the log acquisition configuration request is request information for configuring the log acquisition device triggered by the user through the page of the client. The server receives the log acquisition configuration request, interacts with the client according to the log acquisition configuration request, and displays an acquisition rule configuration page on the client. The acquisition rule configuration page is used for displaying all the Kubernetes clusters in the current Kubernetes cluster system, the area to which each cluster belongs, the containers under each cluster and the file paths of the file catalogs in each container node according to different levels, a user can select the acquisition rule configuration page of the client according to the log acquisition requirement, and the client sends corresponding area selection instructions, cluster selection instructions, container selection instructions and file path selection instructions to the server. The server determines regional option information according to the regional selection instruction, cluster option information according to the cluster selection instruction, container option information according to the container selection instruction, and file path option information according to the file path selection instruction. And the server merges the region option information, the cluster option information, the container option information and the file path option information to generate a log acquisition rule. For example: the user carries out selection operation through the acquisition rule configuration page, the region option information corresponding to the region selection instruction is 'south China', the region option information corresponding to the region selection instruction is 'test', the container option information corresponding to the container selection instruction is 'nginx', the file path option information corresponding to the file path selection instruction is 'data/logs/access.log', and after the user finishes selection, the user clicks a creation button to generate a log acquisition rule named as 'spring-boot'. Where "/data/logs/access. Log" means that the open access log function is used to specify the storage path, format, and cache size of the log file.
When configuring the log collector, the user does not need to completely master the underlying mechanism of the log collector, learn and write complex log collector configuration files, familiarize with mechanisms such as cluster log mounting and the like, but can clearly see configuration option information of each level through a visual web page, and determine corresponding options and generate log collection rules in a mode of clicking by the user based on log collection requirements. The embodiment realizes more efficient and accurate configuration of the log collector, greatly reduces the difficulty of managing the cluster log by the user, saves the time of the user and improves the user experience.
In an embodiment, the collection rule configuration page includes an area cluster configuration page, a container configuration page, and a file path configuration page;
in step S102, the receiving, by the acquisition rule configuration page, a configuration selection instruction includes:
s1021, receiving an area selection instruction and a cluster selection instruction through the area cluster configuration page, determining a container to be selected according to the area selection instruction and the cluster selection instruction, and generating a container configuration page corresponding to the container to be selected;
s1022, receiving a container selection instruction through the container configuration page, determining a file path to be selected according to the container selection instruction, and generating a file path configuration page corresponding to the file path to be selected;
s1023, receiving a file path selection instruction through the file path configuration page.
Understandably, the collection rule configuration page includes an area cluster configuration page, a container configuration page, and a file path configuration page, which correspond to an area level and a cluster level configuration option, a container level configuration option, and a file level configuration option, respectively. After receiving the log acquisition configuration request, the acquisition rule configuration page displayed first is an area cluster configuration page. The client side sends an area selection instruction and a cluster selection instruction to the server side based on the selection operation of the user on the area cluster configuration page, and the server side determines all containers to be selected under the corresponding clusters of the corresponding areas according to the area selection instruction and the cluster selection instruction so as to generate a container configuration page corresponding to the containers to be selected at the client side, wherein the container configuration page is used for the user to select the designated containers as the log acquisition range. The client side sends a container selection instruction to the server side based on the selection operation of the user on the container configuration page, and the server side determines all file paths to be selected under the container node corresponding to the container according to the container selection instruction so as to generate a file path configuration page corresponding to the file paths to be selected at the client side, wherein the file path configuration page is used for the user to select a specified file path, for example, an access log can specify a log file path, a cache size and a log file name. The client side sends a file path selection instruction to the server side based on the selection operation of the user on the file path configuration page, and the server side receives the file path selection instruction.
According to the embodiment, according to the architecture relation among the layers, the to-be-selected options of the next layer are determined through the selection instruction of the previous layer, the number of the to-be-selected options is reduced, the phenomenon of false selection caused by mismatching of the previous layer and the next layer is avoided, and the log acquisition rule is conveniently and rapidly established.
In an embodiment, in step S30, the determining the target container node according to the area parameter, the cluster parameter, and the container parameter includes:
s301, determining a target container query condition according to the region parameter, the cluster parameter and the container parameter;
s302, inquiring a database according to the target container inquiring condition, and determining a target container node corresponding to the target container inquiring condition.
Understandably, the target container query condition is used to query the database for containers corresponding to the log collection requirements, including the region parameters, the cluster parameters, and the container parameters. The area parameter, the cluster parameter, and the container parameter may be an area name, a cluster name, and a container name. The target container node is a node where a container corresponding to the target container query condition is located, and the node information where each container in each Kubernetes cluster is located, namely the association relationship between the container and the node, is stored in the database. Since there are multiple working nodes in the Kubernetes cluster, and each node runs a container engine (e.g., docker) of one container, it is necessary to query the database according to the target container query condition, and determine the target container node corresponding to the target container query condition. In an embodiment, the area name obtained by the server end through analysis is "south China", the cluster name is "test", the container name is "ngix", the ngix containers in the test cluster of the area of south China found by the query database are distributed in 2 nodes, and the corresponding 2 nodes A and B are target container nodes.
According to the method and the device, the target container query condition is determined according to the regional parameters, the cluster parameters and the container parameters, and the database is queried to determine the corresponding target container node, so that the node where the container is located can be accurately determined, and time is saved.
In one embodiment, in step S10, the generating a log collector configuration file according to the file path parameter includes:
s303, acquiring an initial configuration file of a log collector;
s304, updating the initial configuration file of the log collector according to the file path parameters to generate a configuration file of the log collector;
s305, uploading the log collector configuration file to a log collector configuration catalog of the target container node so as to run the log collector on the target container node.
It will be appreciated that the log collector initial configuration file contains general log collector configuration information, such as basic function information and default setting information, but does not contain actual application setting information, such as file path information. The server side obtains the initial configuration file of the log collector, adds the file path parameters to the initial configuration file of the log collector for updating, and generates the configuration file of the log collector. And a log collector configuration catalog is created at the node of each container and is used for deploying log collectors at the container nodes. After the log collector configuration file is generated, the server side uploads the log collector configuration file to a log collector configuration catalog of the target container node based on the determined target container node.
According to the embodiment, the file path parameters are added to the initial configuration file of the log collector to update, so that the configuration file of the log collector is quickly generated, and the configuration efficiency of the log collector is improved.
In one embodiment, in step S40, the collecting, by the log collector, the target log includes:
s401, inquiring a file catalog of the target container node through the log collector, and determining a target path file corresponding to the file path parameter;
s402, acquiring a target log in the target path file through the log collector, and returning the target log to a log service platform.
It will be appreciated that, to facilitate access and management of files, the Kubernetes cluster system builds an index of files, i.e., file directories, which represent the mapping between file names and the physical locations of the files. The file directory establishes an entry for each file, and the entry of the file directory at least comprises a file name, an internal identifier of the file, a type of the file, a file storage address, a length of the file, an access right, an establishment time, an access time and the like. The target path file is file data under a specified file path corresponding to the file path parameter in the target container node, and the target log is log file data in the target path file. The server side queries the file catalogue of the target container node through the log collector, determines a target path file, collects a target log in the target path file, and returns the target log to the log service platform. The log service platform assists users to solve scene problems such as business operation and maintenance, service monitoring and log audit through logs by log retrieval, chart analysis, monitoring alarm, log delivery and the like. In practical application, the appointed log data scattered at each node of the cluster is collected to the log service platform for unified management, and logs of abnormal events can be quickly searched out through keyword retrieval, problem nodes are positioned, and alarming is carried out.
According to the embodiment, the log collector is used for inquiring the file catalogue of the target container node, all current target logs in the target path file are obtained, the current target logs are returned to the log service platform, and the accuracy and the completeness of log collection are guaranteed.
In an embodiment, in step S402, after the target log in the target path file is acquired by the log collector, the method further includes:
s4021, monitoring the target path file through the log collector according to a preset time period, and judging whether a new target log exists in the target path file;
s4022, if the newly-increased target log exists, acquiring the newly-increased target log through the log collector, and returning the newly-increased target log to the log service platform.
It is understood that the preset time period is a preset monitoring period time period of the log collector, for example, the default setting of the system is 1 minute, the preset time period can be modified to be 30 seconds according to the log collection requirement, and the preset time period can be continuous. The newly added target log is newly generated log file data in the target path file in the current time period. In this embodiment, the server monitors and scans the target path file according to the frequency of 1 minute each time through the log collector, determines whether a new target log exists in the target path file according to the scanning result, and when the new target log exists, collects the new target log and returns to the log service platform.
According to the embodiment, the log collector continuously monitors the target path file according to the preset time period, grasps the log updating dynamic state, collects newly-added log file data in time, and ensures the timeliness of log collection.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present invention.
In an embodiment, a cluster log collection device is provided, where the cluster log collection device corresponds to the cluster log collection method in the above embodiment one by one. As shown in fig. 3, the cluster log collection device includes a collection rule generation module 10, a collection configuration parameter acquisition module 20, a collector configuration file generation module 30, and a log collector operation module 40. The functional modules are described in detail as follows:
the acquisition rule generation module 10 is used for receiving configuration option information and generating a log acquisition rule according to the configuration option information;
the acquisition configuration parameter acquiring module 20 is configured to parse the log acquisition rule to obtain log acquisition configuration parameters, where the log acquisition configuration parameters include a regional parameter, a cluster parameter, a container parameter and a file path parameter;
the collector profile generation module 30 is configured to determine a target container node according to the region parameter, the cluster parameter and the container parameter, and generate a log collector profile according to the file path parameter;
and the log collector running module 40 is configured to run the log collector at the target container node according to the log collector configuration file, so as to collect the target log through the log collector.
In one embodiment, the acquisition rule generation module 10 includes:
the acquisition rule configuration page display unit is used for receiving a log acquisition configuration request and displaying an acquisition rule configuration page according to the log acquisition configuration request;
the configuration selection instruction receiving unit is used for receiving a configuration selection instruction through the acquisition rule configuration page, wherein the configuration selection instruction comprises a region selection instruction, a cluster selection instruction, a container selection instruction and a file path selection instruction;
the option information determining unit is used for determining area option information according to the area selection instruction, determining cluster option information according to the cluster selection instruction, determining container option information according to the container selection instruction and determining file path option information according to the file path selection instruction;
and the log acquisition rule generating unit is used for generating the log acquisition rule according to the region option information, the cluster option information, the container option information and the file path option information.
In an embodiment, the acquisition rule generation module 10 further comprises:
the container configuration page generation unit is used for receiving an area selection instruction and a cluster selection instruction through the area cluster configuration page, determining a container to be selected according to the area selection instruction and the cluster selection instruction, and generating a container configuration page corresponding to the container to be selected;
the file path configuration page generation unit is used for receiving a container selection instruction through the container configuration page, determining a file path to be selected according to the container selection instruction, and generating a file path configuration page corresponding to the file path to be selected;
and the file path selection instruction receiving unit is used for receiving the file path selection instruction through the file path configuration page.
In one embodiment, collector profile generation module 30 includes:
the target container query condition determining unit is used for determining the target container query condition according to the region parameter, the cluster parameter and the container parameter;
and the target container node determining unit is used for querying the database according to the target container query condition and determining a target container node corresponding to the target container query condition.
In one embodiment, collector profile generation module 30 further includes:
the initial configuration file acquisition unit is used for acquiring an initial configuration file of the log collector;
the collector configuration file generating unit is used for updating the initial configuration file of the log collector according to the file path parameters to generate a log collector configuration file;
and the collector configuration file uploading unit is used for uploading the log collector configuration file to the log collector configuration catalog of the target container node so as to run the log collector on the target container node.
In one embodiment, log collector execution module 40 includes:
the target path file determining unit is used for inquiring the file catalogue of the target container node through the log collector and determining a target path file corresponding to the file path parameter;
and the target log acquisition unit is used for acquiring a target log in the target path file through the log collector and returning the target log to the log service platform.
In one embodiment, log collector execution module 40 further comprises:
the target path file monitoring unit is used for monitoring the target path file according to a preset time period through the log collector and judging whether a newly added target log exists in the target path file;
and the newly-added target log acquisition unit is used for acquiring the newly-added target log through the log collector and returning the newly-added target log to the log service platform if the newly-added target log exists.
The specific limitation of the cluster log collection device can be referred to the limitation of the cluster log collection method hereinabove, and will not be described herein. The modules in the cluster log collection device can be all or partially realized by software, hardware and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a terminal, and the internal structure of which may be as shown in fig. 4. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a readable storage medium, an internal memory. The non-volatile storage medium stores an operating system and computer readable instructions. The internal memory provides an environment for the execution of an operating system and computer-readable instructions in a readable storage medium. The network interface of the computer device is for communicating with an external server via a network connection. The computer readable instructions, when executed by a processor, implement a cluster log collection method. The readable storage medium provided by the present embodiment includes a nonvolatile readable storage medium and a volatile readable storage medium.
In one embodiment, a computer device is provided that includes a memory, a processor, and computer readable instructions stored on the memory and executable on the processor, when executing the computer readable instructions, performing the steps of:
receiving configuration option information, and generating a log acquisition rule according to the configuration option information;
analyzing the log acquisition rule to obtain log acquisition configuration parameters, wherein the log acquisition configuration parameters comprise regional parameters, cluster parameters, container parameters and file path parameters;
determining a target container node according to the region parameter, the cluster parameter and the container parameter, and generating a log collector configuration file according to the file path parameter;
and running a log collector on the target container node according to the log collector configuration file so as to collect a target log through the log collector.
In one embodiment, one or more computer-readable storage media are provided having computer-readable instructions stored thereon, the readable storage media provided by the present embodiment including non-volatile readable storage media and volatile readable storage media. The readable storage medium has stored thereon computer readable instructions which when executed by one or more processors perform the steps of:
receiving configuration option information, and generating a log acquisition rule according to the configuration option information;
analyzing the log acquisition rule to obtain log acquisition configuration parameters, wherein the log acquisition configuration parameters comprise regional parameters, cluster parameters, container parameters and file path parameters;
determining a target container node according to the region parameter, the cluster parameter and the container parameter, and generating a log collector configuration file according to the file path parameter;
and running a log collector on the target container node according to the log collector configuration file so as to collect a target log through the log collector.
Those skilled in the art will appreciate that implementing all or part of the above described embodiment methods may be accomplished by instructing the associated hardware by computer readable instructions stored on a non-volatile readable storage medium or a volatile readable storage medium, which when executed may comprise the above described embodiment methods. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions.
The above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention, and are intended to be included in the scope of the present invention.
Claims (10)
1. The cluster log acquisition method is characterized by comprising the following steps of:
receiving configuration option information, and generating a log acquisition rule according to the configuration option information;
analyzing the log acquisition rule to obtain log acquisition configuration parameters, wherein the log acquisition configuration parameters comprise regional parameters, cluster parameters, container parameters and file path parameters;
determining a target container node according to the region parameter, the cluster parameter and the container parameter, and generating a log collector configuration file according to the file path parameter;
and running a log collector on the target container node according to the log collector configuration file so as to collect a target log through the log collector.
2. The method of claim 1, wherein the receiving configuration option information and generating log collection rules according to the configuration option information comprises:
receiving a log acquisition configuration request, and displaying an acquisition rule configuration page according to the log acquisition configuration request;
receiving a configuration selection instruction through the acquisition rule configuration page, wherein the configuration selection instruction comprises a region selection instruction, a cluster selection instruction, a container selection instruction and a file path selection instruction;
determining regional option information according to the regional selection instruction, determining cluster option information according to the cluster selection instruction, determining container option information according to the container selection instruction, and determining file path option information according to the file path selection instruction;
and generating the log acquisition rule according to the region option information, the cluster option information, the container option information and the file path option information.
3. The cluster log collection method according to claim 2, wherein the collection rule configuration page includes a regional cluster configuration page, a container configuration page, and a file path configuration page;
the receiving, by the acquisition rule configuration page, a configuration selection instruction includes:
receiving an area selection instruction and a cluster selection instruction through the area cluster configuration page, determining a container to be selected according to the area selection instruction and the cluster selection instruction, and generating a container configuration page corresponding to the container to be selected;
receiving a container selection instruction through the container configuration page, determining a file path to be selected according to the container selection instruction, and generating a file path configuration page corresponding to the file path to be selected;
and receiving a file path selection instruction through the file path configuration page.
4. The cluster log collection method according to claim 1, wherein the determining a target container node according to the region parameter, the cluster parameter, and the container parameter comprises:
determining target container query conditions according to the region parameters, the cluster parameters and the container parameters;
and inquiring the database according to the target container inquiring condition, and determining a target container node corresponding to the target container inquiring condition.
5. The cluster log collection method according to claim 1, wherein the generating a log collector profile according to the file path parameters comprises:
acquiring an initial configuration file of a log collector;
updating the initial configuration file of the log collector according to the file path parameters to generate a configuration file of the log collector;
and uploading the log collector configuration file to a log collector configuration catalog of the target container node to run the log collector at the target container node.
6. The cluster log collection method according to claim 1, wherein the collecting, by the log collector, the target log includes:
inquiring a file catalog of the target container node through the log collector, and determining a target path file corresponding to the file path parameter;
and acquiring a target log in the target path file through the log collector, and returning the target log to a log service platform.
7. The method of collecting cluster logs according to claim 6, wherein after the target logs in the target path file are obtained by the log collector, the method further comprises:
monitoring the target path file according to a preset time period by the log collector, and judging whether a newly added target log exists in the target path file;
if the newly-increased target log exists, the newly-increased target log is acquired through the log collector, and the newly-increased target log is returned to the log service platform.
8. The utility model provides a cluster log collection system which characterized in that includes:
the acquisition rule generation module is used for receiving configuration option information and generating a log acquisition rule according to the configuration option information;
the acquisition configuration parameter acquisition module is used for analyzing the log acquisition rule to acquire log acquisition configuration parameters, wherein the log acquisition configuration parameters comprise regional parameters, cluster parameters, container parameters and file path parameters;
the collector configuration file generation module is used for determining a target container node according to the region parameter, the cluster parameter and the container parameter and generating a log collector configuration file according to the file path parameter;
and the log collector running module is used for running the log collector on the target container node according to the log collector configuration file so as to collect the target log through the log collector.
9. A computer device comprising a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, wherein the processor, when executing the computer readable instructions, implements the cluster log collection method of any one of claims 1 to 7.
10. A computer-readable storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the cluster log collection method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311409267.1A CN117389830A (en) | 2023-10-26 | 2023-10-26 | Cluster log acquisition method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311409267.1A CN117389830A (en) | 2023-10-26 | 2023-10-26 | Cluster log acquisition method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117389830A true CN117389830A (en) | 2024-01-12 |
Family
ID=89437081
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311409267.1A Pending CN117389830A (en) | 2023-10-26 | 2023-10-26 | Cluster log acquisition method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117389830A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117687892A (en) * | 2024-02-04 | 2024-03-12 | 麒麟软件有限公司 | Comprehensive log acquisition method and device suitable for embedded system and storage medium |
-
2023
- 2023-10-26 CN CN202311409267.1A patent/CN117389830A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117687892A (en) * | 2024-02-04 | 2024-03-12 | 麒麟软件有限公司 | Comprehensive log acquisition method and device suitable for embedded system and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11768811B1 (en) | Managing user data in a multitenant deployment | |
| US11757720B2 (en) | Distributed computing dependency management system | |
| CN112910945B (en) | Request link tracking method and service request processing method | |
| TWI728036B (en) | Information processing method, device and system | |
| US9489443B1 (en) | Scheduling of splits and moves of database partitions | |
| US20070266029A1 (en) | Recovery segment identification in a computing infrastructure | |
| US11436116B1 (en) | Recovering pre-indexed data from a shared storage system following a failed indexer | |
| CN109933338B (en) | Block chain deployment method, device, computer equipment and storage medium | |
| CN113037891B (en) | Access method and device for stateful application in edge computing system and electronic equipment | |
| CN112364110A (en) | Metadata management method, device and equipment and computer storage medium | |
| CN114791846B (en) | Method for realizing observability aiming at cloud-originated chaos engineering experiment | |
| CN117389830A (en) | Cluster log acquisition method and device, computer equipment and storage medium | |
| CN113127526A (en) | Distributed data storage and retrieval system based on Kubernetes | |
| CN111274004B (en) | Process instance management method and device and computer storage medium | |
| CN112187509A (en) | Multi-architecture cloud platform execution log management method, system, terminal and storage medium | |
| CN111711695A (en) | Distributed equipment management system based on cloud platform | |
| EP3306471B1 (en) | Automatic server cluster discovery | |
| Singh | Cluster-level logging of containers with containers: Logging challenges of container-based cloud deployments | |
| US20210377718A1 (en) | Pattern affinity for discovery | |
| CN114154825A (en) | Two-dimensional power grid distributed cache service system | |
| US11500874B2 (en) | Systems and methods for linking metric data to resources | |
| CN109684158A (en) | Method for monitoring state, device, equipment and the storage medium of distributed coordination system | |
| CN110896358A (en) | Dynamic updating method and device for resource data in NFV | |
| Sun et al. | RS-pCloud: a peer-to-peer based edge-cloud system for fast remote sensing image processing | |
| CN115348185B (en) | Control method and control device of distributed query engine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |