CN114625594A - Configuration file generation method, log collection method, device, equipment and medium - Google Patents
Configuration file generation method, log collection method, device, equipment and medium Download PDFInfo
- Publication number
- CN114625594A CN114625594A CN202011458909.3A CN202011458909A CN114625594A CN 114625594 A CN114625594 A CN 114625594A CN 202011458909 A CN202011458909 A CN 202011458909A CN 114625594 A CN114625594 A CN 114625594A
- Authority
- CN
- China
- Prior art keywords
- log
- new container
- log collection
- configuration file
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the disclosure relates to a configuration file generation method, a log collection method, a device, equipment and a medium, wherein the method comprises the following steps: monitoring whether a new container is created on the computing device; if a new container is monitored to be generated on the computing equipment, generating a log collection configuration file for the new container based on the log collection configuration file template and a deployment file of the new container; and the log collection configuration file is used for collecting logs generated in the operation process of the new container. According to the embodiment of the invention, the log collection configuration file is automatically generated for the new container, so that the dynamic monitoring of the new container is realized, and the efficiency of log collection is improved.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a method for generating a configuration file, a method for acquiring a log, an apparatus, a device, and a medium.
Background
Edge computing is a computing service deployed at an edge node. The edge computing uses a container technology as a resource carrier, can create computing and network resources at any time according to business requirements without investing equipment purchasing cost in advance, and is a more efficient implementation mode than self-building or renting a physical machine or a virtual server.
Typically, several edge nodes are included in an edge computing system. According to the hardware resource level of each edge node, a plurality of service containers can be deployed on each edge node. The service container will generate running logs continuously in the process of executing specific service logic, and the running logs play an important role in service running state monitoring, troubleshooting and flow statistics.
However, in the existing scheme, in the process of collecting the running log of each container by using a log collector (or called a log collection component), a technician is required to manually add log collection for a newly generated container, which results in low log collection efficiency.
Disclosure of Invention
In order to solve the technical problem or at least partially solve the technical problem, embodiments of the present disclosure provide a configuration file generation method, a log collection method, an apparatus, a device, and a medium.
In a first aspect, an embodiment of the present disclosure provides a method for generating a log collection configuration file, including:
monitoring whether a new container is created on the computing device;
if a new container is monitored to be generated on the computing equipment, generating a log collection configuration file for the new container based on a log collection configuration file template and a deployment file of the new container;
and the log collection configuration file is used for collecting logs generated in the operation process of the new container.
In a second aspect, an embodiment of the present disclosure further provides a log collection method, which is implemented based on a log collection configuration file, where the log collection configuration file is obtained by using any log collection configuration file generation method provided in the embodiment of the present disclosure, and the log collection method includes:
acquiring a log acquisition configuration file of a new container on computing equipment;
and collecting the logs generated in the operation process of the new container based on the log collection configuration file.
In a third aspect, an embodiment of the present disclosure further provides a log collection configuration file generating apparatus, including:
a new container monitoring module to monitor whether a new container is generated on the computing device;
the configuration file generation module is used for generating a log collection configuration file for a new container based on a log collection configuration file template and a deployment file of the new container if the new container is monitored to be generated on the computing equipment;
and the log collection configuration file is used for collecting logs generated in the operation process of the new container.
In a fourth aspect, an embodiment of the present disclosure further provides a log collecting device, which is implemented based on a log collecting configuration file, where the log collecting configuration file is obtained by using any log collecting configuration file generating method provided in the embodiment of the present disclosure, and the log collecting device includes:
the configuration file acquisition module is used for acquiring a log acquisition configuration file of a new container on the computing equipment;
and the log acquisition module is used for acquiring logs generated in the operation process of the new container based on the log acquisition configuration file.
In a fifth aspect, an embodiment of the present disclosure further provides a computing device, where the computing device includes: a processor; a memory for storing the processor-executable instructions or programs; the processor is configured to read the executable instruction or the program from the memory, and execute the executable instruction or the program to implement any log collection configuration file generation method provided by the embodiment of the present disclosure, or to implement any log collection method provided by the embodiment of the present disclosure.
In a sixth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, where the storage medium stores a computer program or an instruction, and the computer program or the instruction, when executed by a processor, implements any log collection configuration file generation method provided in the embodiment of the present disclosure, or implements any log collection method provided in the embodiment of the present disclosure.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages: according to the embodiment of the invention, by monitoring the generation of the new container and automatically generating the log collection configuration file for the new container, the log collection configuration file is used for collecting the log generated in the operation process of the new container, so that the log collection operation aiming at the new container by manual addition is omitted, the problem of low log collection efficiency in the existing scheme is solved, the dynamic monitoring on the new container is realized, and the log collection efficiency is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present disclosure, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a flowchart of a method for generating a log collection configuration file according to an embodiment of the present disclosure;
fig. 2 is a flowchart of another log collection configuration file generation method according to an embodiment of the present disclosure;
fig. 3 is a flowchart of a log collection method according to an embodiment of the present disclosure;
fig. 4 is a flowchart of another log collection method provided by the embodiment of the present disclosure;
fig. 5 is a schematic flow chart of log collection and distribution provided in the embodiment of the present disclosure;
fig. 6 is a schematic structural diagram of a log collection configuration file generation apparatus according to an embodiment of the present disclosure;
fig. 7 is a schematic structural diagram of a log collection device according to an embodiment of the present disclosure
Fig. 8 is a schematic structural diagram of a computing device according to an embodiment of the present disclosure.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, aspects of the present disclosure will be further described below. It should be noted that the embodiments and features of the embodiments of the present disclosure may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced in other ways than those described herein; it is to be understood that the embodiments disclosed in the specification are only a few embodiments of the present disclosure, and not all embodiments.
Fig. 1 is a flowchart of a method for generating a log collection configuration file according to an embodiment of the present disclosure, which may be applied to how to conveniently collect a log generated in an operation process of a container deployed on a computing device. The log collection configuration file generation method can be executed by a log collection configuration file generation device, the device can be realized by software and/or hardware and can be integrated on any computing equipment, and the device can be realized by adopting a container monitoring component, such as a log-pilot (an intelligent container log collection tool) container monitoring component. The computing devices may be computing nodes in an edge computing system.
As shown in fig. 1, a method for generating a log collection configuration file according to an embodiment of the present disclosure may include:
s101, monitoring whether a new container is generated on the computing equipment.
In particular, a container running on the computing device may be monitored by using a preset container monitoring program, and the monitoring information about the container may include, but is not limited to: the number of containers currently running, whether the running state of each container is normal, container identification (or called container ID), IP address and port number. Wherein the container identification, IP address or port number may be used to uniquely identify each container and may be stored in the computing device in the form of an information list.
If the number of currently running containers is monitored to increase, or at least one of a new container identification, a new IP address, and a new port number is monitored, then it can be assumed that a new container is created on the computing device, otherwise no new container is created.
S102, if it is monitored that a new container is generated on the computing equipment, generating a log collection configuration file for the new container based on the log collection configuration file template and the deployment file of the new container.
The log collection configuration file template has universality and is used for defining various information required for generating the log collection configuration file, such as but not limited to container identification, identification of a log file to be collected, a storage path of the log file to be collected, log collection speed limit time, log collection speed limit type and the like. The log file identifier to be collected is used for uniquely identifying the log file type to be collected, and may be a log file name, for example, the log file identifier to be collected includes two types: one type of identification is used for marking the container process standard output log file, and the other type of identification is used for marking the log file generated in the container; the log collection speed limit type comprises line (line) speed limit collection or byte (bytes) speed limit collection and the like in the log collection process. According to the log collection requirement, the preset information in the log collection configuration file template can be flexibly adjusted.
When it is monitored that a new container is generated, the required information can be acquired according to the log collection configuration file template, for example, the required information is acquired from the deployment file of the new container, and then the acquired information is added to the log collection configuration file template, so that the log collection configuration file is generated for the new container. The deployment file (or referred to as container configuration file) of the new container refers to a file generated in the process of creating the new container, and includes any preset configuration information related to the new container. The log collection configuration file is used for collecting logs generated in the operation process of the new container, namely, the log collection configuration file is used as a file basis for collecting the logs. The information which cannot be acquired in the log acquisition configuration file template can be set to be absent by default, for example, for a current new container, if the log acquisition speed limit time cannot be acquired, the information in the log acquisition configuration file template can be set to be absent, so that the log acquisition speed limit time does not need to be considered in the log acquisition stage. For different new containers, the information included in the log collection configuration file generated based on the log collection configuration file template and the deployment file for the new container may be different.
According to the embodiment of the invention, by monitoring the generation of the new container and automatically generating the log collection configuration file for the new container, the log collection configuration file is used for collecting the log generated in the operation process of the new container, so that the log collection operation aiming at the new container by manual addition is omitted, the problem of low log collection efficiency in the existing scheme is solved, the dynamic monitoring on the new container is realized, and the log collection efficiency is improved.
On the basis of the above technical solution, optionally, generating a log collection configuration file for the new container based on the log collection configuration file template and the deployment file of the new container includes:
determining whether a log collection label exists in a deployment file of the new container; the log collection label is used for distinguishing a container needing log collection from a container not needing log collection;
and if the log collection label is determined to exist in the deployment file of the new container, namely the log collection is required, generating a log collection configuration file for the new container based on the log collection configuration file template and the deployment file of the new container.
Any preset configuration information related to the new container, which is included in the deployment file of the new container, includes, but is not limited to, a container identifier, a log file identifier to be collected, a storage path of the log file to be collected, a log collection speed limit opportunity, a log collection speed limit type, a log collection tag, and the like. The log collection tag may be implemented by using a character string in any form, and the embodiment of the present disclosure is not particularly limited. Whether the log collection tag is added to the deployment file or not can be determined by a user according to current service requirements, for example, if a tag adding operation triggered by the user is detected in the process of creating a new container, the log collection tag is added to the deployment file. In the process of generating the log collection configuration file for the new container, the required information can be acquired by analyzing the deployment file of the new container.
By confirming whether the deployment file of the new container has the log collection label or not and generating the log collection configuration file for the new container according to the log collection label, the pertinence of log collection can be improved, the redundant operation of generating the log collection configuration file for the new container without log collection is avoided, and the effectiveness of the generated log collection configuration file is improved.
Fig. 2 is a flowchart of another log collection configuration file generation method provided in the embodiment of the present disclosure, which is further optimized and expanded based on the above technical solution, and may be combined with the above optional embodiments. As shown in fig. 2, the method may include:
s201, monitoring whether a new container is generated on the computing equipment.
In the embodiment of the present disclosure, after it is monitored that a new container is generated on the computing device, it may be further detected whether a log collection configuration file exists in a file directory corresponding to the new container, and if it is determined that the log collection configuration file does not exist, the log collection configuration file is generated for the new container, so that when the current new container is a container recovered from a failure, a phenomenon that the log collection configuration file is repeatedly generated for the new container can be avoided under the condition that the log collection configuration file actually already exists in the directory corresponding to the container.
S202, if it is monitored that a new container is generated on the computing equipment, acquiring the identification of the log file to be acquired, the storage path of the log file to be acquired, the log acquisition speed limit time of the new container and the log acquisition speed limit type from the deployment file of the new container.
S203, adding the identification of the log file to be collected, the storage path of the log file to be collected, the log collection speed limit time and the log collection speed limit type of the new container into a log collection configuration file template, and generating a log collection configuration file for the new container.
For example, the required information may be acquired by identifying fields of each piece of information in the deployment file of the new container, and then, the acquired information is added to the log collection configuration file template by identifying information fields in the log collection configuration file template according to a field matching relationship, thereby generating the log collection configuration file.
It should be noted that, according to the log collection requirement, the embodiment of the present disclosure may flexibly determine the information acquired from the deployment file, and the example in fig. 2 should not be understood as a specific limitation to the embodiment of the present disclosure. For example, generating the log collection configuration file for the new container based on the log collection configuration file template and the deployment file of the new container may include: acquiring a log file identifier to be acquired and a storage path of the log file to be acquired from the deployment file of the new container; and adding the identification of the log file to be collected and the storage path of the log file to be collected into a log collection configuration file template, and generating a log collection configuration file for the new container.
According to the technical scheme of the embodiment of the disclosure, the log file identification to be collected and the storage path of the log file to be collected are obtained, so that the demand log related to a new container can be accurately collected in the log collection stage; by acquiring the time of limiting the log acquisition speed and the type of the log acquisition speed, the data transmission quantity in the log acquisition process can be effectively controlled, the occupation of bandwidth resources on computing equipment is reduced, the log acquisition is effectively limited, and the resource consumption on the computing equipment is relatively controllable.
Optionally, before monitoring whether a new container is generated on the computing device, the method for generating a log collection configuration file according to the embodiment of the present disclosure further includes: historical monitoring events are cleared for whether a new container is generated on the computing device. In the embodiment of the present disclosure, whether a new container is generated on the computing device may be monitored in real time or periodically, and before executing a current monitoring event, a historical monitoring event may be cleared first, where the historical monitoring event may refer to a container monitoring event within a preset time that takes a current time as a time endpoint; and then perform the current container monitoring event. By clearing historical monitoring events, repeated monitoring of outdated container generation events can be prevented, container monitoring effectiveness is improved, and resource consumption of computing equipment is reduced.
For example, the following takes monitoring the generation of a new container by using a log-pilot container monitoring component and generating a log collection configuration file for the new container, and taking log collection by using a fileteam (a lightweight data collector) log collection component as an example, an implementation flow of the container monitoring component is exemplarily described, where the log collection configuration file may refer to the fileteam configuration file:
1) and executing a system starting script: configuring an environment variable, clearing a file mount point, creating an initial filebot configuration file, designating the filebot. tpl file as a template of other configurations, and finally starting a log-pilot binary system;
2) run is executed within the main function, thus executing:
2.1 clearing/prospectors.d of the configuration file left;
2.2 starting the filecut process;
2.3 calling dockeClients () to obtain err channel and msg channel of the acquired container event;
2.4 starting the goroutine to execute a for loop and starting to monitor events;
2.5 executing processAllContainers (), the function is that when log-pilot is started, for each existing container, a corresponding configuration is created and sent to filehead:
2.5.1 calling dockerClient to obtain all container lists, and detecting whether the ID exists in each container ID detection configuration directory, if not, creating the container ID;
2.5.2 invoke newContainer to create a new container fileteam configuration:
2.5.2.1 firstly, a variable with logPrefix in the environment variables in the new container configuration is taken out and put into a Label of the container, the specific implementation of the logPrefix can be preset, and the Label with the prefix is used for identifying the path of the log file to be read;
2.5.2.2 calls getLogConfig to find each container label matching logPrefix;
2.5.2.3, calling parsConfig for each container label meeting the requirement, detecting the key part of the label, if carrying additional marks such as tag and target; then determining a value part, if the value part is stdout, reading LogPath of docker client, and creating corresponding file configuration; finally, processing the output file in the container, calling hostDirOf to inquire the file path of the output file configured by the label in the mount point of the container, and creating a configuration return;
2.5.2.4, after obtaining the configuration of each log path of the container, calling a render () function to generate a filebolt configuration file of the container, writing the file configuration file into a corresponding directory, and finally executing the load to reload the filebolt;
2.5.3 executing processSymlink () deletes the soft link of a container that does not already exist;
2.6 waiting on stopChan, after receiving stop command, function returns, and the whole program exits.
When the program exits, no file cleaning operation can be performed, and the log collection configuration files generated for each container can be cleaned only once when the program is started, so that redundant or invalid configuration files are reduced. And the filecut log collection component reads a filecut configuration file generated in advance and executes log collection operation. In addition, the functions of the container monitoring component including the log-pilot are not limited to monitoring generation of a new container and generating a log collection configuration file for the new container, and other monitoring functions may also be performed in the log collection phase, for example, monitoring whether the amount of log data generated by the new container exceeds a log storage threshold, or monitoring whether the operation state of the new container is switched from a normal operation state to an abnormal operation state, and the like.
Fig. 3 is a flowchart of a log collection method provided in an embodiment of the present disclosure, and the embodiment of the present disclosure may be applied to how to conveniently collect a log generated in an operation process of a container deployed on a computing device. The log collection method may be executed by a log collection apparatus, which may be implemented by software and/or hardware, and may be integrated on any computing device, and the apparatus may be implemented in the form of a log collection component, such as a fileteam (a lightweight data collector) log collection component, and the like. The computing devices may be computing nodes in an edge computing system.
The log collection method provided by the embodiment of the present disclosure is implemented based on the log collection configuration file generated by using the log collection configuration file generation method provided by the embodiment of the present disclosure, that is, the log collection method and the log collection configuration file generation method in the embodiment of the present disclosure are executed in a matching manner, and the contents not explained in detail in the following embodiments may refer to the description in the above embodiments.
As shown in fig. 3, the log collecting method provided by the embodiment of the present disclosure may include:
s301, acquiring a log collection configuration file of a new container on the computing equipment.
By monitoring whether a new container is generated on the computing equipment and generating a log acquisition configuration file for the new container, the log acquisition configuration file of the current new container can be obtained according to the corresponding relation between the container and the log acquisition configuration file in the log acquisition stage. For example, the log collection configuration file corresponding to the container ID may be obtained based on the container ID of the new container, and the container ID may be used to uniquely identify each container, so that the obtaining accuracy of the log collection configuration file may be ensured, and the successful implementation of log collection may be further ensured.
S302, collecting logs generated in the operation process of the new container based on the log collection configuration file.
After the log collection configuration file is obtained, the log collection configuration file can be analyzed and identified by analyzing and identifying information included in the file, such as but not limited to the identification of the log file to be collected, the storage path of the log file to be collected, the log collection speed limit time, the log collection speed limit type and the like, and the log generated in the operation process of a new container is collected according to the information. The acquired logs can be used for log analysis, so that a user is helped to control the operation performance of each container and assist in analyzing the operation state of the service and the like.
According to the technical scheme of the embodiment, the log collection configuration file is automatically generated for the new container in advance and is directly used in the log collection stage, so that the log collection operation for the new container by manual addition is omitted, the problem of low log collection efficiency in the existing scheme is solved, and the log collection efficiency is improved.
On the basis of the above technical solution, optionally, in the new container operation process, the log collection method provided in the embodiment of the present disclosure further includes:
monitoring whether the log data volume generated by the new container exceeds a log storage threshold or monitoring whether the running state of the new container is switched from a normal running state to an abnormal running state;
if yes, based on the file reference technology, in the process of calling the operating system function of the computing equipment to clear the log generated by the new container, the log generated by the new container is sent to the target log receiver.
The log storage threshold (sizeLimit) is an upper limit of a log storage space preset on the computing device for each container based on a normal operating state of the container, and a specific value may be determined according to a requirement. By setting the log storage threshold, log data can be prevented from being output without limit in the operation process of the container. If the amount of the log data generated by the new container exceeds the log storage threshold, which means that the log output of the new container is greater than the log acquisition amount in the log acquisition stage, the log data generated in excess cannot be acquired and processed, which does not belong to a normal container operation state, so that in order to reduce the consumption of computing resources on the computing device and effectively manage the container state, the new container needs to be removed from the computing device (referred to as container destruction), and meanwhile, the logs generated in the operation process of the new container are cleaned. Similarly, if the operation state of the new container is directly monitored to be switched from the normal operation state to the abnormal operation state, such as a container failure or an abnormal network connection, the new container also needs to be removed from the computing device, and meanwhile, the log generated in the operation process of the new container is cleaned.
In the embodiment of the present disclosure, the log file acquired in the log acquisition process is the same file as the log file read by the operating system of the computing device, and based on a file reference technology (the implementation principle can refer to the existing implementation principle), even if the operating system of the computing device executes the operation of clearing the log, the log is not really deleted due to the existence of the file reference relationship, so that the log of the new container can still be successfully and completely sent to the target log receiver in the log acquisition stage, and the integrity of the log acquisition for any container is ensured. The target log receiving party can be determined according to business logic, and may be, for example, a log search server, a log analysis tool, a message middleware (or referred to as a message queue), or a database.
After the log is completely sent to the target log receiver, namely the file reference relation is released, the complete deletion operation of the log can be executed again, and the occupation of the storage space of the computing device is released.
Moreover, by utilizing the file reference technology, even if the container fails, the logs generated in the normal operation process of the container cannot be really deleted, when the container recovers from the failure, the logs can be continuously generated, the log collection operation can be continuously executed, and the logs generated in the operation process of any container can be completely collected. It should be noted that, in the embodiment of the present disclosure, a file reference technology may be used to collect logs generated during the operation of a container at any time in the log collection phase.
Fig. 4 is a flowchart of another log collection method provided in the embodiment of the present disclosure, which is further optimized and expanded based on the above technical solution, and may be combined with each of the above optional embodiments. As shown in fig. 4, the method may include:
s401, acquiring a log collection configuration file of a new container on the computing equipment.
S402, reading the identification of the log file to be acquired, the storage path of the log file to be acquired, the time limit of log acquisition and the type of the log acquisition speed limit in the log acquisition configuration file.
And S403, acquiring logs corresponding to the identification of the log file to be acquired, which are generated in the operation process of the new container, according to the storage path of the log file to be acquired, the log acquisition speed limit time and the acquisition speed corresponding to the log acquisition speed limit type.
The information included in the log collection configuration file is related to the information included in the log collection configuration file template, and the information included in the log collection configuration file template is related to the log collection requirement. Fig. 4 shows, as an example, that information such as a log file identifier to be acquired, a storage path of a log file to be acquired, a log acquisition speed limit time, and a log acquisition speed limit type may be simultaneously read from a log acquisition configuration file in a log acquisition stage.
According to the technical scheme of the embodiment of the disclosure, the requirement log related to the new container can be accurately acquired in the log acquisition stage by acquiring the identification of the log file to be acquired and the storage path of the log file to be acquired; by acquiring the log acquisition speed-limiting time and the log acquisition speed-limiting type, the data transmission amount in the log acquisition process can be effectively controlled, the occupation of bandwidth resources on computing equipment is reduced, the log acquisition is effectively limited, and the resource consumption on the computing equipment is relatively controllable.
Further, in the embodiment of the present disclosure, the time limit for log collection includes: and reading the log to be collected according to the storage path of the log file to be collected. Specifically, the log collection stage comprises two sub-stages of reading (reading) a log to be collected and collecting (harvest) the log, and log collection speed limitation is performed in the log reading process, so that on one hand, speed limitation operation is simple and convenient to realize, and on the other hand, the earlier the speed limitation operation execution time is, the more effective the speed limitation effect is.
Optionally, the collecting the log generated in the operation process of the new container according to the log collection speed limit time and the collection speed corresponding to the log collection speed limit type includes:
calling a preset speed limit acquisition function in the process of executing a read function in a data acquisition thread; the log collection speed-limiting type and the collection speed corresponding to the log collection speed-limiting type are defined in the speed-limiting collection function, and the speed-limiting collection function can be realized based on a token bucket algorithm; the acquisition speed (rate _ limit) may be in seconds, e.g. set to 1M/s;
and reading logs generated by a new container corresponding to the storage path of the log file to be acquired according to the acquisition speed defined in the speed-limiting acquisition function.
Taking a filebeat log collector for collecting logs as an example, the data collection thread may be a harvester thread, and the read function may be a reader function. The log collection speed limit type comprises row speed limit collection or byte speed limit collection and the like in the log collection process, and taking a token bucket algorithm as an example, when the log is collected according to byte speed limit, one token (token) can represent one byte; when the log is collected according to the row speed limit, one token can represent one row in the log file; in the log collection process, when the token is insufficient, the current log collection thread (goroutine) can be blocked by calling time (wait) function until there are enough tokens, so as to continue to execute the log collection operation. Regarding the setting of the speed limit position, for example, the following method can be adopted:
1) adding a ratelimit.
2) Creating a bucket in the newHarvester according to the configuration;
3) wait call is added after h.reader.next of harvest.
On the basis of the above technical solution, optionally, after acquiring the log generated in the operation process of the new container based on the log acquisition configuration file, the log acquisition method provided in the embodiment of the present disclosure further includes:
encrypting the collected log; the log encryption can be realized by any available encryption algorithm;
and based on the network transmission state between the current computing equipment and the target log receiver, sending the encrypted log to the target log receiver according to preset times (values can be flexibly set).
Generally, after the log collection is completed, data transmission is performed by using a public network in the process of being sent to a target log receiver, and in order to ensure the security of the log data, the log data can be encrypted and then distributed. In the log distribution process, the network transmission state between the current computing device and the target log receiver, such as stable or unstable network transmission quality, can be determined in a manner of transmitting heartbeat data packets between the current computing device and the target log receiver; the encrypted log is then sent at least once to the intended log receiver based on the determined network transmission status, i.e. a successful transmission of the log data is ensured based on a retry mechanism.
Taking a target log receiver as a card (kafka) message middleware as an example, the log encryption and the safe transmission are realized through a filecut log collector compatible with a safl protocol of kafka. In addition, in the log distribution process, the caching function of the filebeat log collector can be started, when the link fault occurs between the filebeat log collector and the kafka, log data can be temporarily cached in the collector, and the log data can be sent to the kafka again after the link fault is recovered.
Fig. 5 is a schematic flow chart of log collection and distribution provided in the embodiment of the present disclosure, specifically taking a filecut log collector as an example. As shown in fig. 5, the fileteam journal collector may collect a journal generated during the running process of a container from a computing device according to a storage path of a journal file to be collected, cache the journal, for example, store the journal in a splooer, and then distribute the journal in a message queue to a target journal receiver according to a journal analysis requirement, for example, the journal receiver may be a journal search server (such as an elastic search, etc.), a journal analysis tool (such as logstack, etc.), a message middleware (such as kafka, etc.), or a database (such as redis, etc.).
Further, with reference to fig. 5, the following description is an exemplary description of a main implementation flow of the filebeat journal collector, and should not be construed as a specific limitation to the embodiment of the present disclosure:
1) harvester: one input source (e.g., a log file) in each input, would be managed by a harvester for collecting data;
2) starting a harvester by a Run method, reading a file according to lines, and sending the read data to a specified output source in an event form;
3) reader interface is responsible for executing actual file reading operation;
4) crawler: holding pointers of all the inputs, and being responsible for starting all the inputs and collecting data to be sent to a publisher of the libbaby;
5) a module: logs of common application programs can be predefined by using a module, so that the use is simplified;
6) registrar: recording the reading state of the data source after the transmission of a section of data is finished;
7) input: each input corresponds to an input item in the configuration and corresponds to a specific input source, such as a log file, a container standard output and the like, but an input may correspond to a plurality of input sources, such as a plurality of log files and the like, and has a registry for managing all the haverster.
The linkage among input, crawler, harvester and reader is realized as follows:
1) after a Filebeat. run entry function enters, creating a new crawler, and calling the crawler.Start;
2) the function internally executes a for loop, executes startInput () for input in each configuration, internally creates a new input.
3) Run method is performed inside the function:
3.1 firstly, a Runner.input.run function is executed once;
3.2 if p.once is configured, the function returns;
3.3 otherwise, executing for circulation, simultaneously waiting for done channel and ScanFrequency overtime channel messages, exiting if done message, and continuing to call p.input.run execution if overtime message;
4) run. input. log package realization of run function:
4.1 execution of p.scan: for each file path for which the input is responsible, checking whether the file has a file state record which is left once;
4.1.1 if not, perform startHarvester: creating a harvester object, calling Setup to create a reader, and opening a file; then calling SendStateUpdate to update the state in the state of the SendStateUpdate, and then sending the state to the registry; finally, executing registry.Start, placing the newly created harvester object into the map of registry for recording, then creating goroutine, calling harvester.run internally, and starting to execute the cycle of the harvester;
4.1.2 otherwise, perform harvestExistingFile:
4.1.2.1 if it is found that this file is already finished, although there is an old state, and this harvester has stopped destroying, then call startHarvester to create a new harvester;
4.1.2.2 detecting whether the newly found file is an old file or not and is renamed, if so, directly updating the file record without creating a new harvester;
4.1.2.3 if state. finished is not true, it means that harvester is still executing, no operation needs to be performed;
5) log.harvester.run function implementation: next, reading the next row of data and returning a message.
Fig. 6 is a schematic structural diagram of a log collection configuration file generation apparatus provided in an embodiment of the present disclosure, and the embodiment of the present disclosure may be applied to how to conveniently collect logs generated in an operation process of a container deployed on a computing device. The log collection configuration file generation device can be realized by adopting software and/or hardware and can be integrated on any computing equipment.
As shown in fig. 6, the log collection configuration file generation apparatus provided in the embodiment of the present disclosure includes a new container monitoring module 601 and a configuration file generation module 602, where:
a new container monitoring module 601 for monitoring whether a new container is generated on the computing device;
a configuration file generation module 602, configured to generate a log collection configuration file for a new container based on the log collection configuration file template and a deployment file of the new container if it is monitored that the new container is generated on the computing device;
and the log collection configuration file is used for collecting logs generated in the operation process of the new container.
Optionally, the configuration file generating module 602 includes:
the log collection tag determining unit is used for determining whether a log collection tag exists in a deployment file of a new container if the new container is monitored to be generated on the computing equipment;
and the configuration file generating unit is used for generating the log collection configuration file for the new container based on the log collection configuration file template and the deployment file of the new container if the log collection label is determined to exist in the deployment file of the new container.
Optionally, the configuration file generating module 602 includes:
the first acquisition unit is used for acquiring the identification of the log file to be acquired and the storage path of the log file to be acquired from the deployment file of the new container if the new container is monitored to be generated on the computing equipment;
and the configuration file generating unit is used for adding the log file identification to be acquired and the storage path of the log file to be acquired into the log acquisition configuration file template and generating the log acquisition configuration file for the new container.
Optionally, the configuration file generating module 602 further includes:
and the second acquisition unit is used for acquiring the log acquisition speed limit time and the log acquisition speed limit type of the new container from the deployment file of the new container so as to participate in generating the log acquisition configuration file of the new container.
Optionally, the apparatus for generating a log collection configuration file provided in the embodiment of the present disclosure further includes:
and the event clearing module is used for clearing the historical monitoring events aiming at whether the new container is generated on the computing device before the new container monitoring module 601 carries out the operation of monitoring whether the new container is generated on the computing device.
The log collection configuration file generation device provided by the embodiment of the disclosure can execute any log collection configuration file generation method provided by the embodiment of the disclosure, and has corresponding functional modules and beneficial effects of the execution method. Reference may be made to the description of any method embodiment of the disclosure that may not be described in detail in the embodiments of the apparatus of the disclosure.
Fig. 7 is a schematic structural diagram of a log collection apparatus provided in an embodiment of the present disclosure, which may be suitable for a case how to conveniently collect logs generated in an operation process of a container deployed on a computing device. The log collection device can be realized by adopting software and/or hardware and can be integrated on any computing equipment.
The log collection device provided by the embodiment of the disclosure is realized based on a log collection configuration file, and the log collection configuration file is obtained by using any log collection configuration file generation method provided by the embodiment of the disclosure. For a specific explanation of the log collection configuration file generation method, reference may be made to the description in the above embodiments.
As shown in fig. 7, the log collecting apparatus provided in the embodiment of the present disclosure may include a configuration file obtaining module 701 and a log collecting module 702, where:
a configuration file obtaining module 701, configured to obtain a log collection configuration file of a new container on a computing device;
and the log collection module 702 is configured to collect logs generated in the operation process of the new container based on the log collection configuration file.
Optionally, the log collection module 702 includes:
the first reading unit is used for reading the log file identification to be collected and the storage path of the log file to be collected in the log collection configuration file;
and the log acquisition unit is used for acquiring logs corresponding to the log file identification to be acquired, which are generated in the operation process of the new container, according to the storage path of the log file to be acquired.
Optionally, the log collecting module 702 further includes:
the second reading unit is used for reading the log acquisition speed limit time and the log acquisition speed limit type in the log acquisition configuration file;
and the log acquisition unit is used for acquiring logs generated in the operation process of the new container according to the log acquisition speed-limiting time and the acquisition speed corresponding to the log acquisition speed-limiting type.
Optionally, the log collection speed limit time includes: and reading the log to be collected according to the storage path of the log file to be collected.
Optionally, the log collecting unit includes:
the function calling subunit is used for calling a preset speed limit acquisition function in the process of executing the function reading in the data acquisition thread; the log collection speed limit type and the collection speed corresponding to the log collection speed limit type are defined in the speed limit collection function;
and the log reading subunit is used for reading the log generated by the new container corresponding to the storage path of the log file to be collected according to the collection speed defined in the speed-limiting collection function.
Optionally, the log collecting device provided in the embodiment of the present disclosure may further include:
the data volume or running state monitoring module is used for monitoring whether the log data volume generated by the new container exceeds a log storage threshold value or not or monitoring whether the running state of the new container is switched from a normal running state to an abnormal running state or not in the running process of the new container;
and the log sending module is used for sending the logs generated by the new container to the target log receiver in the process of calling the operating system function of the computing equipment to clear the logs generated by the new container based on the file reference technology if the log is the target log receiver.
Optionally, the log collecting device provided in the embodiment of the present disclosure may further include:
a log encryption module, configured to encrypt the acquired log after the log acquisition module 702 executes an operation of acquiring the log generated in the running process of the new container based on the log acquisition configuration file;
and the encrypted log sending module is used for sending the encrypted log to the target log receiver according to the preset times based on the network transmission state between the current computing equipment and the target log receiver.
The log collection device provided by the embodiment of the disclosure can execute any log collection method provided by the embodiment of the disclosure, and has corresponding functional modules and beneficial effects of the execution method. Reference may be made to the description of any method embodiment of the disclosure that may not be described in detail in the embodiments of the apparatus of the disclosure.
Fig. 8 is a schematic structural diagram of a computing device according to an embodiment of the present disclosure. As shown in fig. 8, computing device 800 includes one or more processors 801 and memory 802.
The processor 801 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the computing device 800 to perform desired functions.
In one example, the computing device 800 may also include: an input device 803 and an output device 804, which are interconnected by a bus system and/or other form of connection mechanism (not shown).
The input device 803 may also include, for example, a keyboard, a mouse, and the like.
The output device 804 may output various information including the determined distance information, direction information, and the like to the outside. The output devices 804 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, among others.
Of course, for simplicity, only some of the components of the computing device 800 relevant to the present disclosure are shown in fig. 8, omitting components such as buses, input/output interfaces, and so forth. In addition, computing device 800 may include any other suitable components depending on the particular application.
In addition to the methods and apparatus described above, embodiments of the present disclosure may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform any of the log collection profile generation methods or log collection methods provided by embodiments of the present disclosure.
The computer program product may write program code for performing the operations of embodiments of the present disclosure in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present disclosure may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform any of the log collection profile generation methods or the log collection methods provided by the embodiments of the present disclosure.
A computer-readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present disclosure, which enable those skilled in the art to understand or practice the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (17)
1. A log collection configuration file generation method is characterized by comprising the following steps:
monitoring whether a new container is created on the computing device;
if a new container is monitored to be generated on the computing equipment, generating a log collection configuration file for the new container based on a log collection configuration file template and a deployment file of the new container;
and the log collection configuration file is used for collecting logs generated in the operation process of the new container.
2. The method of claim 1, wherein generating the log collection profile for the new container based on the log collection profile template and the deployment file for the new container comprises:
determining whether a log collection tag exists in a deployment file of the new container;
and if the log collection label is determined to exist in the deployment file of the new container, generating a log collection configuration file for the new container based on the log collection configuration file template and the deployment file of the new container.
3. The method of claim 1, wherein generating the log collection profile for the new container based on the log collection profile template and the deployment file for the new container comprises:
acquiring a log file identifier to be acquired and a storage path of the log file to be acquired from the deployment file of the new container;
and adding the identification of the log file to be collected and the storage path of the log file to be collected into the log collection configuration file template, and generating a log collection configuration file for the new container.
4. The method of claim 3, wherein generating the log collection profile for the new container based on the log collection profile template and the deployment file for the new container further comprises:
and acquiring the log acquisition speed limit time and the log acquisition speed limit type of the new container from the deployment file of the new container so as to participate in generating the log acquisition configuration file of the new container.
5. The method of claim 1, further comprising, prior to the monitoring whether a new container is generated on the computing device:
clearing historical monitoring events for whether a new container is generated on the computing device.
6. A log collection method, which is implemented based on a log collection configuration file obtained by using the log collection configuration file generation method according to any one of claims 1 to 5, and includes:
acquiring a log acquisition configuration file of a new container on computing equipment;
and collecting the logs generated in the operation process of the new container based on the log collection configuration file.
7. The method of claim 6, wherein collecting logs generated during the operation of the new container based on the log collection configuration file comprises:
reading the log file identification to be collected and the storage path of the log file to be collected in the log collection configuration file;
and collecting logs corresponding to the log file identification to be collected, which are generated in the operation process of the new container, according to the storage path of the log file to be collected.
8. The method of claim 7, wherein collecting logs generated during the operation of the new container based on the log collection configuration file further comprises:
reading the log acquisition speed limit time and the log acquisition speed limit type in the log acquisition configuration file;
and collecting the logs generated in the operation process of the new container according to the log collection speed limit time and the collection speed corresponding to the log collection speed limit type.
9. The method of claim 8, wherein the log collection speed limit opportunity comprises: and reading the log to be collected according to the storage path of the log file to be collected.
10. The method of claim 9, wherein the collecting the logs generated in the operation process of the new container according to the log collection speed-limiting time and the collection speed corresponding to the log collection speed-limiting type comprises:
calling a preset speed limit acquisition function in the process of executing a read function in a data acquisition thread; the log collection speed limit type and the collection speed corresponding to the log collection speed limit type are defined in the speed limit collection function;
and reading the logs generated by the new container corresponding to the storage path of the log file to be acquired according to the acquisition speed defined in the speed-limiting acquisition function.
11. The method of claim 6, further comprising, during operation of the new vessel:
monitoring whether the log data volume generated by the new container exceeds a log storage threshold or not, or monitoring whether the running state of the new container is switched from a normal running state to an abnormal running state or not;
if so, based on the file reference technology, in the process of calling the operating system function of the computing equipment to clear the log generated by the new container, the log generated by the new container is sent to a target log receiver.
12. The method according to claim 6, further comprising, after collecting the log generated during the operation of the new container based on the log collection configuration file:
encrypting the collected log;
and sending the encrypted log to a target log receiver according to preset times based on the network transmission state between the current computing equipment and the target log receiver.
13. The method of claim 6, wherein obtaining a log collection profile of a new container on a computing device comprises:
and acquiring a log collection configuration file corresponding to the container ID based on the container ID of the new container.
14. A log collection configuration file generation apparatus, comprising:
a new container monitoring module to monitor whether a new container is generated on the computing device;
the configuration file generation module is used for generating a log collection configuration file for a new container based on a log collection configuration file template and a deployment file of the new container if the new container is monitored to be generated on the computing equipment;
and the log collection configuration file is used for collecting logs generated in the operation process of the new container.
15. A log collection device implemented based on a log collection configuration file obtained by the log collection configuration file generation method according to any one of claims 1 to 5, the log collection device comprising:
the configuration file acquisition module is used for acquiring a log acquisition configuration file of a new container on the computing equipment;
and the log acquisition module is used for acquiring logs generated in the operation process of the new container based on the log acquisition configuration file.
16. A computing device, comprising:
a processor;
a memory for storing the processor-executable instructions or programs;
the processor is configured to read the executable instructions or the program from the memory and execute the executable instructions or the program to implement the log collection configuration file generation method according to any one of claims 1 to 5, or to implement the log collection method according to any one of claims 6 to 13.
17. A computer-readable storage medium, characterized in that the storage medium stores a computer program or instructions which, when executed by a processor, implement the log collection profile generation method of any of the above claims 1-5 or the log collection method of any of the above claims 6-13.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011458909.3A CN114625594A (en) | 2020-12-11 | 2020-12-11 | Configuration file generation method, log collection method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011458909.3A CN114625594A (en) | 2020-12-11 | 2020-12-11 | Configuration file generation method, log collection method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114625594A true CN114625594A (en) | 2022-06-14 |
Family
ID=81894808
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011458909.3A Pending CN114625594A (en) | 2020-12-11 | 2020-12-11 | Configuration file generation method, log collection method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114625594A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116701336A (en) * | 2023-05-19 | 2023-09-05 | 国网物资有限公司 | Power data log processing method, electronic device and computer readable medium |
-
2020
- 2020-12-11 CN CN202011458909.3A patent/CN114625594A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116701336A (en) * | 2023-05-19 | 2023-09-05 | 国网物资有限公司 | Power data log processing method, electronic device and computer readable medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10735345B2 (en) | Orchestrating computing resources between different computing environments | |
| US7552447B2 (en) | System and method for using root cause analysis to generate a representation of resource dependencies | |
| US8799923B2 (en) | Determining relationship data associated with application programs | |
| US20080275976A1 (en) | Information gathering tool for systems administration | |
| KR100606025B1 (en) | Network management apparatus and method based on simple network management protocol | |
| US20230040635A1 (en) | Graph-based impact analysis of misconfigured or compromised cloud resources | |
| CN109462507B (en) | Configuration updating method, device and system and electronic equipment | |
| US20230214229A1 (en) | Multi-tenant java agent instrumentation system | |
| CN114595201A (en) | Method, equipment and storage medium for inquiring acquisition record of interface access log | |
| US7275250B1 (en) | Method and apparatus for correlating events | |
| CN111817984A (en) | Message sending method, device, equipment and storage medium | |
| US10394531B2 (en) | Hyper dynamic Java management extension | |
| CN114625594A (en) | Configuration file generation method, log collection method, device, equipment and medium | |
| US10949306B2 (en) | System and method of a cloud service provider virtual machine recovery | |
| US10180914B2 (en) | Dynamic domain name service caching | |
| US11030184B2 (en) | Systems and methods for database active monitoring | |
| KR20170122874A (en) | Apparatus for managing log of application based on data distribution service | |
| CN100382506C (en) | Method for verifying network-unit server in network management system | |
| US11210156B1 (en) | Intelligent distributed tracing | |
| CN110888663B (en) | Application updating method, service server, transfer server, cluster and storage medium | |
| US20200099788A1 (en) | Context data management interface for contact center | |
| WO2003003244A1 (en) | Method of rapidly eliminating different information in databases | |
| CN113407504B (en) | Data processing method, user space file system and storage medium | |
| CN110764882A (en) | Distributed management method, distributed management system and device | |
| US20050257103A1 (en) | System and method to observe user behavior and perform actions introspectable objects |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |