CN117909180A - Log information processing method, device, computer equipment and storage medium - Google Patents

Log information processing method, device, computer equipment and storage medium Download PDF

Info

Publication number
CN117909180A
CN117909180A CN202410113474.0A CN202410113474A CN117909180A CN 117909180 A CN117909180 A CN 117909180A CN 202410113474 A CN202410113474 A CN 202410113474A CN 117909180 A CN117909180 A CN 117909180A
Authority
CN
China
Prior art keywords
log
application
information
container
acquisition
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202410113474.0A
Other languages
Chinese (zh)
Inventor
晏彦
任政
白佳乐
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Industrial and Commercial Bank of China Ltd ICBC
Original Assignee
Industrial and Commercial Bank of China Ltd ICBC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Industrial and Commercial Bank of China Ltd ICBC filed Critical Industrial and Commercial Bank of China Ltd ICBC
Priority to CN202410113474.0A priority Critical patent/CN117909180A/en
Publication of CN117909180A publication Critical patent/CN117909180A/en
Pending legal-status Critical Current

Links

Classifications

    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Debugging And Monitoring (AREA)

Abstract

The present application relates to a log information processing method, apparatus, computer device, storage medium and computer program product, which can be used in the field of financial science and technology or other related fields. The method comprises the following steps: acquiring container environment variable information; the container environment variable information comprises application deployment information, log path information and log name information corresponding to the target application; according to the application type identification in the log name information, determining a matched log acquisition template aiming at the target application from log acquisition templates corresponding to different application types pre-configured in a log acquisition container; filling application deployment information and log path information into a matched log acquisition template to obtain a configuration file of a log acquisition container; based on the configuration file, starting the log collection container, so that the log collection container collects corresponding log collection index data for the target application according to at least one index item pre-configured in the configuration file. By adopting the method, the memory resource can be accurately monitored.

Description

Log information processing method, device, computer equipment and storage medium
Technical Field
The present application relates to the field of computer technology and the field of financial technology, and in particular, to a log information processing method, apparatus, computer device, storage medium, and computer program product.
Background
With the development of computer technology, more and more business organizations develop business based on the Java ecosystem, and developers often need to monitor resources in the process of business development to find possible instantaneous memory problems.
At present, when memory resource monitoring is performed on a business system developed based on a Java ecological system, java programs are often managed and monitored through Java-based management expansion. However, the Java-based management expansion is based on memory resource monitoring by sampling, so that instantaneous memory problems are difficult to find, and accurate memory resource monitoring cannot be performed.
Therefore, the conventional technology has the problem that the memory resource cannot be monitored accurately.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a log information processing method, apparatus, computer device, computer readable storage medium, and computer program product that can accurately monitor memory resources.
A log information processing method includes:
Acquiring container environment variable information of a log acquisition container; the container environment variable information comprises application deployment information, log path information and log name information corresponding to the target application; the log name information comprises an application type identifier for representing the application type of the target application;
The control log collection container determines a matching log collection template aiming at the target application in the log collection templates corresponding to different preset application types according to the application type identification in the log name information;
The log collection container is controlled to fill application deployment information and log path information into a matched log collection template to obtain a configuration file of the log collection container;
Starting a log acquisition container based on the configuration file to control the log acquisition container to acquire corresponding log acquisition index data for the target application according to at least one index item pre-configured in the configuration file.
In one embodiment, the log collection container is controlled to fill application deployment information and log path information into a matching log collection template to obtain a configuration file of the log collection container, including:
the control log acquisition container determines an application information filling identifier and a path information filling identifier in the matched log acquisition template;
And the log collection container is controlled to replace the application information filling identification with application deployment information in the matching log collection template, and the content corresponding to the path information filling identification in the matching log collection template is replaced with log path information, so that the configuration file of the log collection container is obtained.
In one embodiment, the determining, by the control log collection container according to the application type identifier in the log name information, a matching log collection template for the target application in the log collection templates corresponding to different pre-configured application types includes:
The control log acquisition container determines the application type of the target application according to the application type identifier in the log name information;
and the control log collection container selects a matching log collection template aiming at the target application from log collection templates corresponding to different preset application types according to the application type of the target application.
In one embodiment, after the step of starting the log collection container based on the configuration file to control the log collection container to collect the corresponding log collection index data for the target application according to at least one index item pre-configured in the configuration file, the method further includes:
And newly creating an index according to the application dimension by using the acquired log acquisition index data corresponding to the target application, and storing the index into a distributed log storage cluster which is pre-configured with a search engine.
In one embodiment, after the step of creating an index for log collection index data corresponding to the collected target application according to the application dimension and storing the index data in a distributed log storage cluster configured with a search engine in advance, the method further includes:
configuring a log analysis data source aiming at a log monitoring platform; the log analysis data source characterizes the data source corresponding to each log acquisition index data;
The log monitoring platform is used for displaying all logs which are sent from the log analysis data source and aim at target application to collect index data.
In one embodiment, obtaining container environment variable information for a log collection container includes:
Acquiring task configuration information corresponding to a log acquisition task through a cloud platform; the task configuration information is used for representing configuration information of a log acquisition task of a target application by a user;
And generating container environment variable information according to the task configuration information, calling an application configuration calling interface through the cloud platform, and injecting the container environment variable information into the log acquisition container.
In one embodiment, the method further comprises:
Acquiring log acquisition template configuration information aiming at different application types; the log acquisition template configuration information corresponding to any application type comprises a log acquisition index item identification set, an application query display identification and a log acquisition result output position aiming at an application belonging to the application type;
And copying the configuration information of the log acquisition templates of different application types into an original image file of the log acquisition container to generate the image file of the log acquisition container.
A log information processing apparatus, the apparatus comprising:
The acquisition module is used for acquiring the container environment variable information of the log acquisition container; the container environment variable information comprises application deployment information, log path information and log name information corresponding to the target application; the log name information comprises an application type identifier for representing the application type of the target application;
the determining module is used for controlling the log collection container to determine a matched log collection template aiming at the target application in the log collection templates corresponding to different preset application types according to the application type identification in the log name information;
The filling module is used for controlling the log collection container to fill the application deployment information and the log path information into the matched log collection template to obtain a configuration file of the log collection container;
The acquisition module is used for starting the log acquisition container based on the configuration file so as to control the log acquisition container to acquire corresponding log acquisition index data for the target application according to at least one index item pre-configured in the configuration file.
A computer device comprising a memory storing a computer program and a processor implementing the steps of the method described above when the processor executes the computer program.
A computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method described above.
A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method described above.
The log information processing method, the log information processing device, the computer equipment, the storage medium and the computer program product are used for acquiring container environment variable information of a container for log acquisition; the container environment variable information comprises application deployment information, log path information and log name information corresponding to the target application; the log name information comprises an application type identifier for representing the application type of the target application; the control log collection container determines a matching log collection template aiming at the target application in the log collection templates corresponding to different preset application types according to the application type identification in the log name information; the log collection container is controlled to fill application deployment information and log path information into a matched log collection template to obtain a configuration file of the log collection container; starting a log acquisition container based on the configuration file to control the log acquisition container to acquire corresponding log acquisition index data for the target application according to at least one index item pre-configured in the configuration file; therefore, the matching log acquisition templates for the target application can be determined in the pre-configured log acquisition templates for different application types according to the application types corresponding to the target application, so that the log of the target application can be flexibly analyzed according to the application types of the target application, and meanwhile, the log acquisition index of the target application can be quickly and accurately obtained based on each pre-configured index item in the matching log acquisition templates of the target application, and the accurate monitoring of the memory resource is facilitated.
Drawings
FIG. 1 is a flow chart of a log information processing method according to an embodiment;
FIG. 2 is a diagram corresponding to log collection index data in one embodiment;
FIG. 3 is a schematic diagram illustrating a change of log collected index data according to an embodiment;
FIG. 4 is a block diagram of the structure of a logging system in one embodiment;
FIG. 5 is a flowchart of a log information processing method according to another embodiment;
FIG. 6 is a block diagram showing a configuration of a log information processing apparatus in one embodiment;
fig. 7 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
It should be noted that the present application discloses a log information processing method, a log information processing device, a log information processing computer readable storage medium and a log information processing computer program product, which can be applied to the technical field of financial science and technology.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the disclosure described herein may be capable of operation in sequences other than those illustrated or described herein. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure.
The log information processing method provided by the embodiment of the application can be applied to a log system. The log system is a system for specially collecting application logs and providing storage, retrieval and analysis capabilities, and can adopt an ELK log system framework (a framework for solving log data). The method comprises the steps that a log system obtains container environment variable information aiming at a log collection container; the container environment variable information comprises application deployment information, log path information and log name information corresponding to a target application; the log name information comprises an application type identifier for representing the application type of the target application; the log system controls the log collection container to determine a matched log collection template aiming at the target application from log collection templates corresponding to different preset application types according to the application type identification in the log name information; the log system controls the log collection container to fill the application deployment information and the log path information into the matched log collection template to obtain a configuration file of the log collection container; and the log system starts the log acquisition container based on the configuration file to control the log acquisition container to acquire corresponding log acquisition index data for the target application according to at least one index item pre-configured in the configuration file.
In one embodiment, as shown in fig. 1, a log information processing method is provided, and the log information processing method is applied to a log system for illustration, and includes the following steps:
Step S102, obtaining container environment variable information of a log acquisition container; the container environment variable information comprises application deployment information, log path information and log name information corresponding to the target application; the log name information comprises an application type identifier for representing the application type of the target application.
The log collection container may be a virtual container for collecting logs, for example, the log collection container may be fluented containers for collecting logs of applications. The log collection container can be used for collecting gc logs, wherein the gc logs are logs generated in the garbage collection process when the java virtual machine carries out garbage collection, and the jvm virtual machine releases the memory used by the java program when carrying out garbage collection.
Wherein, the container environment variable information may refer to an environment variable within the container.
The target application may be any application.
The application deployment information may include information such as an application node IP (hostIP), an application node name (hostName), and an application name (appName), where the application node is a node where the target application is deployed.
The log path information may be a storage path of the log.
The log name information may be a file name of the log.
The application type may refer to a type to which the application belongs. In practice, different application types may correspond to different types of jdks (Java Development Kits, java development kit/package).
The application type identifier may refer to a type identifier corresponding to an application, for example, an identifier corresponding to the application OpenJDK may be "OpenJDK".
In the specific implementation, the log system acquires container environment variable information aiming at the log acquisition container, namely, acquires application deployment information, log path information and log name information corresponding to the target application.
Step S104, the control log collection container determines a matching log collection template aiming at the target application from the log collection templates corresponding to different preset application types according to the application type identification in the log name information.
The log collection template may be a template used for collecting logs of the application.
The matching log collection template may be a log collection template corresponding to an application type to which the target application belongs.
In the specific implementation, the log system controls the log collection container to determine a matched log collection template aiming at a target application from log collection templates corresponding to different application types pre-configured in the log collection container according to application type identifiers in log name information by running a container start script in the log collection container.
And S106, controlling the log collection container to fill the application deployment information and the log path information into the matched log collection template to obtain a configuration file of the log collection container.
The configuration file may refer to a file required by the log collection container during running.
In the specific implementation, the log system controls the log collection container to fill application deployment information and log path information into a matched log collection template by running a container start script in the log collection container, so as to obtain a configuration file of the log collection container.
Step S108, starting a log acquisition container based on the configuration file to control the log acquisition container to acquire corresponding log acquisition index data for the target application according to at least one index item pre-configured in the configuration file.
The index item may be an item corresponding to an acquisition index of the log, for example, the index item may refer to an acquisition index of gc times.
The log collection index data may refer to an index value corresponding to the log collection index, for example, the log collection index data may refer to a collection value corresponding to the collection index of gc times.
In the specific implementation, the log system starts the log collection container based on the configuration file by running a container start script in the log collection container, so as to control the log collection container to collect corresponding log collection index data for the target application according to at least one index item pre-configured in the configuration file.
The log information processing method, the log information processing device, the computer equipment, the storage medium and the computer program product are used for acquiring container environment variable information of a container for log acquisition; the container environment variable information comprises application deployment information, log path information and log name information corresponding to the target application; the log name information comprises an application type identifier for representing the application type of the target application; the control log collection container determines a matching log collection template aiming at the target application in the log collection templates corresponding to different preset application types according to the application type identification in the log name information; the log collection container is controlled to fill application deployment information and log path information into a matched log collection template to obtain a configuration file of the log collection container; starting a log acquisition container based on the configuration file to control the log acquisition container to acquire corresponding log acquisition index data for the target application according to at least one index item pre-configured in the configuration file; therefore, the matching log acquisition templates for the target application can be determined in the pre-configured log acquisition templates for different application types according to the application types corresponding to the target application, so that the log of the target application can be flexibly analyzed according to the application types of the target application, and meanwhile, the log acquisition index of the target application can be quickly and accurately obtained based on each pre-configured index item in the matching log acquisition templates of the target application, and the accurate monitoring of the memory resource is facilitated.
In another embodiment, the log collection container is controlled to fill application deployment information and log path information into a matching log collection template to obtain a configuration file of the log collection container, including: the control log acquisition container determines an application information filling identifier and a path information filling identifier in the matched log acquisition template; and the log collection container is controlled to replace the application information filling identification with application deployment information in the matching log collection template, and the content corresponding to the path information filling identification in the matching log collection template is replaced with log path information, so that the configuration file of the log collection container is obtained.
The application information filling identifier may refer to an identifier corresponding to a position in the log acquisition template, where the application information needs to be filled in.
The path information filling identifier may refer to an identifier corresponding to a position in the log acquisition template, where the path information needs to be filled in.
In the specific implementation, the log system controls the log collection container to determine an application information filling identifier and a path information filling identifier in the matched log collection template by running a container starting script in the log collection container, then controls the log collection container to replace the application information filling identifier with application deployment information in the matched log collection template, and replaces the content corresponding to the path information filling identifier in the matched log collection template with log path information so as to obtain a configuration file of the log collection container.
According to the technical scheme, an application information filling identifier and a path information filling identifier are determined in a matched log acquisition template by controlling a log acquisition container; the method comprises the steps of controlling a log collection container to replace application information filling identification with application deployment information in a matched log collection template, and replacing content corresponding to a path information filling identification in the matched log collection template with log path information to obtain a configuration file of the log collection container; therefore, the information related to the log collection of the target application in the container environment variable information can be rapidly filled into the matched log collection template of the target application, so that the configuration file of the log collection container can be rapidly generated, and the log collection efficiency can be improved.
In another embodiment, the control log collection container determines a matching log collection template for the target application from the log collection templates corresponding to different pre-configured application types according to the application type identifier in the log name information, including: the control log acquisition container determines the application type of the target application according to the application type identifier in the log name information; and the control log collection container selects a matching log collection template aiming at the target application from log collection templates corresponding to different preset application types according to the application type of the target application.
In the specific implementation, the log system controls the log collection container to determine the application type of the target application according to the application type identification in the log name information by running the container start script in the log collection container, and then controls the log collection container to select a matched log collection template for the target application from the log collection templates corresponding to different pre-configured application types according to the application type of the target application by running the container start script.
According to the technical scheme of the embodiment, the application type of the target application is determined by controlling the log acquisition container according to the application type identifier in the log name information; the control log collection container selects a matching log collection template aiming at the target application from log collection templates corresponding to different preset application types according to the application type of the target application; therefore, the log collection template matched with the target application can be accurately matched according to the application type identifier in the log name information, and the log analysis efficiency of the target application is improved.
In another embodiment, after the step of starting the log collection container based on the configuration file to control the log collection container to collect the corresponding log collection index data for the target application according to the at least one index item pre-configured in the configuration file, the method further includes: and newly creating an index according to the application dimension by using the acquired log acquisition index data corresponding to the target application, and storing the index into a distributed log storage cluster which is pre-configured with a search engine.
Wherein the distributed log storage cluster may be an elastsearch (a distributed, open-source search analysis engine).
In the specific implementation, the log system creates an index according to the application dimension by using the log acquisition index data corresponding to the acquired target application through the log acquisition container, and stores the index data into a distributed log storage cluster which is pre-configured with a search engine.
According to the technical scheme, the acquired log acquisition index data corresponding to the target application is newly built according to the application dimension and stored in the distributed log storage cluster which is pre-configured with the search engine, so that the log acquisition index data can be efficiently stored, and the log acquisition index data can be conveniently and quickly called later.
In another embodiment, after the step of creating an index for log collection index data corresponding to the collected target application according to the application dimension and storing the index data in a distributed log storage cluster configured with a search engine in advance, the method further includes: configuring a log analysis data source aiming at a log monitoring platform; the log analysis data source characterizes the data source corresponding to each log acquisition index data; the log monitoring platform is used for displaying all logs which are sent from the log analysis data source and aim at target application to collect index data.
The log monitoring platform can be Grafana (an open-source data visualization tool) which is a log acquisition index analysis platform.
The log analysis data source can be a data source of log analysis data required by the log monitoring platform when log analysis is performed.
In a specific implementation, the log system configures a log analysis data source for the log monitoring platform, wherein the log analysis data source characterizes a data source corresponding to each log acquisition index data, and the log monitoring platform can display each log acquisition index data sent from the log analysis data source and aiming at a target application.
According to the technical scheme, a log analysis data source aiming at a log monitoring platform is configured; the log analysis data source characterizes the data source corresponding to each log acquisition index data; the log monitoring platform is used for displaying each log acquisition index data aiming at the target application, which is sent from the log analysis data source; therefore, the real-time visual display of the collected index data of each log of the target application can be realized by configuring a log analysis data source mode in the log monitoring platform.
In another embodiment, obtaining container environment variable information for a log collection container includes: acquiring task configuration information corresponding to a log acquisition task through a cloud platform; the task configuration information is used for representing configuration information of a log acquisition task of a target application by a user; and generating container environment variable information according to the task configuration information, calling an application configuration calling interface through the cloud platform, and injecting the container environment variable information into the log acquisition container.
The cloud platform may refer to a cloud service platform of a business mechanism.
The log collection task may refer to a task corresponding to a log collection object when log collection is performed on the log collection object, and the log collection object may refer to any application.
The task configuration information may include application deployment information of a target application required by the log acquisition task to perform log acquisition, log name information and log path information corresponding to a log storing log acquisition results acquired by the log acquisition task.
The application configuration call interface may refer to an application programming interface, such as Kubernetes API.
In a specific implementation, the log system acquires task configuration information corresponding to a log acquisition task through a cloud platform, wherein the task configuration information is used for representing configuration information of the log acquisition task of a target application by a user, for example, a log name or a log path for storing an acquisition result, and the log system generates a container environment variable according to the task configuration information and invokes an application configuration calling interface through the cloud platform to inject the container environment variable information into a log acquisition container.
According to the technical scheme, task configuration information corresponding to a log acquisition task is obtained through a cloud platform; the task configuration information is used for representing configuration information of a log acquisition task of a target application by a user; generating container environment variable information according to the task configuration information, calling an application configuration calling interface through a cloud platform, and injecting the container environment variable information into a log acquisition container; therefore, task configuration information corresponding to the log acquisition task configured by the user on the cloud platform can be obtained, and customized acquisition can be realized according to the user requirements.
In another embodiment, the method further comprises: acquiring log acquisition template configuration information aiming at different application types; the log acquisition template configuration information corresponding to any application type comprises a log acquisition index item identification set, an application query display identification and a log acquisition result output position aiming at an application belonging to the application type; and copying the configuration information of the log acquisition templates of different application types into an original image file of the log acquisition container to generate the image file of the log acquisition container.
The log collection index item identification set may be a set including a plurality of log collection index item identifications, where the log collection index item identification is an identification corresponding to the log collection index item.
The application query presentation identifier may refer to an identifier that is used to characterize an application when information query or information presentation is performed later.
The log collection result output position may refer to a position where the log collection result is stored.
The original image file may refer to an image file corresponding to the container before the container is started.
In a specific implementation, the log system acquires log acquisition template configuration information aiming at different application types, wherein the log acquisition template configuration information corresponding to any application type comprises a log acquisition index item identification set aiming at an application belonging to the application type, an application query display identification and a log acquisition result output position, and copies the log acquisition template configuration information of different application types into an original image file of a log acquisition container to generate the image file of the log acquisition container.
For the convenience of understanding of those skilled in the art, the following exemplary method for generating a configuration file of a log collection container is described by taking a container fluented as an example, and the following specific steps when configuring a log collection template in a container fluentd include:
Step 1: and configuring a log acquisition template of the complete fluentd container. The file of the log acquisition template contains source, filter, match parts.
Firstly, configuring a source part in a file, so that the source part can be subjected to regular analysis later to obtain gc indexes required to be acquired. Specifically, for the format of gc logs of IBM JDK and Open JDK, fluented container profiles with different regular expressions are configured. Taking IBM JDK as an example, the gc LOG is in xml format, each collection index (collection index includes in-heap memory before and after gc, newgeneration memory, senile memory, gc time consuming, gc type, etc.) of the gc process is concentrated in gc-start tag and gc-end tag, multiple rows of regular expressions can be written according to gc-start tag and gc-end tag formats to realize analysis of each collection index of the gc LOG, and the LOG PATH can be configured into a parameter form (the value is # # app_log_path#).
The filter section in the configuration file is then provided. Specifically, the fields hostIP (application node IP), hostName (application node name), and appName (application name) are respectively configured as parameter forms, and are used as subsequent query display identifiers, so that subsequent queries and displays can be performed by querying the display identifiers to screen and display gc acquisition indexes of an application, for example, the gc acquisition indexes of an application are screened and displayed by the application name identifiers, so that monitoring is realized.
Finally, the match section in the configuration file. Specifically, the collected gc log is directly sent to the elastic search for storage, so that the collection efficiency of log collection indexes can be improved.
By configuring source, filter, match three parts for different JDK types, a log acquisition template of a complete fluentd container for the different JDK types can be configured, and gc log analysis supporting multiple JDKs is realized.
Step 2: and (3) making a new mirror image on the basis of the original mirror image file fluented, copying the files of the log acquisition templates supporting the gc logs of a plurality of JDKs into the new mirror image, thereby generating a new mirror image file, and realizing configuration of the gc log acquisition templates aiming at different JDKs in a fluentd container.
According to the technical scheme, the configuration information of the template is acquired by acquiring logs aiming at different application types; the log acquisition template configuration information corresponding to any application type comprises a log acquisition index item identification set, an application query display identification and a log acquisition result output position aiming at an application belonging to the application type; copying configuration information of log acquisition templates of different application types into an original image file of a log acquisition container to generate an image file of the log acquisition container; therefore, corresponding log acquisition templates can be configured for applications of different application types, and log acquisition and log analysis can be flexibly performed for the applications of different application types.
For the convenience of understanding of those skilled in the art, the following exemplary method for implementing gc index collection based on fluented containers is provided, which specifically includes:
Step 1: the user configures gc log collection tasks at the cloud platform of the business organization, and the task configuration information includes a log name (e.g., IBMJDK _ gclog or OpenJDK _ gclog) and a log path. Then, the cloud platform acquires the application configuration call K8S interface and injects the task configuration information configured by the user and the application deployment information of the target application (for example, hostIP, i.e., application node IP, hostName, i.e., application node name, and appName, i.e., application name) into the container as container environment variables when the fluentd container is started.
Step 2: running a container startup script in fluentd containers, acquiring container environment variables, judging the log name to be IBMJDK or OpenJDK based on the container environment variables, and selecting a log acquisition template in fluented containers according to the type of the characterization of the log name.
Step 3: and acquiring a LOG PATH in the environment variable of the container, and replacing the parameter part "#APP_LOG_PATH#" in the LOG acquisition template with the LOG PATH.
Step 4: running a container startup script in fluentd containers, replacing hostIP (application node IP), hostName (application node name) and appName (application name) in container environment variables to hostIP parameter parts, hostName parameter parts and appName parameter parts in a log acquisition template to obtain a configuration file.
Step 5: and running a container starting script in the fluentd container, starting the fluentd container by using the configuration file with the replaced parameters as the configuration file of the fluentd container, so that acquisition of gc logs can be realized, and storing the gc logs in an elastic search according to the newly added index of the application dimension after analyzing the index of the gc logs. The index name may be named with the application name of the target application as a prefix, and in this way, only the gc index of the target application may be obtained.
The following further exemplarily provides a method for implementing presentation of the plurality of log collection index data through Grafana, which specifically includes:
Step 1: the log analysis data source is configured at Grafana with the index stored in the elastic search as the data source.
Step 2: the panel rules are customized on Grafana, so that gc times, gc time consumption, gc front and back memory curves and the like are displayed, cloud native monitoring of the gc process is realized, and accordingly memory resource conditions can be intuitively monitored. The graph shown in fig. 2 shows gc indicators of "gc times/day", "average gc times/day", "global gc time/day", "last 1 minute gc times", "last 1 minute average gc time", and the like. Meanwhile, fig. 3 also exemplarily shows 21 that gc times are on a certain day: 50-23: the change condition diagrams corresponding to the gc times of 50 time periods, the global gc times, the total memory available in the heap, the nursery memory available, the tenure memory available and the LOA memory available are shown.
Step 2: alarm rules are customized at Grafana, for example, when gc per minute is greater than 10 alarms.
By configuring the log acquisition template in the log system, the real-time analysis, display and alarm of the gc log can be realized, the problem that the traditional gc log analysis depends on offline analysis of a tool is solved, and the requirement of application on the monitoring precision of memory resources is met.
Fig. 4 shows a block diagram of a log system that enables real-time monitoring of gc logs. The log system in fig. 4 includes an acquisition layer, a storage layer, an alarm layer, and a presentation layer. The acquisition layer is mainly used for acquiring gc logs output in an application process through a fluented container, obtaining index data required to be acquired by a user after the user passes through a filter, and storing the index data into a distributed log cluster (ELASTICSEARCH CLUSTER) of the storage layer according to an application dimension newly added index; the storage layer is mainly used for taking indexes stored in the distributed log clusters (ELASTICSEARCH CLUSTER) as a log monitoring platform (Grafana) data source in the alarm layer; the alarm layer is mainly used for customizing alarm rules on a log monitoring platform (Grafana) so as to realize the alarm function of the alarm platform, and customizing panel rules on the log monitoring platform (Grafana) so as to realize the display of index data on a display layer; the presentation layer is used for realizing cloud native monitoring of the gc log according to alarm rules and data sources customized by a log monitoring platform (Grafana) of the alarm layer.
In another embodiment, as shown in fig. 5, a log information processing method is provided, and the method is applied to a log system for illustration, and includes the following steps:
Step S502, obtaining container environment variable information of a log acquisition container; the container environment variable information comprises application deployment information, log path information and log name information corresponding to a target application; and the log name information comprises an application type identifier for representing the application type of the target application.
Step S504, controlling the log collection container to determine a matching log collection template aiming at the target application from the log collection templates corresponding to different pre-configured application types according to the application type identification in the log name information.
And step S506, controlling the log collection container to determine an application information filling identifier and a path information filling identifier in the matched log collection template.
Step S508, controlling the log collection container to replace the application information filling identifier with the application deployment information in the matched log collection template, and replacing the content corresponding to the path information filling identifier in the matched log collection template with the log path information, so as to obtain the configuration file of the log collection container.
Step S510, starting the log collection container based on the configuration file, so as to control the log collection container to collect corresponding log collection index data for the target application according to at least one index item pre-configured in the configuration file.
Note that, the specific limitation of the above steps may be referred to as the specific limitation of one log information processing method.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a log information processing device for realizing the above related log information processing method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in one or more embodiments of the log information processing device provided below may refer to the limitation of the log information processing method hereinabove, and will not be described herein.
In one embodiment, as shown in fig. 6, there is provided a log information processing apparatus including:
an acquisition module 602, configured to acquire container environment variable information for a log acquisition container; the container environment variable information comprises application deployment information, log path information and log name information corresponding to the target application; the log name information comprises an application type identifier for representing the application type of the target application;
A determining module 604, configured to control the log collection container to determine a matching log collection template for the target application from the log collection templates corresponding to different pre-configured application types according to the application type identifier in the log name information;
The filling module 606 is configured to control the log collection container to fill the application deployment information and the log path information into the matching log collection template, so as to obtain a configuration file of the log collection container;
The collection module 608 is configured to start the log collection container based on the configuration file, so as to control the log collection container to collect corresponding log collection index data for the target application according to at least one index item pre-configured in the configuration file.
In one embodiment, the filling module 606 is specifically configured to control the log collection container to determine an application information filling identifier and a path information filling identifier in the matching log collection template; and the log collection container is controlled to replace the application information filling identification with application deployment information in the matching log collection template, and the content corresponding to the path information filling identification in the matching log collection template is replaced with log path information, so that the configuration file of the log collection container is obtained.
In one embodiment, the determining module 604 is specifically configured to control the log collection container to determine an application type to which the target application belongs according to the application type identifier in the log name information; and the control log collection container selects a matching log collection template aiming at the target application from log collection templates corresponding to different preset application types according to the application type of the target application.
In one embodiment, the apparatus further comprises: and the storage module is used for newly creating an index according to the application dimension and storing the acquired log acquisition index data corresponding to the target application into a distributed log storage cluster which is pre-configured with a search engine.
In one embodiment, the apparatus further comprises: the configuration module is used for configuring a log analysis data source aiming at the log monitoring platform; the log analysis data source characterizes the data source corresponding to each log acquisition index data; the log monitoring platform is used for displaying all logs which are sent from the log analysis data source and aim at target application to collect index data.
In one embodiment, the obtaining module 602 is specifically configured to obtain, through the cloud platform, task configuration information corresponding to a log collection task; the task configuration information is used for representing configuration information of a log acquisition task of a target application by a user; and generating container environment variable information according to the task configuration information, calling an application configuration calling interface through the cloud platform, and injecting the container environment variable information into the log acquisition container.
In one embodiment, the apparatus further comprises: the generation module is used for acquiring log acquisition template configuration information aiming at different application types; the log acquisition template configuration information corresponding to any application type comprises a log acquisition index item identification set, an application query display identification and a log acquisition result output position aiming at an application belonging to the application type; and copying the configuration information of the log acquisition templates of different application types into an original image file of the log acquisition container to generate the image file of the log acquisition container.
Each of the modules in the log information processing apparatus described above may be implemented in whole or in part by software, hardware, or a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 7. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing log information processing data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a log information processing method.
It will be appreciated by those skilled in the art that the structure shown in FIG. 7 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In one embodiment, a computer device is provided that includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of a log information processing method as described above. The steps of a log information processing method herein may be the steps of a log information processing method of the above-described respective embodiments.
In one embodiment, a computer-readable storage medium is provided, in which a computer program is stored which, when executed by a processor, causes the processor to perform the steps of a log information processing method as described above. The steps of a log information processing method herein may be the steps of a log information processing method of the above-described respective embodiments.
In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, causes the processor to perform the steps of a log information processing method as described above. The steps of a log information processing method herein may be the steps of a log information processing method of the above-described respective embodiments.
The user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or sufficiently authorized by each party.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high density embedded nonvolatile memory, resistive random access memory (ReRAM), magneto-resistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (PHASE CHANGE memory, PCM), graphene memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in various forms such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.

Claims (11)

1. A log information processing method, the method comprising:
Acquiring container environment variable information of a log acquisition container; the container environment variable information comprises application deployment information, log path information and log name information corresponding to a target application; the log name information comprises an application type identifier for representing the application type of the target application;
Controlling the log collection container to determine a matched log collection template aiming at the target application in the log collection templates corresponding to different preset application types according to the application type identification in the log name information;
controlling the log collection container to fill the application deployment information and the log path information into the matched log collection template to obtain a configuration file of the log collection container;
and starting the log acquisition container based on the configuration file to control the log acquisition container to acquire corresponding log acquisition index data for the target application according to at least one index item pre-configured in the configuration file.
2. The method of claim 1, wherein the controlling the log collection container to populate the application deployment information and the log path information into the matching log collection template to obtain the profile of the log collection container comprises:
Controlling the log acquisition container to determine an application information filling identifier and a path information filling identifier in the matched log acquisition template;
And controlling the log collection container to replace the application information filling identification with the application deployment information in the matched log collection template, and replacing the content corresponding to the path information filling identification in the matched log collection template with the log path information to obtain a configuration file of the log collection container.
3. The method according to claim 1, wherein the controlling the log collection container to determine a matching log collection template for the target application from the log collection templates corresponding to different pre-configured application types according to the application type identifier in the log name information includes:
controlling the log acquisition container to determine the application type of the target application according to the application type identifier in the log name information;
and controlling the log acquisition container to select a matching log acquisition template aiming at the target application from log acquisition templates corresponding to different preset application types according to the application type of the target application.
4. The method of claim 1, wherein after the step of starting the log collection container based on the configuration file to control the log collection container to collect the corresponding log collection index data for the target application according to at least one index item pre-configured in the configuration file, the method further comprises:
and newly creating an index according to the application dimension by using the acquired log acquisition index data corresponding to the target application, and storing the index into a distributed log storage cluster which is pre-configured with a search engine.
5. The method according to claim 4, wherein after the step of creating an index according to an application dimension for storing the acquired log acquisition index data corresponding to the target application in a distributed log storage cluster configured with a search engine in advance, the method further comprises:
Configuring a log analysis data source aiming at a log monitoring platform; the log analysis data source represents the data source corresponding to each log acquisition index data;
the log monitoring platform is used for displaying the log acquisition index data aiming at the target application, wherein the log acquisition index data is sent from the log analysis data source.
6. The method of claim 1, wherein the obtaining container environment variable information for the log collection container comprises:
Acquiring task configuration information corresponding to a log acquisition task through a cloud platform; the task configuration information is used for representing configuration information of a user on a log acquisition task of the target application;
And generating the container environment variable information according to the task configuration information, calling an application configuration calling interface through the cloud platform, and injecting the container environment variable information into the log acquisition container.
7. The method according to claim 1, wherein the method further comprises:
Acquiring log acquisition template configuration information aiming at different application types; the log acquisition template configuration information corresponding to any application type comprises a log acquisition index item identification set, an application query display identification and a log acquisition result output position aiming at an application belonging to the application type;
copying the configuration information of the log acquisition templates of the different application types into an original image file of the log acquisition container to generate the image file of the log acquisition container.
8. A log information processing apparatus, characterized in that the apparatus comprises:
The acquisition module is used for acquiring the container environment variable information of the log acquisition container; the container environment variable information comprises application deployment information, log path information and log name information corresponding to a target application; the log name information comprises an application type identifier for representing the application type of the target application;
The determining module is used for controlling the log collection container to determine a matching log collection template aiming at the target application from log collection templates corresponding to different preset application types according to the application type identification in the log name information;
The filling module is used for controlling the log collection container to fill the application deployment information and the log path information into the matched log collection template to obtain a configuration file of the log collection container;
The acquisition module is used for starting the log acquisition container based on the configuration file so as to control the log acquisition container to acquire corresponding log acquisition index data for the target application according to at least one index item pre-configured in the configuration file.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 7 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.
11. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.
CN202410113474.0A 2024-01-26 2024-01-26 Log information processing method, device, computer equipment and storage medium Pending CN117909180A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410113474.0A CN117909180A (en) 2024-01-26 2024-01-26 Log information processing method, device, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410113474.0A CN117909180A (en) 2024-01-26 2024-01-26 Log information processing method, device, computer equipment and storage medium

Publications (1)

Publication Number Publication Date
CN117909180A true CN117909180A (en) 2024-04-19

Family

ID=90685800

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410113474.0A Pending CN117909180A (en) 2024-01-26 2024-01-26 Log information processing method, device, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN117909180A (en)

Similar Documents

Publication Publication Date Title
US11429600B2 (en) Loading queries using search points
US11366686B1 (en) Managing user data in a multitenant deployment
US11698900B2 (en) Leveraging search query history in a search interface
US8839107B2 (en) Context based script generation
US9305176B2 (en) Database generation from a spreadsheet
US7093207B1 (en) Data analysis flow engine
CN109002334B (en) Operation and maintenance platform and data processing method thereof
CN112433712A (en) Report display method and device, computer equipment and storage medium
US20180365312A1 (en) Systems, Methods, and Apparatuses for Creating, Linking and Discovering Business Navigation Maps for Analyzing Data
CN114238085A (en) Interface testing method and device, computer equipment and storage medium
CN111061733A (en) Data processing method and device, electronic equipment and computer readable storage medium
US9164746B2 (en) Automatic topology extraction and plotting with correlation to real time analytic data
CN113885860A (en) Method and equipment for automatically configuring management page to generate interface service
CN112035555B (en) Information display method, device and equipment
CN117909180A (en) Log information processing method, device, computer equipment and storage medium
US20080250050A1 (en) Method and system for developing a desired set of configuration profiles for an application program and storage medium for storing a set of computer instructions which effectuate the method
CN112527747A (en) Log template configuration and display method and device
CN113704593B (en) Operation data processing method and related device
US20150046358A1 (en) Importing tested objects into benefits programs deployed on production systems
US11567743B1 (en) Integrated testing tool for data replication
CN117389605A (en) Service interface configuration method, device, equipment, storage medium and program product
CN117707678A (en) Editing method and device of script file, computer equipment and storage medium
CN116756022A (en) Data preparation method, device, computer equipment and storage medium
CN116974720A (en) Method and device for executing data processing task, computer equipment and storage medium
CN117689083A (en) Method, device, system and medium for managing solar power

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination