CN115129673A - Log processing method and device, electronic equipment and storage medium - Google Patents

Log processing method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN115129673A
CN115129673A CN202210836317.3A CN202210836317A CN115129673A CN 115129673 A CN115129673 A CN 115129673A CN 202210836317 A CN202210836317 A CN 202210836317A CN 115129673 A CN115129673 A CN 115129673A
Authority
CN
China
Prior art keywords
log
information
application
processed
determining
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210836317.3A
Other languages
Chinese (zh)
Inventor
陈飞跃
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Agricultural Bank of China
Original Assignee
Agricultural Bank of China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Agricultural Bank of China filed Critical Agricultural Bank of China
Priority to CN202210836317.3A priority Critical patent/CN115129673A/en
Publication of CN115129673A publication Critical patent/CN115129673A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815—Journaling file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
    • G06Q10/06311—Scheduling, planning or task assignment for a person or group
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
    • G06Q10/06313—Resource planning in a project environment
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/20—Administration of product repair or maintenance

Landscapes

  • Business, Economics & Management (AREA)
  • Engineering & Computer Science (AREA)
  • Human Resources & Organizations (AREA)
  • Theoretical Computer Science (AREA)
  • Economics (AREA)
  • Strategic Management (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Entrepreneurship & Innovation (AREA)
  • General Business, Economics & Management (AREA)
  • Tourism & Hospitality (AREA)
  • Quality & Reliability (AREA)
  • Operations Research (AREA)
  • Marketing (AREA)
  • Development Economics (AREA)
  • Game Theory and Decision Science (AREA)
  • Educational Administration (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The embodiment of the invention discloses a log processing method and device, electronic equipment and a storage medium. The log processing method may specifically include: acquiring a log to be processed, and determining log application information of the log to be processed; determining a log processing rule corresponding to the log to be processed according to the log application information; and determining the log content information of the log to be processed, and performing log splitting processing on the log content information according to the log processing rule. According to the technical scheme of the embodiment of the invention, the log content can be quickly positioned when the log types and the log formats are numerous, and the log analysis efficiency of operation and maintenance personnel is improved.

Description

Log processing method and device, electronic equipment and storage medium
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a log processing method and device, electronic equipment and a storage medium.
Background
At present, more application software exists and the application software operates independently, and operation and maintenance logs and transaction logs generated by application programs in the process of business transaction occurrence are scattered on respective application servers and stored by each application. When the operation and maintenance personnel perform log analysis, report summarization or view display, the operation and maintenance personnel need to analyze the log content of the operation and maintenance log, so that the log content needs to be processed.
The existing log processing method usually processes a specific application log or a certain type of application log, but the operation and maintenance logs and the transaction logs generated by the existing application program in the process of business transaction occurrence scatter on respective application servers, so that the types and the formats of the logs are numerous, and the specific logs are difficult to locate in a short time when the contents of the logs are queried.
Disclosure of Invention
The embodiment of the invention provides a log processing method and device, electronic equipment and a storage medium, which can be used for quickly positioning log contents and improving the log analysis efficiency of operation and maintenance personnel when the types and the formats of logs are numerous.
According to an aspect of the present invention, there is provided a log processing method, including:
acquiring a log to be processed, and determining log application information of the log to be processed;
determining a log processing rule corresponding to the log to be processed according to the log application information;
determining the log content information of the log to be processed, and performing log splitting processing on the log content information according to the log processing rule.
According to another aspect of the present invention, there is provided a log processing apparatus including:
the log application information determining module is used for acquiring a log to be processed and determining log application information of the log to be processed;
the log processing rule determining module is used for determining a log processing rule corresponding to the log to be processed according to the log application information;
and the log splitting processing module is used for determining the log content information of the log to be processed and performing log splitting processing on the log content information according to the log processing rule.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the log processing method according to any of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the log processing method according to any one of the embodiments of the present invention when the computer instructions are executed.
According to the technical scheme of the embodiment of the invention, the log to be processed is obtained, the log application information of the log to be processed is determined, the log processing rule corresponding to the log to be processed is determined according to the log application information, and the log content information of the log to be processed is determined, so that the log splitting processing is carried out on the log content information according to the log processing rule, the problem that the log content cannot be located in a short time when the log types and the log formats are numerous in the existing log processing method is solved, the log content can be located quickly, and the log analysis efficiency of operation and maintenance personnel is improved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a flowchart of a log processing method according to an embodiment of the present invention;
fig. 2 is a flowchart of a log processing method according to a second embodiment of the present invention;
fig. 3 is a schematic diagram of application log format information according to a second embodiment of the present invention;
FIG. 4 is a diagram of a result of splitting a pending log according to a second embodiment of the present invention;
fig. 5 is a schematic diagram of a log processing apparatus according to a third embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device implementing the log processing method according to the embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example one
Fig. 1 is a flowchart of a log processing method according to an embodiment of the present invention, where this embodiment is applicable to a case where log content is quickly located when log types and log formats are numerous, and the method may be executed by a log processing apparatus, where the apparatus may be implemented in a software and/or hardware manner, and may generally be directly integrated in an electronic device that executes the method, where the electronic device may be a terminal device or a server device, and the embodiment of the present invention does not limit the type of the electronic device that executes the log processing method. Specifically, as shown in fig. 1, the log processing method may specifically include the following steps:
s110, obtaining a log to be processed, and determining log application information of the log to be processed.
The log to be processed can be any log needing to be split. Optionally, the to-be-processed log may be a to-be-processed operation and maintenance log. The log application information may be any information of the application software that generates the log to be processed, and may be used to identify the application software of the log to be processed. For example, the log application information may be system ID (Identity Document) information of the application software that generates the log to be processed, and the like, which is not limited in this embodiment of the present invention.
In the embodiment of the invention, the operation and maintenance logs of each application software can be collected, stored, displayed and the like through the log center platform. After the operation and maintenance logs are generated by the application software, the log center platform collects the operation and maintenance logs generated by the application software and stores the collected operation and maintenance logs, and after the operation and maintenance logs are stored by the log center platform, the logs to be processed can be further obtained, and log application information of the logs to be processed is determined. It can be understood that the log to be processed may include attribute information of a plurality of operation and maintenance logs, for example, log application information, and the like, which is not limited in this embodiment of the present invention.
Optionally, after the log center platform stores the operation and maintenance log, the log to be processed may be further obtained by the log processing plug-in of the log center platform, and the log application information of the log to be processed is determined.
Optionally, the log-centric platform may employ an ELK framework. Specifically, the ELK framework consists of three parts, namely an Elasticissearch part, Logstash and Kibana. The Elastic Search (ES) is a distributed RESTful style search and analysis engine, which is an open-source distributed, extensible and highly available real-time document storage component and provides real-time search and analysis capability. Logstash is an open-source server-side data processing pipeline, and can collect data from multiple sources at the same time, convert the data, and then send the data to a favorite repository of a user. Kibana is an open-source free tool, can provide log analysis-friendly web pages for ES and Logstash, and can help to summarize, analyze and search important data logs.
It should be noted that Logstash has numerous processing plug-ins in log format, such as grok, json, ruby, or disect. Taking the grok as an example, the grok can conveniently split the log into a specified format according to configured grok rules. Assuming that the log to be processed is 55.3.244.1GET/index.html 158240.043', parsing the message into fields through a grak configuration, and applying a filter to process the log to be processed to comprise the following fields of 55.3.244.1; method is GET; html; 15824 for bytes; duration: 0.043'.
Although the grok plug-in is convenient to use when processing logs, the plug-in is very suitable for system logs, such as Apache, nginx or other webserver logs. Therefore, although Logstash carries numerous log format processing plug-ins, log content cannot be located quickly when log types and log formats are numerous.
And S120, determining a log processing rule corresponding to the log to be processed according to the log application information.
The log processing rule may be a rule to be followed when the log to be processed is processed.
In the embodiment of the invention, after the log center platform acquires the log to be processed and determines the log application information of the log to be processed, the log center platform can further determine the log processing rule corresponding to the log to be processed according to the log application information. It can be understood that, for different logs to be processed, corresponding log processing rules may be different or the same, and the embodiment of the present invention does not limit this.
Optionally, the log processing rule corresponding to the log to be processed is determined according to the log application information, and the log processing rule corresponding to the log to be processed may be determined according to the log application information by a log processing plug-in of the log center platform.
Optionally, before determining the log processing rule corresponding to the log to be processed, the log processing rule of the log to be processed may be preset, so that when the log to be processed is processed, the log to be processed can be processed according to the preset rule.
S130, determining the log content information of the log to be processed, and performing log splitting processing on the log content information according to the log processing rule.
The log content information may be log content generated by any application software during running. Optionally, the log content information may be operation and maintenance log content information. It can be understood that the log to be processed may include attribute information of a plurality of operation and maintenance logs, for example, log content information, and the like, which is not limited in this embodiment of the present invention. It can be further understood that, when the operation and maintenance personnel maintain the application software, the operation and maintenance personnel need to perform log analysis according to the log content generated by the application software.
In the embodiment of the invention, after the log center platform determines the log processing rule corresponding to the log to be processed according to the log application information, the log center platform can further determine the log content information of the log to be processed, so as to perform log splitting processing on the log content information according to the log processing rule.
Optionally, the log content information of the log to be processed is determined, the log content information is split according to the log processing rule, the log content information of the log to be processed can be determined through a log processing plug-in of the log center platform, and the log content information is split according to the log processing rule.
According to the technical scheme, the log to be processed is obtained, the log application information of the log to be processed is determined, the log processing rule corresponding to the log to be processed is determined according to the log application information, the log content information of the log to be processed is determined, and therefore the log splitting processing is carried out on the log content information according to the log processing rule, the problem that the log content cannot be located in a short time when the log types and the log formats are numerous in the existing log processing method is solved, the log content can be located quickly when the log types and the log formats are numerous, and the log analysis efficiency of operation and maintenance personnel is improved.
Example two
Fig. 2 is a flowchart of a log processing method provided in the second embodiment of the present invention, which further details the above technical solutions and provides a plurality of specific selectable implementation manners for determining log application information of the log to be processed, determining a log processing rule corresponding to the log to be processed according to the log application information, and performing log splitting processing on the log content information according to the log processing rule. The solution in this embodiment may be combined with the individual alternatives in one or more of the embodiments described above. As shown in fig. 2, the method may include the steps of:
s210, determining at least one current application software, and acquiring user configuration information corresponding to each current application software.
The current application software may be any application software that the current log center can collect the operation and maintenance logs. The user configuration information may be information configured by the operation and maintenance personnel for the current application software.
In the embodiment of the invention, the log center platform determines at least one current application software and acquires user configuration information corresponding to each current application software. It can be understood that the log center platform may determine a plurality of different application software to obtain user configuration information corresponding to the different application software, so that the log center platform can process the operation and maintenance logs of the different application software when there are many log types and log formats, and further quickly locate the log content.
Optionally, at least one current application software may be determined through a log center configuration page of the log center platform, and user configuration information corresponding to each current application software is obtained.
S220, determining log format information according to the user configuration information, and determining software application information of each current application software.
The log format information may be a log format of the log to be processed. It is understood that the log format information may be configured by the operation and maintenance personnel according to actual needs. The software application information may be information for identifying the current application software, for example, ID information for identifying the current application software, and the like, which is not limited in this embodiment of the present invention.
In the embodiment of the present invention, after the log center platform obtains the user configuration information corresponding to each current application software, the log center platform may further determine the log format information according to the user configuration information, and determine the software application information of each current application software.
Optionally, the log format information may be determined according to the user configuration information through a log center configuration page of the log center platform, and the software application information of each current application software may be determined.
And S230, splicing the log format information and the software application information into application log format information corresponding to the current application software, and storing the application log format information into an application log format database.
The application log format information may include software application information and log format information of the application software. The application log format database may be a database storing application log format information. It will be appreciated that the application log format database may be a database in a log centric platform.
In the embodiment of the invention, after determining the log format information according to the user configuration information and determining the software application information of each current application software, the log center platform can further splice the log format information and the software application information into the application log format information corresponding to the current application software and store the application log format information into the application log format database. It is understood that in the application log format database, the log format information and the software application information of each current application software are in one-to-one correspondence.
Optionally, the log format information and the software application information may be spliced into application log format information corresponding to the current application software through a log center configuration page of the log center platform, and the application log format information is stored in an application log format database.
S240, under the condition that the current time interval reaches a preset time interval threshold, all application log format information stored in the application log format database is obtained, and each application log format information is stored in a first container.
Wherein the current time interval may be a time interval of the current system time. The preset time interval threshold may be a preset time interval threshold, for example, 2 minutes, and the like, which is not limited in this embodiment of the present invention. It is understood that the preset time interval threshold may be set according to actual requirements. The first container may be a storage space for temporarily storing the application log format information.
In the embodiment of the invention, the log center platform can determine the current time interval, and when the current time interval reaches the preset time interval threshold, all application log format information stored in the application log format database is acquired, and each application log format information is stored in the first container.
Optionally, the current time interval may be determined by a log processing plug-in of the log center platform, and when the current time interval reaches a preset time interval threshold, all application log format information stored in the application log format database is acquired, and each application log format information is stored in the first container. Alternatively, the first container may be a temporary storage space in the log processing plug-in.
Optionally, the log processing plug-in may obtain all Application log format information stored in the Application log format database through an Application Programming Interface (API) Interface, and store each Application log format information in the first container. Illustratively, the first container may be a HashMap (an implementation of a hash-table based Map interface).
And S250, acquiring the log to be processed, and determining the log application information of the log to be processed.
Optionally, determining the log application information of the log to be processed may include: acquiring log identity attribute information and log source attribute information of a log to be processed; and determining the log application information of the log to be processed according to the log identity attribute information and the log source attribute information.
The log identity attribute information may be attribute information representing a log identity of the log to be processed, for example, the log identity attribute information may be a subject name of the log to be processed, and the embodiment of the present invention does not limit this. The log source attribute information may be attribute information characterizing a log source of the log to be processed.
Specifically, the log center platform may obtain the log identity attribute information and the log source attribute information of the log to be processed, so as to determine the log application information of the log to be processed according to the log identity attribute information and the log source attribute information.
Optionally, the log identity attribute information and the log source attribute information of the log to be processed may be obtained through a log processing plug-in of the log center platform, so as to determine the log application information of the log to be processed according to the log identity attribute information and the log source attribute information.
It should be noted that fig. 2 is a schematic diagram of an implementation manner, steps S210 to S230, step S240 and step S250 do not have a sequential relationship, and steps S210 to S230, step S240 and step S250 may be implemented first, step S240, step S210 to step S230 and step S250 may be implemented first, or step S250, step S210 to step S230 and step S240 may be implemented first, or the three steps may be implemented in parallel or alternatively implemented.
S260, according to the log application information, obtaining target application log format information corresponding to the log application information in the first container.
The target application log format information may be information stored in the first container and corresponding to the log to be processed. It is understood that if the software application information stored in the application log format information of the first container is the same as the log application information, the application log format information may be determined as the target application log format information.
In the embodiment of the present invention, after determining the log application information of the log to be processed, the log center platform may further obtain, according to the log application information, target application log format information corresponding to the log application information in the first container.
Optionally, the target application log format information corresponding to the log application information in the first container may be acquired according to the log application information by a log processing plug-in of the log center platform.
S270, determining target log format information corresponding to the log application information according to the target application log format information.
The target log format information may be log format information in the target application log format information. It is understood that if the software application information stored in the application log format information of the first container is the same as the log application information, the log format information in the application log format information may be determined as the target log format information.
In the embodiment of the present invention, after the log center platform obtains the target application log format information corresponding to the log application information in the first container according to the log application information, the target log format information corresponding to the log application information may be further determined according to the target application log format information.
Optionally, the target log format information corresponding to the log application information may be determined according to the target application log format information by a log processing plug-in of the log center platform.
S280, carrying out standardization processing on the target log format information to obtain a standardized regular expression, and determining the standardized regular expression as the log processing rule.
The normalized regular expression may be a standard regular expression capable of matching log contents.
In the embodiment of the invention, after determining the target log format information corresponding to the log application information according to the target application log format information, the log center platform can further perform standardization processing on the target log format information to obtain a standardized regular expression, so as to determine the standardized regular expression as the log processing rule. It can be understood that, in order to make the field in the operation and maintenance log more clearly defined by the operation and maintenance staff, the application log format information stored in the application log format database may be the field name and the field content customized by the operation and maintenance staff, and before performing log splitting processing on the log content information according to the log processing rule, the target log format information needs to be standardized.
Optionally, the log processing plug-in of the log center platform is used for standardizing the target log format information to obtain a standardized regular expression, and the standardized regular expression is determined as the log processing rule.
S290, determining the log content information of the log to be processed, and performing log splitting processing on the log content information according to the log processing rule.
Optionally, the method may further include: determining a log splitting field name according to the target application log format information; performing log splitting processing on the log content information according to the log processing rule, wherein the log splitting processing comprises the following steps: acquiring a log splitting field value corresponding to a log splitting field name in log content information according to a log processing rule; and splicing the log splitting field name and the log splitting field value corresponding to the log splitting field name into log splitting information, and storing the log splitting information into a second container.
The log splitting field name may be a field name corresponding to the split log content. The log split field value may be a field value in the split log content. The log splitting information may be information obtained by splitting a log of the log to be processed. The second container may be a storage space for temporarily storing the log splitting information.
Specifically, the log center platform may determine a log splitting field name according to the target application log format information, and obtain a log splitting field value corresponding to the log splitting field name in the log content information according to the log processing rule, so as to splice the log splitting field name and the log splitting field value into the log splitting information, thereby storing the log splitting information in the second container. It can be understood that the log split field name and the log split field value in the log split information are corresponding.
Optionally, the log splitting field name may be determined according to the target application log format information by a log processing plug-in of the log center platform, and the log splitting field value corresponding to the log splitting field name in the log content information is obtained according to the log processing rule, so that the log splitting field name and the log splitting field value are spliced into the log splitting information, and the log splitting information is stored in the second container. It will be appreciated that the second container may be a temporary storage space in the log processing plug-in.
Optionally, after performing log splitting processing on the log content information according to the log processing rule, the method may further include: and converting the log splitting processing result into a log splitting processing result with a set format, and storing the log splitting processing result with the set format into a preset document storage component.
Wherein the log splitting processing result may be log splitting information stored in the second container. The preset document storage component can be a preset component for storing log splitting information in the log center platform. For example, the formatted log splitting processing result may be a json formatted log splitting processing result. The pre-set document storage component may be an ES component in a log-centric platform.
Specifically, after the log center platform performs log splitting processing on the log content information according to the log processing rule, the log splitting processing result can be further converted into a log splitting processing result in a set format, and the log splitting processing result in the set format is stored in the preset document storage component.
Optionally, the log splitting processing result may be converted into a log splitting processing result in a set format by a log processing plug-in of the log center platform, and the log splitting processing result in the set format is stored in the preset document storage component.
In a specific example of the embodiment of the present invention, fig. 3 is a schematic diagram of application log format information provided in the second embodiment of the present invention, and fig. 4 is a schematic diagram of a to-be-processed log splitting result provided in the second embodiment of the present invention. Assuming that the log to be processed is "2021-11-1511: 24:31,182ppNoteId | | InterfaceId | | cost | 3| | flag ═ 1", and the application log format information stored in the application log format database is as shown in fig. 3, the specific log format information may be "% logTime # (\\ d 4} - \ d {2} -, d {2} \ d }, d +) noteid:% \\\ applnoteid [ (+) -% >, and"% > -% | _ c \\ ost | -%, cost _ int \\\\\ \ 2}, and "% > -%, the result of processing is as shown in a graph of"% "-%,"%, i \\\\\ \ c \\\ \ cost \\\ \ int \\ \ int \ 2}, and "(+) \\.
According to the technical scheme of the embodiment, at least one current application software is determined, the user configuration information corresponding to each current application software is obtained, the log format information is determined according to the user configuration information, the software application information of each current application software is determined, the log format information and the software application information are spliced into the application log format information corresponding to the current application software, and therefore the application log format information is stored in the application log format database. And under the condition that the current time interval reaches a preset time interval threshold, acquiring all application log format information stored in the application log format database, and storing all application log format information into a first container. Acquiring a log to be processed, determining log application information of the log to be processed, acquiring target application log format information corresponding to the log application information in a first container according to the log application information, determining target log format information corresponding to the log application information according to the target application log format information, standardizing the target log format information to obtain a standardized regular expression, determining the standardized regular expression as a log processing rule, determining the log content information of the log to be processed, therefore, the log splitting processing is carried out on the log content information according to the log processing rule, and the problem that when the log types and the log formats are numerous in the existing log processing method is solved, the problem that the log content cannot be located in a short time can be solved, the log content can be located quickly when a plurality of log types and log formats exist, and the log analysis efficiency of operation and maintenance personnel is improved.
EXAMPLE III
Fig. 5 is a schematic diagram of a log processing apparatus according to a third embodiment of the present invention, and as shown in fig. 5, the apparatus includes: a log application information determining module 510, a log processing rule determining module 520, and a log splitting processing module 530, wherein:
a log application information determining module 510, configured to obtain a log to be processed, and determine log application information of the log to be processed;
a log processing rule determining module 520, configured to determine, according to the log application information, a log processing rule corresponding to the log to be processed;
the log splitting processing module 530 is configured to determine log content information of the log to be processed, and perform log splitting processing on the log content information according to the log processing rule.
According to the technical scheme, the log to be processed is obtained, the log application information of the log to be processed is determined, the log processing rule corresponding to the log to be processed is determined according to the log application information, the log content information of the log to be processed is determined, and therefore the log splitting processing is carried out on the log content information according to the log processing rule, the problem that the log content cannot be located in a short time when the log types and the log formats are numerous in the existing log processing method is solved, the log content can be located quickly when the log types and the log formats are numerous, and the log analysis efficiency of operation and maintenance personnel is improved.
Optionally, the log application information determining module 510 may be specifically configured to: acquiring log identity attribute information and log source attribute information of the log to be processed; and determining the log application information of the log to be processed according to the log identity attribute information and the log source attribute information.
Optionally, the apparatus may be further specifically configured to: determining at least one current application software and acquiring user configuration information corresponding to each current application software; determining log format information according to the user configuration information, and determining software application information of each current application software; and splicing the log format information and the software application information into application log format information corresponding to the current application software, and storing the application log format information into an application log format database.
Optionally, the apparatus may be further specifically configured to: and under the condition that the current time interval reaches a preset time interval threshold, acquiring all application log format information stored in the application log format database, and storing each application log format information into a first container.
Optionally, the log processing rule determining module 520 may be specifically configured to: acquiring target application log format information corresponding to the log application information in the first container according to the log application information; determining target log format information corresponding to the log application information according to the target application log format information; and carrying out standardization processing on the target log format information to obtain a standardized regular expression, and determining the standardized regular expression as the log processing rule.
Optionally, the apparatus may be further specifically configured to: determining a log splitting field name according to the target application log format information; correspondingly, the log splitting processing module 530 may be specifically configured to: according to the log processing rule, acquiring a log splitting field value corresponding to the log splitting field name in the log content information; and splicing the log splitting field name and the log splitting field value corresponding to the log splitting field name into log splitting information, and storing the log splitting information into a second container.
Optionally, the log splitting processing module 530 may be further specifically configured to: and converting the log splitting processing result into a log splitting processing result with a set format, and storing the log splitting processing result with the set format into a preset document storage component.
The log processing device provided by the embodiment of the invention can execute the log processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Example four
FIG. 6 illustrates a block diagram of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 6, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM)12, a Random Access Memory (RAM)13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM)12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 may also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 11 performs the various methods and processes described above, such as a log processing method.
In some embodiments, the log processing method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the log processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the log processing method by any other suitable means (e.g. by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user may provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.
The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A log processing method, comprising:
acquiring a log to be processed, and determining log application information of the log to be processed;
determining a log processing rule corresponding to the log to be processed according to the log application information;
determining the log content information of the log to be processed, and performing log splitting processing on the log content information according to the log processing rule.
2. The method of claim 1, wherein the determining log application information of the pending log comprises:
acquiring log identity attribute information and log source attribute information of the log to be processed;
and determining the log application information of the log to be processed according to the log identity attribute information and the log source attribute information.
3. The method of claim 1, further comprising:
determining at least one current application software and acquiring user configuration information corresponding to each current application software;
determining log format information according to the user configuration information, and determining software application information of each current application software;
and splicing the log format information and the software application information into application log format information corresponding to the current application software, and storing the application log format information into an application log format database.
4. The method of claim 3, further comprising, after storing the application log format information to an application log format database:
and under the condition that the current time interval reaches a preset time interval threshold, acquiring all application log format information stored in the application log format database, and storing each application log format information into a first container.
5. The method according to claim 4, wherein the determining a log processing rule corresponding to the log to be processed according to the log application information comprises:
acquiring target application log format information corresponding to the log application information in the first container according to the log application information;
determining target log format information corresponding to the log application information according to the target application log format information;
and carrying out standardization processing on the target log format information to obtain a standardized regular expression, and determining the standardized regular expression as the log processing rule.
6. The method of claim 5, further comprising:
determining a log splitting field name according to the target application log format information;
the log splitting processing of the log content information according to the log processing rule includes:
according to the log processing rule, acquiring a log splitting field value corresponding to the log splitting field name in the log content information;
and splicing the log splitting field name and the log splitting field value corresponding to the log splitting field name into log splitting information, and storing the log splitting information into a second container.
7. The method according to claim 1, further comprising, after the log splitting processing the log content information according to the log processing rule:
and converting the log splitting processing result into a log splitting processing result with a set format, and storing the log splitting processing result with the set format into a preset document storage component.
8. A log processing apparatus, comprising:
the log application information determining module is used for acquiring a log to be processed and determining log application information of the log to be processed;
the log processing rule determining module is used for determining a log processing rule corresponding to the log to be processed according to the log application information;
and the log splitting processing module is used for determining the log content information of the log to be processed and splitting the log according to the log processing rule.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the log processing method of any one of claims 1-7.
10. A computer-readable storage medium storing computer instructions for causing a processor to implement the log processing method of any one of claims 1 to 7 when executed.
CN202210836317.3A 2022-07-15 2022-07-15 Log processing method and device, electronic equipment and storage medium Pending CN115129673A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210836317.3A CN115129673A (en) 2022-07-15 2022-07-15 Log processing method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210836317.3A CN115129673A (en) 2022-07-15 2022-07-15 Log processing method and device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN115129673A true CN115129673A (en) 2022-09-30

Family

ID=83383419

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210836317.3A Pending CN115129673A (en) 2022-07-15 2022-07-15 Log processing method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN115129673A (en)

Similar Documents

Publication Publication Date Title
CN107809331B (en) Method and device for identifying abnormal flow
EP4099170B1 (en) Method and apparatus of auditing log, electronic device, and medium
CN112527649A (en) Test case generation method and device
CN113836314B (en) Knowledge graph construction method, device, equipment and storage medium
CN114816578A (en) Method, device and equipment for generating program configuration file based on configuration table
CN115048352B (en) Log field extraction method, device, equipment and storage medium
CN114880498B (en) Event information display method and device, equipment and medium
CN116414814A (en) Data checking method, device, equipment, storage medium and program product
CN113590447B (en) Buried point processing method and device
CN115544010A (en) Mapping relation determining method and device, electronic equipment and storage medium
CN115422275A (en) Data processing method, device, equipment and storage medium
CN115687406A (en) Sampling method, device and equipment of call chain data and storage medium
CN112887426B (en) Information stream pushing method and device, electronic equipment and storage medium
CN115757304A (en) Log storage method, device and system, electronic equipment and storage medium
CN115129673A (en) Log processing method and device, electronic equipment and storage medium
CN115328898A (en) Data processing method and device, electronic equipment and medium
CN114218313A (en) Data management method, device, electronic equipment, storage medium and product
CN115858325B (en) Project log adjusting method, device, equipment and storage medium
CN115964409A (en) Management method, device, equipment and storage medium for system access data
CN115220722A (en) Method, device, equipment and medium for back-end internationalization
CN115391618A (en) Data identification method and device, electronic equipment and storage medium
CN115017882A (en) Data blood relationship determination method and device, electronic equipment and storage medium
CN114417219A (en) Information configuration method and device and electronic equipment
CN115686517A (en) Front-end page point burying method and device based on mobile development platform, electronic equipment and storage medium
CN113095788A (en) Question distribution method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination