CN114785604A - Dynamic log analysis method, device, equipment and storage medium - Google Patents

Dynamic log analysis method, device, equipment and storage medium Download PDF

Info

Publication number
CN114785604A
CN114785604A CN202210461097.0A CN202210461097A CN114785604A CN 114785604 A CN114785604 A CN 114785604A CN 202210461097 A CN202210461097 A CN 202210461097A CN 114785604 A CN114785604 A CN 114785604A
Authority
CN
China
Prior art keywords
log
format
output
template
analyzed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210461097.0A
Other languages
Chinese (zh)
Other versions
CN114785604B (en
Inventor
柯明明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Anbotong Jin'an Technology Co ltd
Original Assignee
Beijing Anbotong Jin'an Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Anbotong Jin'an Technology Co ltd filed Critical Beijing Anbotong Jin'an Technology Co ltd
Priority to CN202210461097.0A priority Critical patent/CN114785604B/en
Publication of CN114785604A publication Critical patent/CN114785604A/en
Application granted granted Critical
Publication of CN114785604B publication Critical patent/CN114785604B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
    • H04L63/1425—Traffic logging, e.g. anomaly detection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
    • H04L63/1416—Event detection, e.g. attack signature detection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/22—Parsing or analysis of headers

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Hardware Design (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention relates to a dynamic log analysis method, a device, equipment and a storage medium, wherein the method comprises the following steps: configuring a log template according to the log format of the safety equipment; acquiring a log to be analyzed of the safety equipment, and analyzing the log template to obtain structured data of the log template; the structured data comprises an input log format and an output log format; analyzing the log to be analyzed according to the input log format to obtain log analysis data; and formatting the log analysis data according to the output log format to obtain an output log, and sending the output log to a log center. According to the dynamic log analysis method provided by the invention, the log is analyzed through the log template, the log template is pre-loaded and analyzed into structured data, the analysis efficiency is not reduced due to the increase of log formats, the irregular expression professional format is adopted to open the compiling for analysts, the error probability is reduced, and the maintenance cost is reduced.

Description

Dynamic log analysis method, device, equipment and storage medium
Technical Field
The present invention relates to the field of computer network technologies, and in particular, to a method, an apparatus, a device, and a storage medium for dynamic log parsing.
Background
Any device or service in the information security system may output a log: the system comprises a firewall, a Web server, an IPS engine, a database, terminal software and the like, wherein logs contain a large amount of information which is interested by security management personnel, operation and maintenance personnel, business analysis personnel and the like. Taking a user behavior awareness device as an example, for example: the IP of the visitor, the access time, the destination resource to be accessed, the client information of the visitor, the number of accesses, and the like.
When the log data size is small, various built-in or open-source tools such as gawk, grep, cat and the like which are ready for the operating system are beneficial devices for log analysis. If more complex logic is available, various scripting languages, such as Python and Shell, can be used to solve all the problems, but most of the tools or scripting languages rely on regular expressions.
With the gradual increase of the system volume, the types of devices included in one system also increase, when people need to generate thousands or tens of thousands of log formats for dozens or hundreds of different devices, the method based on the regular expression is very complicated under the condition that the log formats are gradually increased, and the regular expression becomes abnormally difficult to maintain, so that the efficiency is low, the maintenance cost is increased, and the error processing is seriously caused.
Disclosure of Invention
In view of the above, it is desirable to provide a dynamic log parsing method, apparatus, device and storage medium, which are used to solve the problems in the prior art that when the log data size is large, the regular expression is inefficient in parsing the log, the maintenance cost is high, and errors may occur.
In order to achieve the technical purpose, the invention adopts the following technical scheme:
in a first aspect, the present invention provides a dynamic log parsing method, including:
configuring a log template according to the log format of the safety equipment;
acquiring a log to be analyzed of the safety equipment, and analyzing the log template to obtain structured data of the log template; the structured data comprises an input log format and an output log format;
analyzing the log to be analyzed according to the input log format to obtain log analysis data;
and formatting the log analysis data according to the output log format to obtain an output log, and sending the output log to a log center.
Preferably, analyzing the log template to obtain the structured data of the log template includes:
acquiring an input request of a log template, and inputting the log template according to the input request;
and circularly analyzing the log template to obtain the node type of the structured data.
Preferably, the analyzing the log to be analyzed according to the input log format to obtain log analysis data includes:
if the node type is an input parameter node, analyzing the log to be analyzed;
if the node type is an output parameter node, outputting list parameters after log analysis;
if the node type is the end node, the analysis process is ended.
Preferably, if the node type is an input parameter node, the analyzing the log to be analyzed includes:
identifying the format and the field name of the log to be analyzed;
separating the log to be analyzed according to the log format to be analyzed to obtain a separator list;
and analyzing the log to be analyzed according to the separator list and the field name to obtain a log analysis result.
Preferably, the analyzing the log to be analyzed according to the separator list and the field name to obtain a log analysis result, including:
associating the field name with a separator list to obtain a separator analysis result;
and analyzing the log to be analyzed according to the separator analysis result to obtain a log analysis result.
Preferably, the processing of formatting the log analysis data according to the output log format to obtain an output log, and sending the output log to the log center includes:
outputting the output log to a log channel;
and sending the output log to a corresponding log center for storage according to the channel configuration of the log channel.
Preferably, the input log format comprises only one format and the output log format comprises a plurality of formats.
In a second aspect, the present invention further provides a dynamic log parsing apparatus, including:
the configuration module is used for configuring a log template according to the log format of the safety equipment;
the template analysis module is used for acquiring a log to be analyzed of the safety equipment and analyzing the log template to obtain structured data of the log template; the structured data comprises an input log format and an output log format;
the log analysis module is used for analyzing the log to be analyzed according to the input log format to obtain log analysis data;
and the output module is used for formatting the log analysis data according to the output log format to obtain an output log and sending the output log to a log center.
In a third aspect, the present invention also provides an electronic device comprising a memory and a processor, wherein,
a memory for storing a program;
and a processor, coupled to the memory, for executing the program stored in the memory to implement the steps in the dynamic log parsing method in any of the above-mentioned implementation manners.
In a fourth aspect, the present invention further provides a computer-readable storage medium for storing a computer-readable program or instruction, where the program or instruction, when executed by a processor, can implement the steps in the dynamic log parsing method in any one of the above-mentioned implementation manners.
The beneficial effects of adopting the above embodiment are: according to the dynamic log analysis method, the device, the equipment and the storage medium, the corresponding log template is configured according to the log format of the safety equipment, the log to be analyzed is analyzed through the log template to obtain the output log, the log template is pre-loaded and analyzed into the structured data, the log analysis efficiency cannot be reduced due to the increase of the log formats, compared with a regular expression, the writing requirement of the log template on the specialty is low, a large amount of writing is not needed, the log template is opened to be written by an analyst, the error probability is reduced, and the maintenance cost is reduced.
Drawings
FIG. 1 is a flowchart illustrating a dynamic log parsing method according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a dynamic log parsing apparatus according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate preferred embodiments of the invention and together with the description, serve to explain the principles of the invention and not to limit the scope of the invention.
In the description of the present application, "a plurality" means two or more unless specifically limited otherwise.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the invention. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
Before the embodiments of the present invention are explained, the related terms are explained as follows:
UUID: universal Unique Identifier (universal Unique Identifier);
IPS: intrusion Prevention System (Intrusion Prevention System);
gawk: a program language for modifying, comparing and extracting the data in the text file;
grep: the global search Regular Expression (RE) and print out the line, searching the regular expression and printing out the lines comprehensively, is a powerful text search tool, can search the text by using the regular expression and print out the matched lines;
and cat: full configure, command is used to connect the file and print to a standard output device.
The invention provides a dynamic log analysis method, a device, equipment and a storage medium, which are respectively explained below.
Referring to fig. 1, fig. 1 is a schematic flow chart of an embodiment of a dynamic log parsing method provided by the present invention, and an embodiment of the present invention discloses a dynamic log parsing method, including:
s101, configuring a log template according to a log format of the safety equipment;
s102, obtaining a log to be analyzed of the safety equipment, and analyzing the log template to obtain structured data of the log template; the structured data comprises an input log format and an output log format;
s103, analyzing the log to be analyzed according to the input log format to obtain log analysis data;
and S104, formatting the log analysis data according to the output log format to obtain an output log, and sending the output log to a log center.
In a specific embodiment of the present invention, step S101 configures a log template for each security device according to the difference of the security devices, and extracts a log to be analyzed from the security devices. Because the log formats of the security devices may be different, for security devices of different formats, a log template of a corresponding format needs to be configured for each security device, and log analysis is implemented through the log template.
The invention provides a specific embodiment of a configuration log template, and analysts write demand information of a log to be analyzed according to the following template, which specifically comprises the following information:
input _ log _ info (input Log Format information)
type (log type), text (play), json (json), auto (auto);
separation () is space (), comma (), semicolon (;), user-defined, and default is comma (,)
{ N } (field name, format string) (which may be unfilled, parsed by string by default)
{ N }, separation (delimiter) ═ space (), comma (), semicolon (;), user-defined, default to comma (,)
output _ log _ info (output Log Format information) (there may be more than one node, which represents the requirement of outputting multiple formats, see example Log template data 3 illustrated in the figure)
separation (delimiter) ═ space (), comma (), semicolon (;), user-defined, default to comma (,)
{ input Log field name } List, output field name, Format string List (may not output, resolve by string by default)
{ log _ center }, array list, log center information may be configured for sending the merged log to a corresponding log center, for example: the domain name or IP address information of the log center is configured, log center ═ 192.168.1.1:2088, test.
N exists in two formats:
type is plain, format is% NUM, NUM supports 1-20 (this patent supports maximum 20 parameters for example), and x.y can be used to represent the subparameters of X parameter, such as: {% 2.1}, which represents the first subparameter of the 2 nd parameter.
type is json, format is x.y, for example: signature _ id, indicating the signature _ id data 955193 of the node that acquired the alert.
Section denotes the delimiter of the Nth parameter, requiring that the Nth parameter be of string type.
Format string support includes, but is not limited to, basically covering 95% of logs, and custom:
% T, timestamp (number of seconds elapsed relative to 1/1970 00:00: 00), for example: 1645528793776.
% D, date, for example: 02/22/2022-19:03:43.068719, 2022-02-22T19:03:42.971776+ 0800.
% M, MAC address, e.g.: AF 24:54:4A: BB: 43.
% P, IP address, e.g.: 192.168.1.1.
% U, UUID, universally unique identifier of a device or service, for example: 783a6c01-99e1-5f7e-9b06-24a7d415c4a 2.
% d, the number of shapes, can also fit hexadecimal numbers with 0X or 0X.
% s, character string.
% f, floating point number.
% x, hexadecimal integer number.
% A, the known port is shown as an application or service.
The { input log field name } list must be defined in the "{ N } ═ field name" under the input _ log _ info node, for example: { attack time }, { source IP }/{ source port }, { source IP }: source port } are reasonable.
It can be understood that the invention can write the corresponding log template according to a plurality of log formats to realize the analysis of a plurality of different log formats.
In a specific embodiment of the present invention, after the log template is configured in step S102, the log template is analyzed to know each part of the log template, so as to facilitate analysis of data of the log to be analyzed. It should be noted that the input log format may be text (play), json (json), and the like, and the possible input log format is not further limited in the present invention.
In a specific embodiment of the present invention, step S103 inputs a log format of input _ log _ info, determines information such as a log type, a separator, and a field name by inputting information of the log format, parses corresponding information in the log to be parsed, obtains log parsing data, and sends the log parsing data to the output module.
In a specific embodiment of the present invention, step S104 formats the log analysis data according to the output log format, so that the output log format is output according to the set format, and the output log format is stored in the log center.
Compared with the prior art, the dynamic log analysis method provided by the embodiment has the advantages that the corresponding log template is configured according to the log format of the safety equipment, the log to be analyzed is analyzed through the log template to obtain the output log, the log template is pre-loaded and analyzed into the structured data, the log analysis efficiency cannot be reduced due to the increase of the log formats, compared with a regular expression, the writing requirement of the log template on the specialty is low, a large amount of writing is not needed, the log template is opened to be written by an analyst, the error probability is reduced, and the maintenance cost is reduced.
In some embodiments of the present invention, parsing the log template to obtain the structured data of the log template includes:
acquiring an input request of a log template, and inputting the log template according to the input request;
and circularly analyzing the log template to obtain the node type of the structured data.
In the above embodiment, before parsing the log template, an input request of the log template needs to be waited, and after the input request of the log template is obtained, the corresponding log template is input according to the request. And circularly analyzing each row of the log template, identifying the node type of the structured data, determining the corresponding operation of log analysis by the node type, and analyzing the log based on the node type.
In some embodiments of the present invention, parsing a log to be parsed according to an input log format to obtain log parsing data includes:
if the node type is an input parameter node, analyzing the log to be analyzed;
if the node type is an output parameter node, outputting list parameters after log analysis;
if the node type is the end node, the analysis process is ended.
In the embodiment, when the node type is identified as the input parameter node, corresponding analysis operation is executed, and analysis processing is performed on the log to be analyzed; when the node type is identified as an output parameter node, the analysis work of the log is processed, the output operation of the log is executed, and list parameters are output according to a log template; and when the node type is the end node, directly ending the whole analysis process. Note that the end node is an end pointer.
In some embodiments of the present invention, if the node type is an input parameter node, parsing the log to be parsed includes:
identifying the format and the field name of the log to be analyzed;
separating the log to be analyzed according to the log format to be analyzed to obtain a separator list;
and analyzing the log to be analyzed according to the separator list and the field name to obtain a log analysis result.
In the above embodiment, the log type is recognized first, and the log type may be in a text and JSON format, or in other formats. And processing the separator nodes and the field name nodes in the log to be analyzed to obtain an analyzed separator list, and realizing log analysis through the separator list.
In some embodiments of the present invention, parsing the to-be-parsed log according to the separator list and the field name to obtain a log parsing result, including:
associating the field name with a separator list to obtain a separator analysis result;
and analyzing the log to be analyzed according to the separator analysis result to obtain a log analysis result.
In the above embodiment, the parameters corresponding to the separators in the separator list in the log to be analyzed are associated, and the separators are analyzed to obtain a separator analysis result. It can be understood that the field names are multiple, the log parsing process needs to associate the field names with the separator list to obtain multiple separator parsing results, and is a process of repeating association and separation processing, and the log to be parsed is completely parsed by loop processing to finally obtain a log parsing result, i.e. the parsing of the log is realized.
It can be understood that, since the log content may include various parameter information, when the log is parsed, it is necessary to perform a repeated process, and data of the same type in the log is repeatedly sorted.
In some embodiments of the present invention, formatting the log parsing data according to an output log format to obtain an output log, and sending the output log to a log center, includes:
outputting the output log to a log channel;
and sending the output log to a corresponding log center for storage according to the channel configuration of the log channel.
In the above embodiment, after performing log analysis processing on the log to be analyzed, an output log is obtained, the output log is output to a log channel, the log channel configures a channel for the output log, and sets log center information, that is, sets a storage address, and then sends the output log and the storage address to a log center for storage.
In some embodiments of the invention, the input log format comprises only one format and the output log format comprises a plurality of formats.
In the above embodiment, the input log format can only correspond to one format, that is, each log template can only accept the input of logs in one format, and when the input log formats are different, only the log template is re-established to perform log analysis; the output log format can have a plurality of different formats to meet the output requirements of different formats.
The invention also provides an embodiment for analyzing the plain format log, which comprises the following steps:
the method comprises the steps of obtaining a log to be analyzed from the safety equipment, wherein example data of the log to be analyzed are as follows:
02/22/2022-19:03:42.971776[ ] 1:955193:0] Linux vector rpc.statd Remote Format String Vulnerability [ ] Classification: Web attack ] [ Priority:6] { UDP }10.0.0.158:807- >10.0.0.163:956
Separating the input according to the separation of the log template example data, and the separated token list results are as follows:
1 02/22/2022-19:03:42.971776
2[1:955193:0]Linux Vendor rpc.statd Remote Format String Vulnerability
3[ Classification: Web attack ] [ Priority:6] { UDP }10.0.0.158:807- >10.0.0.163:956
Then, associating the separated token lists according to the following content of the log template sample data;
{1} (attack time,% D)
{2} (attack description,% s
{3} (attack details,% s)
After correlation, the results of parsing the log are as follows:
{1} (attack time, 02/22/2022-19:03: 42.971776)
{2} (attack description), [1:955193:0] Linux Vendor rpc.stat Remote Format String Vulnerability
(3) {3} (attack details), [ Priority:6] { UDP }10.0.0.158:807- >10.0.0.163:956 } Classification: Web attack ] [ Priority:6] (UDP) }10.0.0.158:807- ]
According to "{ 3}. separation" ", the result of {3} after the association needs to be further separated, the separator is a space, and the separated token list result is as follows:
1[ Classification: Web attack ]
2[Priority:6]
3{UDP}
4 10.0.0.158:807
5 ->
6 10.0.0.163:956
And repeatedly carrying out separation and association processing on the log content, and further extracting log data:
{3.1} -, attack type, [ Classification:% s ]
{3.2} -, Priority, [ Priority:% d ]
{3.3} -, protocol type, {% s }
{3.4} ═ source IP/port,% s
{3.6} -, destination IP/port,% s
After all log contents are processed, the analysis result is as follows:
attack time, 02/22/2022-19:03:42.971776
{2}, attack description, [1:955193:0] Linux vector rpc
(3) {3} (attack details), [ Priority:6] { UDP }10.0.0.158:807- >10.0.0.163:956 } Classification: Web attack ] [ Priority:6] (UDP) }10.0.0.158:807- ]
{3.1}, a type of attack, a Web attack
{3.2} -, priority, 6
{3.3} ═ protocol type, UDP
{3.4} ═ source IP/port, 10.0.0.158:807
Destination IP/port, 10.0.0.163:956
{3.4.1} ═ source IP, 10.0.0.15
{3.4.2}, source port, 807
Destination IP 10.0.0.163
Destination port, 956
After all input _ log _ info child nodes of log template sample data are processed, the key-value pair (key, value) structural arrangement of the log is completed, and when the subsequent output _ log _ info outputs the log, the key is adopted to directly use value assembly:
{ attack time } -, 02/22/2022-19:03:42.971776
{ attack description } - [1:955193:0] Linux Vendor rpc.stat Remote Format String Vulneravailability
{ attack details } - [ Classification: Web attack ] [ Priority:6] { UDP }10.0.0.158:807- >10.0.0.163:956
{ attack type } ═ Web attack
{ priority } ═ 6
{ protocol type } ═ UDP
{ source IP/port } ═ 10.0.0.158:807
{ destination IP/port } - { 10.0.0.163:956 }
{ source IP } ═ 10.0.0.15
Source port 807
Destination IP 10.0.0.163
{ destination port } - { 956
Therefore, the analysis of the log in the plain format is completed, and the log processing process of other formats is similar, which is not described herein again.
In order to better implement the dynamic log parsing method in the embodiment of the present invention, on the basis of the dynamic log parsing method, correspondingly, please refer to fig. 2, fig. 2 is a schematic structural diagram of an embodiment of a dynamic log parsing apparatus provided in the present invention, and an embodiment of the present invention provides a dynamic log parsing apparatus 200, including:
a configuration module 201, configured to configure a log template according to a log format of the security device;
the template analysis module 202 is configured to obtain a log to be analyzed of the security device, and analyze the log template to obtain structured data of the log template; the structured data comprises an input log format and an output log format;
the log analysis module 203 is configured to analyze the log to be analyzed according to the input log format to obtain log analysis data;
and the output module 204 is configured to format the log analysis data according to the output log format to obtain an output log, and send the output log to a log center.
Here, it should be noted that: the apparatus 200 provided in the foregoing embodiment may implement the technical solutions described in the foregoing method embodiments, and the specific implementation principles of the foregoing modules or units may refer to the corresponding contents in the foregoing method embodiments, which are not described herein again.
Referring to fig. 3, fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the invention. Based on the above dynamic log analysis method, the invention also provides a dynamic log analysis device, which can be a mobile terminal, a desktop computer, a notebook, a palm computer, a server and other computing devices. The dynamic log resolution device includes a processor 310, a memory 320, and a display 330. Fig. 3 shows only some of the components of the electronic device, but it is to be understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead.
The storage 320 may be an internal storage unit of the dynamic log parsing device in some embodiments, such as a hard disk or a memory of the dynamic log parsing device. The memory 320 may also be an external storage device of the dynamic log parsing device in other embodiments, such as a plug-in hard disk provided on the dynamic log parsing device, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like. Further, the memory 320 may also include both an internal storage unit of the dynamic log parsing device and an external storage device. The memory 320 is used for storing application software installed in the dynamic log parsing apparatus and various data, such as program codes for installing the dynamic log parsing apparatus. The memory 320 may also be used to temporarily store data that has been output or is to be output. In an embodiment, the memory 320 stores a dynamic log parsing program 340, and the dynamic log parsing program 340 can be executed by the processor 310, so as to implement the dynamic log parsing method according to the embodiments of the present application.
Processor 310 may be, in some embodiments, a Central Processing Unit (CPU), microprocessor or other data Processing chip configured to execute program code stored in memory 320 or process data, such as performing dynamic log parsing methods.
The display 330 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch panel, or the like in some embodiments. The display 330 is used to display information at the dynamic log parsing device and to display a user interface for visualization. The components 310 and 330 of the dynamic log parsing device communicate with each other via a system bus.
In one embodiment, the steps in the dynamic log resolution method described above are implemented when processor 310 executes dynamic log resolution program 340 in memory 320.
The present embodiment also provides a computer readable storage medium having a dynamic log parsing program stored thereon, which when executed by a processor, implements the steps of:
configuring a log template according to the log format of the safety equipment;
acquiring a log to be analyzed of the safety equipment, and analyzing the log template to obtain structured data of the log template; the structured data comprises an input log format and an output log format;
analyzing the log to be analyzed according to the input log format to obtain log analysis data;
and formatting the log analysis data according to the output log format to obtain an output log, and sending the output log to a log center.
In summary, according to the dynamic log parsing method, apparatus, device, and storage medium provided in this embodiment, a corresponding log template is configured according to a log format of a security device, a log to be parsed is parsed through the log template to obtain an output log, the log template is pre-loaded and parsed into structured data, which does not reduce the efficiency of log parsing due to an increase in log formats.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention.

Claims (10)

1. A method for dynamic log parsing, comprising:
configuring a log template according to the log format of the safety equipment;
acquiring a log to be analyzed of the safety equipment, and analyzing the log template to obtain structured data of the log template; the structured data comprises an input log format and an output log format;
analyzing the log to be analyzed according to the input log format to obtain log analysis data;
and formatting the log analysis data according to the output log format to obtain an output log, and sending the output log to a log center.
2. The dynamic log parsing method of claim 1, wherein parsing the log template to obtain the structured data of the log template comprises:
acquiring an input request of the log template, and inputting the log template according to the input request;
and circularly analyzing the log template to obtain the node type of the structured data.
3. The dynamic log parsing method according to claim 2, wherein parsing the log to be parsed according to the input log format to obtain log parsing data comprises:
if the node type is an input parameter node, analyzing the log to be analyzed;
if the node type is an output parameter node, outputting list parameters after log analysis;
and if the node type is the end node, ending the analysis process.
4. The method according to claim 3, wherein if the node type is an input parameter node, the analyzing the log to be analyzed includes:
identifying the format and the field name of the log to be analyzed;
separating the log to be analyzed according to the log format to be analyzed to obtain a separator list;
and analyzing the log to be analyzed according to the separator list and the field name to obtain a log analysis result.
5. The method according to claim 4, wherein the parsing the log to be parsed according to the delimiter list and the field name to obtain a log parsing result, comprises:
associating the field name with the separator list to obtain a separator analysis result;
and analyzing the log to be analyzed according to the separator analysis result to obtain a log analysis result.
6. The dynamic log parsing method of claim 1, wherein the formatting the log parsing data according to the output log format to obtain an output log, and sending the output log to a log center comprises:
outputting the output log to a log channel;
and sending the output log to a corresponding log center for storage according to the channel configuration of the log channel.
7. The dynamic log parsing method of any one of claims 1-6, wherein the input log format comprises only one format and the output log format comprises a plurality of formats.
8. A dynamic log parsing apparatus, comprising:
the configuration module is used for configuring the log template according to the log format of the safety equipment;
the template analysis module is used for acquiring a log to be analyzed of the safety equipment and analyzing the log template to obtain structured data of the log template; the structured data comprises an input log format and an output log format;
the log analysis module is used for analyzing the log to be analyzed according to the input log format to obtain log analysis data;
and the output module is used for formatting the log analysis data according to the output log format to obtain an output log and sending the output log to a log center.
9. An electronic device comprising a memory and a processor, wherein,
the memory is used for storing programs;
the processor, coupled to the memory, is configured to execute the program stored in the memory to implement the steps in the dynamic log parsing method of any one of the above claims 1 to 7.
10. A computer-readable storage medium storing a computer-readable program or instructions, which when executed by a processor, implement the steps of the dynamic log parsing method as claimed in any one of claims 1 to 7.
CN202210461097.0A 2022-04-28 2022-04-28 Dynamic log analysis method, device, equipment and storage medium Active CN114785604B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210461097.0A CN114785604B (en) 2022-04-28 2022-04-28 Dynamic log analysis method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210461097.0A CN114785604B (en) 2022-04-28 2022-04-28 Dynamic log analysis method, device, equipment and storage medium

Publications (2)

Publication Number Publication Date
CN114785604A true CN114785604A (en) 2022-07-22
CN114785604B CN114785604B (en) 2023-11-07

Family

ID=82435061

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210461097.0A Active CN114785604B (en) 2022-04-28 2022-04-28 Dynamic log analysis method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114785604B (en)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160292263A1 (en) * 2015-04-03 2016-10-06 Oracle International Corporation Method and system for implementing a log parser in a log analytics system
CN106055450A (en) * 2016-05-20 2016-10-26 北京神州绿盟信息安全科技股份有限公司 Binary log analysis method and apparatus
CN106815306A (en) * 2016-12-16 2017-06-09 中铁程科技有限责任公司 Daily record analysis method and device
KR20180061891A (en) * 2016-11-30 2018-06-08 정준용 Log generator and big data analysis preprocessing system including the log generator
CN110826299A (en) * 2019-10-25 2020-02-21 上海工业自动化仪表研究院有限公司 General template log analysis method based on classification
CN111708860A (en) * 2020-06-15 2020-09-25 北京优特捷信息技术有限公司 Information extraction method, device, equipment and storage medium
CN112162965A (en) * 2020-10-12 2021-01-01 平安科技（深圳）有限公司 Log data processing method and device, computer equipment and storage medium
CN112632960A (en) * 2021-01-06 2021-04-09 北京启明星辰信息安全技术有限公司 Log analysis method and system based on dynamic field template
US20220092062A1 (en) * 2015-04-03 2022-03-24 Oracle International Corporation Method and system for implementing a log parser in a log analytics system

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160292263A1 (en) * 2015-04-03 2016-10-06 Oracle International Corporation Method and system for implementing a log parser in a log analytics system
US20220092062A1 (en) * 2015-04-03 2022-03-24 Oracle International Corporation Method and system for implementing a log parser in a log analytics system
CN106055450A (en) * 2016-05-20 2016-10-26 北京神州绿盟信息安全科技股份有限公司 Binary log analysis method and apparatus
KR20180061891A (en) * 2016-11-30 2018-06-08 정준용 Log generator and big data analysis preprocessing system including the log generator
CN106815306A (en) * 2016-12-16 2017-06-09 中铁程科技有限责任公司 Daily record analysis method and device
CN110826299A (en) * 2019-10-25 2020-02-21 上海工业自动化仪表研究院有限公司 General template log analysis method based on classification
CN111708860A (en) * 2020-06-15 2020-09-25 北京优特捷信息技术有限公司 Information extraction method, device, equipment and storage medium
CN112162965A (en) * 2020-10-12 2021-01-01 平安科技（深圳）有限公司 Log data processing method and device, computer equipment and storage medium
CN112632960A (en) * 2021-01-06 2021-04-09 北京启明星辰信息安全技术有限公司 Log analysis method and system based on dynamic field template

Also Published As

Publication number Publication date
CN114785604B (en) 2023-11-07

Similar Documents

Publication Publication Date Title
US11252168B2 (en) System and user context in enterprise threat detection
US9529662B1 (en) Dynamic rule-based automatic crash dump analyzer
US7984373B2 (en) EDI instance based transaction set definition
US9483583B2 (en) Syslog parser
CN111683066B (en) Heterogeneous system integration method, heterogeneous system integration device, computer equipment and storage medium
CN111131320B (en) Asset identification method, device, system and medium
US8799923B2 (en) Determining relationship data associated with application programs
WO2021218144A1 (en) Data processing method and apparatus, computer device, and storage medium
CN110753050B (en) Method and device for generating protocol document, computer storage medium and electronic equipment
US8327324B1 (en) Message logging system
US10979295B2 (en) Automatically discovering topology of an information technology (IT) infrastructure
CN112395843A (en) PHP code-based service processing method, device, equipment and medium
US8984124B2 (en) System and method for adaptive data monitoring
CN116860350A (en) Jailhouse tool configuration method, electronic device and computer readable storage medium
US11552868B1 (en) Collect and forward
CN111680288A (en) Command execution method, device and equipment for container and storage medium
CN115357286B (en) Program file comparison method and device, electronic equipment and storage medium
CN114785604B (en) Dynamic log analysis method, device, equipment and storage medium
US8856152B2 (en) Apparatus and method for visualizing data
CN115033451A (en) Data generation method, data processing device, electronic device, and medium
WO2010025062A1 (en) Automatic test map generation for system verification test
CN114186958A (en) Method, computing device and storage medium for exporting list data as spreadsheet
CN111294232A (en) Client server model for multiple document editors
CN117171800B (en) Sensitive data identification method and device based on zero trust protection system
CN112162738B (en) Data conversion method and device, terminal equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant