CN114785604B - Dynamic log analysis method, device, equipment and storage medium - Google Patents

Dynamic log analysis method, device, equipment and storage medium Download PDF

Info

Publication number
CN114785604B
CN114785604B CN202210461097.0A CN202210461097A CN114785604B CN 114785604 B CN114785604 B CN 114785604B CN 202210461097 A CN202210461097 A CN 202210461097A CN 114785604 B CN114785604 B CN 114785604B
Authority
CN
China
Prior art keywords
log
format
template
analyzed
output
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210461097.0A
Other languages
Chinese (zh)
Other versions
CN114785604A (en
Inventor
柯明明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Anbotong Jin'an Technology Co ltd
Original Assignee
Beijing Anbotong Jin'an Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Anbotong Jin'an Technology Co ltd filed Critical Beijing Anbotong Jin'an Technology Co ltd
Priority to CN202210461097.0A priority Critical patent/CN114785604B/en
Publication of CN114785604A publication Critical patent/CN114785604A/en
Application granted granted Critical
Publication of CN114785604B publication Critical patent/CN114785604B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
    • H04L63/1425—Traffic logging, e.g. anomaly detection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
    • H04L63/1416—Event detection, e.g. attack signature detection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/22—Parsing or analysis of headers

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Hardware Design (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The application relates to a dynamic log analysis method, a device, equipment and a storage medium, wherein the method comprises the following steps: configuring a log template according to a log format of the security device; the method comprises the steps of obtaining a log to be analyzed of the security equipment, and analyzing a log template to obtain structured data of the log template; the structured data includes an input log format and an output log format; according to the input log format, analyzing the log to be analyzed to obtain log analysis data; according to the output log format, formatting the log analysis data to obtain an output log, and sending the output log to a log center. According to the dynamic log analysis method provided by the application, the log is analyzed through the log template, the log template is preloaded and analyzed into the structured data, the analysis efficiency is not reduced due to the increase of the log format, the analysis personnel are opened to write by adopting the special format of the irregular expression, the error probability is reduced, and the maintenance cost is reduced.

Description

Dynamic log analysis method, device, equipment and storage medium
Technical Field
The present application relates to the field of computer networks, and in particular, to a method, an apparatus, a device, and a storage medium for analyzing a dynamic log.
Background
Any device or service in the information security system may output a log: firewall, web server, IPS engine, database and terminal software, and the log contains a large amount of information of interest to security manager, operation and maintenance personnel, service analysis personnel, etc. Taking the example of a user behavior aware device, for example: visitor IP, access time, destination resource of access, client information of visitor, access times, etc.
When the log data size is small, various built-in or open-source tools such as gawk, grep, cat which are ready for the operating system are all sharp instruments for log analysis. Various scripting languages, such as Python, shell, can be used if there is more complex logic, and basically all the problems can be solved, but these tools or scripting languages rely mostly on regular expressions.
With the gradual increase of the system body, the types of equipment contained in a system are increased, when thousands and tens of thousands of log formats are required to be generated for tens and hundreds of different equipment, the regular expression-based method is proved to be a fly-through method under the condition that the log formats are gradually increased, and the regular expression is difficult to maintain due to the fact that the expertise becomes abnormal, the efficiency is low due to the fact that the regular expression is light, the maintenance cost is increased, and the mishandling is seriously caused.
Disclosure of Invention
In view of the foregoing, it is necessary to provide a method, apparatus, device and storage medium for dynamic log parsing, which are used for solving the problems of low efficiency, high maintenance cost and possible errors in log parsing by regular expressions when log data is large in size in the prior art.
In order to achieve the technical purpose, the application adopts the following technical scheme:
in a first aspect, the present application provides a method for parsing a dynamic log, including:
configuring a log template according to a log format of the security device;
the method comprises the steps of obtaining a log to be analyzed of the security equipment, and analyzing a log template to obtain structured data of the log template; the structured data includes an input log format and an output log format;
according to the input log format, analyzing the log to be analyzed to obtain log analysis data;
according to the output log format, formatting the log analysis data to obtain an output log, and sending the output log to a log center.
Preferably, the log template is parsed to obtain structured data of the log template, including:
acquiring an input request of a log template, and inputting the log template according to the input request;
and circularly analyzing the log template to obtain the node type of the structured data.
Preferably, according to an input log format, the log to be parsed is parsed to obtain log parsed data, including:
if the node type is the input parameter node, analyzing the log to be analyzed;
if the node type is the output parameter node, outputting the list parameter after log analysis;
if the node type is the ending node, ending the analysis flow.
Preferably, if the node type is an input parameter node, performing parsing processing on the log to be parsed, including:
identifying the format and field name of the log to be analyzed;
separating the logs to be analyzed according to the log format to be analyzed to obtain a separator list;
and analyzing the log to be analyzed according to the separator list and the field name to obtain a log analysis result.
Preferably, the analyzing the log to be analyzed according to the separator list and the field name to obtain a log analyzing result includes:
associating the field name with the separator list to obtain a separator analysis result;
and analyzing the log to be analyzed according to the separator analysis result to obtain a log analysis result.
Preferably, according to the output log format, the log analysis data is formatted to obtain an output log, and the output log is sent to a log center, including:
outputting the output log to a log channel;
and sending the output log to a corresponding log center for storage according to the channel configuration of the log channel.
Preferably, the input log format contains only one format and the output log format contains multiple formats.
In a second aspect, the present application further provides a dynamic log parsing apparatus, including:
the configuration module is used for configuring a log template according to the log format of the security equipment;
the template analysis module is used for acquiring a log to be analyzed of the security equipment, analyzing the log template and obtaining structured data of the log template; the structured data includes an input log format and an output log format;
the log analysis module is used for analyzing the log to be analyzed according to the input log format to obtain log analysis data;
and the output module is used for formatting the log analysis data according to the output log format to obtain an output log and sending the output log to a log center.
In a third aspect, the application also provides an electronic device comprising a memory and a processor, wherein,
a memory for storing a program;
and the processor is coupled with the memory and is used for executing the program stored in the memory to realize the steps in the dynamic log parsing method in any implementation mode.
In a fourth aspect, the present application further provides a computer readable storage medium storing a computer readable program or instructions, where the program or instructions, when executed by a processor, implement the steps in the method for dynamic log parsing in any of the above implementations.
The beneficial effects of adopting the embodiment are as follows: according to the dynamic log analysis method, the device, the equipment and the storage medium, the corresponding log template is configured according to the log format of the security equipment, analysis of the log to be analyzed is realized through the log template, the output log is obtained, the log template is preloaded and analyzed into the structured data, the efficiency of log analysis cannot be reduced due to the fact that the log format is increased, compared with a regular expression, the requirement of writing of the log template on the specialization is low, a large number of writing is not needed, the log template is opened for the analyst to write, the error probability is reduced, and the maintenance cost is reduced.
Drawings
FIG. 1 is a flow chart of an embodiment of a method for dynamic log parsing according to the present application;
FIG. 2 is a schematic diagram illustrating a structure of an embodiment of a dynamic log parsing apparatus according to the present application;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The following detailed description of preferred embodiments of the application is made in connection with the accompanying drawings, which form a part hereof, and together with the description of the embodiments of the application, are used to explain the principles of the application and are not intended to limit the scope of the application.
In the description of the present application, the meaning of "a plurality" is two or more, unless explicitly defined otherwise.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.
Before describing embodiments of the present application, related terms will be explained:
UUID: a universally unique identification code (Universally Unique Identifier);
IPS: an intrusion prevention system (Intrusion Prevention System);
gawk: a program language for modifying, comparing, extracting the data in the text file;
grep: global search Regular Expression (RE) and print out the line, searching through regular expressions and printing lines out, is a powerful text search tool that can search text using regular expressions and print out matching lines;
cat: the full command is used to connect the file and print onto a standard output device.
The application provides a dynamic log analysis method, a device, equipment and a storage medium, which are respectively described below.
Referring to fig. 1, fig. 1 is a flow chart of an embodiment of a dynamic log parsing method provided by the present application, and a specific embodiment of the present application discloses a dynamic log parsing method, which includes:
s101, configuring a log template according to a log format of security equipment;
s102, acquiring a log to be analyzed of the security equipment, and analyzing a log template to obtain structured data of the log template; the structured data includes an input log format and an output log format;
s103, according to the input log format, analyzing the log to be analyzed to obtain log analysis data;
s104, according to the format of the output log, formatting the log analysis data to obtain the output log, and sending the output log to a log center.
In a specific embodiment of the present application, step S101 configures a log template for each security device according to the security device, and extracts a log to be parsed from the security device. Since the log formats of the security devices may be different, for the security devices with different formats, a log template with a corresponding format needs to be configured for each security device, and log analysis is implemented through the log template.
The application provides a specific embodiment of a configuration log template, which comprises the following steps of compiling requirement information of a log to be analyzed by an analyst according to the following template:
input_log_info (input log format information)
type (log type) =text (plan), JSON (JSON), auto (auto);
separation (= space (), comma (,), semicolon (;) user-defined default to comma (,)
{ N } = field name, format string (may not be filled in, default is parsed by string)
{ N }. Separation (= space (), comma (,), semicolon (;) user-defined default to comma (,)
output_log_info (output log format information) (there may be multiple nodes to indicate output of multiple format requirements, see log template example data 3 illustrated in the accompanying drawings)
separation (= space (), comma (,), semicolon (;) user-defined default to comma (,)
{ input Log field name } List, output field name, format string List (may not be output, default is parsed according to string)
{ log_center }, array list, log center information may be configured for sending the merged log to the corresponding log center, for example: configuring domain name or IP address information of the log center, log_center= [192.168.1.1:2088, test.log.com:8088].
N exists in two formats:
type is play, format%num, NUM supports 1-20 (this patent exemplifies a maximum of 20 parameters), and the subparameters of the X parameter can be represented by x.y, for example: {%2.1}, the first subparameter of the 2 nd parameter.
type is json, format is. X.y, for example: alert_id, representing the acquisition of alert node's signature_id data 955193.
The separation represents the delimiter of the nth parameter, requiring that the nth parameter be of the string type.
Format string support includes, but is not limited to, substantially covering 95% journaling, and supporting customization:
% T, timestamp (relative to the number of seconds elapsed in 1970, 1 month, 1 day, 00:00:00), for example: 1645528793776.
% D, date, e.g.: 02/22/2022-19:03:43.068719, 2022-02-22T19:03:42.971776+0800.
% M, MAC address, e.g.: AF:24:54:4A:BB:43.
% P, IP address, e.g.: 192.168.1.1.
% U, UUID, a universally unique identification code for a device or service, such as: 783a6c01-99e1-5f7e-9b06-24a7d415c4a2.
% d, the shaped number, also can be adapted to the hexadecimal number with 0X or 0X.
% s, character string.
% f, floating point number.
% x, hexadecimal shaping number.
% A, a well-known port is shown as an application or service.
The { input log field name } list must be defined in "{ N } = field name" under the input_log_info node, for example: { attack time }, { Source IP }/{ Source Port }, { Source IP }: { Source Port }, are all reasonable.
It can be appreciated that the present application can write corresponding log templates according to a plurality of log formats to achieve parsing of a plurality of different log formats.
In a specific embodiment of the present application, after the log template is configured in step S102, the log template is parsed, and each part of the log template is known, so that the data of the log to be parsed is conveniently parsed. It should be noted that, the log format of the input may be text (play), JSON (JSON), etc., and the present application does not limit the possible log format of the input.
In a specific embodiment of the present application, step S103 inputs log format as input_log_info, determines information such as log type, separator, field name, etc. by inputting information of log format, parses out corresponding information in log to be parsed, obtains log parsed data, and sends the log parsed data to an output module.
In a specific embodiment of the present application, step S104 formats the log analysis data according to the output log format, so that the output log format is output according to the set format, and the output log format is stored in the log center.
Compared with the regular expression, the method for analyzing the dynamic log has the advantages that the requirement on the professionality of the writing of the log template is low, a large amount of writing is not needed, the log template is opened for the writing of analysts, the error probability is reduced, and the maintenance cost is reduced.
In some embodiments of the present application, parsing the log template to obtain structured data of the log template includes:
acquiring an input request of a log template, and inputting the log template according to the input request;
and circularly analyzing the log template to obtain the node type of the structured data.
In the above embodiment, before the log template is parsed, it is necessary to wait for an input request of the log template, and after the input request of the log template is obtained, the corresponding log template is input according to the request. And carrying out cyclic analysis on each row of the log template, identifying the node type of the structured data, wherein the node type determines the corresponding operation of log analysis, and realizing the analysis of the log based on the node type.
In some embodiments of the present application, according to an input log format, a log to be parsed is parsed to obtain log parsed data, including:
if the node type is the input parameter node, analyzing the log to be analyzed;
if the node type is the output parameter node, outputting the list parameter after log analysis;
if the node type is the ending node, ending the analysis flow.
In the above embodiment, when the node type is identified as the input parameter node, the corresponding parsing operation is executed, and the log to be parsed is parsed; when the node type is identified as the output parameter node, the analysis of the log is finished, the output operation of the log is executed, and the list parameters are output according to the log template; and when the node type is the ending node, directly ending the whole analysis flow. The end node is an end symbol.
In some embodiments of the present application, if the node type is an input parameter node, performing parsing processing on the log to be parsed includes:
identifying the format and field name of the log to be analyzed;
separating the logs to be analyzed according to the log format to be analyzed to obtain a separator list;
and analyzing the log to be analyzed according to the separator list and the field name to obtain a log analysis result.
In the above embodiment, the log type is first identified, and the log type may be text and JSON format, or may be other formats, and the present application does not limit the possible types of the log. And processing the separator node and the field name node in the log to be analyzed to obtain a separated list after analysis, and realizing log analysis through the separated list.
In some embodiments of the present application, the parsing of the log to be parsed according to the separator list and the field name to obtain the log parsing result includes:
associating the field name with the separator list to obtain a separator analysis result;
and analyzing the log to be analyzed according to the separator analysis result to obtain a log analysis result.
In the above embodiment, parameters corresponding to separators in the separator list in the log to be analyzed are associated, and the separator is analyzed, so as to obtain a separator analysis result. It can be understood that the number of field names is plural, and the process of resolving the log needs to correlate the field names with the separator list to obtain a plurality of separator resolving results, which is a process of repeating the correlation and separation processes, and the log to be resolved is completely resolved through cyclic processing, so as to finally obtain a log resolving result, namely, achieve log resolving.
It can be appreciated that, since the log content may contain various parameter information, the analysis of the log must be a repeated process, and the same type of data in the log is repeatedly arranged.
In some embodiments of the present application, according to an output log format, formatting log analysis data to obtain an output log, and sending the output log to a log center, including:
outputting the output log to a log channel;
and sending the output log to a corresponding log center for storage according to the channel configuration of the log channel.
In the above embodiment, after log analysis processing is performed on the log to be analyzed, an output log is obtained, the output log is output to a log channel, the log channel is configured as an output log, log center information is set, that is, a storage address is set, and then the output log and the storage address are sent to a log center for storage.
In some embodiments of the application, the input log format contains only one format and the output log format contains multiple formats.
In the above embodiment, the input log format can only correspond to one format, that is, each log template can only accept the input of a log in one format, and when the input log formats are different, log analysis can only be performed by reestablishing the log template; while the output log format may have a variety of different formats to accommodate output requirements of different formats.
The application also provides an embodiment for analyzing the plain format log, which comprises the following steps:
the method comprises the steps of obtaining a log to be analyzed from a safety device, wherein example data of the log to be analyzed are specifically as follows:
02/22/2022-19:03:42.971776[ [1:955193:0]Linux Vendor rpc.statd Remote Format String Vulnerability ] [ Classification ] Web attack ] [ Priority:6] { UDP }10.0.0.158:807- >10.0.0.163:956
According to the separation= '[ x ]', of log template example data, the input is separated, and the separated token list results are as follows:
1 02/22/2022-19:03:42.971776
2[1:955193:0]Linux Vendor rpc.statd Remote Format String Vulnerability
3[ classification: web attack ] [ Priority:6] { UDP } 10.0.158:807- >10.0.0.163:956
Then according to the following example data of the log template, associating the separated token list;
{1} = attack time,%d
{2} = attack description,% s
{3} = attack details,% s
After correlation, the results of parsing the log are as follows:
{1} = attack time, 02/22/2022-19:03:42.971776
{2} = attack description, [1:955193:0]Linux Vendor rpc.statd Remote Format String Vulnerability ]
{3} = attack details, [ Classification: web attack ] [ Priority:6] { UDP } 10.0.158:807- >10.0.0.163:956
According to "{3}. Separation=" "of log template example data, the separation processing needs to be further performed on the result of {3} after the association, the separator is a space, and the separated token list results are as follows:
1[ classification: web attack ]
2[Priority:6]
3{UDP}
4 10.0.0.158:807
5 ->
6 10.0.0.163:956
Repeatedly carrying out separation and association processing on the log content, and further extracting log data:
{3.1} = attack type, [ Classification:% s ]
{3.2} = Priority } - [ Priority:% d ]
{3.3} = protocol type, {% s }
{3.4} = source IP/port,% s
{3.6} = destination IP/port,% s
After all log contents are processed, the analysis results are as follows:
time of attack, 02/22/2022-19:03:42.971776
{2} = attack description, [1:955193:0]Linux Vendor rpc.statd Remote Format String Vulnerability ]
{3} = attack details, [ Classification: web attack ] [ Priority:6] { UDP } 10.0.158:807- >10.0.0.163:956
{3.1} = attack type, web attack
{3.2} = priority, 6
{3.3} = protocol type, UDP
{3.4} = source IP/port, 10.0.0.158:807
{3.6} = destination IP/port, 10.0.0.163:956
{3.4.1} = source IP,10.0.0.15
{3.4.2} = source port, 807
{3.6.1} = destination IP,10.0.0.163
{3.6.2} = destination port, 956
After the input_log_info child nodes of the log template example data are processed, the key-value pair (key, value) structure of the log is completed, and when the log is output by the subsequent output_log_info, the key is directly used for assembly, so that the log is assembled by the value:
{ attack time } = 02/22/2022-19:03:42.971776
{ attack description } = [1:955193:0]Linux Vendor rpc.statd Remote Format String Vulnerability ]
{ attack details } = [ Classification: web attack ] [ Priority:6] { UDP } 10.0.158:807- >10.0.0.163:956
{ attack type } = Web attack
{ priority } = 6
{ protocol type } = UDP
{ Source IP/Port } = 10.0.0.158:807
{ destination IP/port } = 10.0.0.163:956
{ Source IP } = 10.0.0.15
{ Source Port } = 807
{ destination IP } = 10.0.0.163
{ destination port } = 956
The analysis of the log in the plan format is completed, and the log processing process of other formats is similar, and the application is not repeated here.
In order to better implement the dynamic log parsing method according to the embodiment of the present application, referring to fig. 2 correspondingly, fig. 2 is a schematic structural diagram of an embodiment of a dynamic log parsing device according to the present application, where the embodiment of the present application provides a dynamic log parsing device 200, including:
a configuration module 201, configured to configure a log template according to a log format of the security device;
the template analysis module 202 is configured to obtain a log to be analyzed of the security device, analyze the log template, and obtain structured data of the log template; the structured data includes an input log format and an output log format;
the log parsing module 203 is configured to parse the log to be parsed according to the input log format, so as to obtain log parsed data;
and the output module 204 is configured to perform formatting processing on the log analysis data according to the output log format, obtain an output log, and send the output log to a log center.
What needs to be explained here is: the device 200 provided in the foregoing embodiments may implement the technical solutions described in the foregoing method embodiments, and the specific implementation principles of the foregoing modules or units may be referred to the corresponding content in the foregoing method embodiments, which is not described herein again.
Referring to fig. 3, fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the application. Based on the dynamic log analysis method, the application also correspondingly provides dynamic log analysis equipment which can be computing equipment such as a mobile terminal, a desktop computer, a notebook computer, a palm computer, a server and the like. The dynamic log parsing apparatus includes a processor 310, a memory 320, and a display 330. Fig. 3 shows only some of the components of the electronic device, but it should be understood that not all of the illustrated components are required to be implemented and that more or fewer components may be implemented instead.
Memory 320 may be an internal storage unit of the dynamic log parsing device in some embodiments, such as a hard disk or memory of the dynamic log parsing device. The memory 320 may also be an external storage device of the dynamic log parsing device in other embodiments, such as a plug-in hard disk, smart Media Card (SMC), secure Digital (SD) Card, flash Card (Flash Card) or the like. Further, the memory 320 may also include both internal storage units and external storage devices of the dynamic log parsing device. The memory 320 is used for storing application software installed in the dynamic log analyzing device and various data, such as program codes for installing the dynamic log analyzing device. The memory 320 may also be used to temporarily store data that has been output or is to be output. In one embodiment, the memory 320 stores a dynamic log parsing program 340, and the dynamic log parsing program 340 is executable by the processor 310 to implement the dynamic log parsing method according to the embodiments of the present application.
The processor 310 may in some embodiments be a central processing unit (Central Processing Unit, CPU), microprocessor or other data processing chip for executing program code or processing data stored in the memory 320, such as performing a dynamic log parsing method or the like.
The display 330 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, or the like in some embodiments. Display 330 is used to display information at the dynamic log parsing device and to display a visual user interface. The components 310-330 of the dynamic log parsing device communicate with each other over a system bus.
In one embodiment, the steps in the dynamic log parsing method as described above are implemented when processor 310 executes dynamic log parsing program 340 in memory 320.
The present embodiment also provides a computer-readable storage medium having stored thereon a dynamic log parsing program that when executed by a processor performs the steps of:
configuring a log template according to a log format of the security device;
the method comprises the steps of obtaining a log to be analyzed of the security equipment, and analyzing a log template to obtain structured data of the log template; the structured data includes an input log format and an output log format;
according to the input log format, analyzing the log to be analyzed to obtain log analysis data;
according to the output log format, formatting the log analysis data to obtain an output log, and sending the output log to a log center.
In summary, according to the method, the device, the equipment and the storage medium for analyzing the dynamic log, the corresponding log template is configured according to the log format of the security equipment, the log to be analyzed is analyzed through the log template, the output log is obtained, the log template is preloaded and analyzed into the structured data, the efficiency of log analysis cannot be reduced due to the increase of the log format, compared with a regular expression, the requirement of writing of the log template on the professionality is low, a large number of writing is not needed, the log template is opened for the analyst to write, the error probability is reduced, and the maintenance cost is reduced.
The present application is not limited to the above-mentioned embodiments, and any changes or substitutions that can be easily understood by those skilled in the art within the technical scope of the present application are intended to be included in the scope of the present application.

Claims (6)

1. A method for dynamic log parsing, comprising:
configuring a log template according to the log format and the log type of the security device; wherein the log types comprise text, JSON and automation;
the method comprises the steps of obtaining a log to be analyzed of the security equipment, and analyzing the log template to obtain structured data of the log template; the structured data includes an input log format and an output log format;
according to the input log format, analyzing the log to be analyzed to obtain log analysis data;
according to the output log format, formatting the log analysis data to obtain an output log, and sending the output log to a log center;
the analyzing the log template to obtain the structured data of the log template includes:
acquiring an input request of the log template, and inputting the log template according to the input request;
circularly analyzing the log template to obtain the node type of the structured data;
the analyzing the log to be analyzed according to the input log format to obtain log analysis data includes:
if the node type is the input parameter node, analyzing the log to be analyzed;
if the node type is the output parameter node, outputting the list parameter after log analysis;
if the node type is the ending node, ending the analysis flow;
if the node type is an input parameter node, the analyzing the log to be analyzed includes:
identifying the format and field name of the log to be analyzed;
separating the log to be analyzed according to the log format to be analyzed to obtain a separator list;
analyzing the log to be analyzed according to the separator list and the field name to obtain a log analysis result;
the analyzing the log to be analyzed according to the separator list and the field name to obtain a log analysis result, which comprises the following steps:
associating the field name with the separator list to obtain a separator analysis result;
and analyzing the log to be analyzed according to the separator analysis result to obtain a log analysis result.
2. The method for dynamic log parsing according to claim 1, wherein the formatting the log parsed data according to the output log format to obtain an output log, and transmitting the output log to a log center, includes:
outputting the output log to a log channel;
and according to the channel configuration of the log channel, the output log is sent to a corresponding log center for storage.
3. The method of dynamic log parsing according to any one of claims 1-2, wherein the input log format comprises only one format and the output log format comprises a plurality of formats.
4. A dynamic log parsing apparatus, comprising:
the configuration module is used for configuring a log template according to the log format of the security equipment;
the template analysis module is used for acquiring a log to be analyzed of the security equipment, analyzing the log template and obtaining structured data of the log template; the structured data includes an input log format and an output log format;
the log analysis module is used for analyzing the log to be analyzed according to the input log format to obtain log analysis data;
the output module is used for formatting the log analysis data according to the output log format to obtain an output log and sending the output log to a log center;
the analyzing the log template to obtain the structured data of the log template includes:
acquiring an input request of the log template, and inputting the log template according to the input request;
circularly analyzing the log template to obtain the node type of the structured data;
the analyzing the log to be analyzed according to the input log format to obtain log analysis data includes:
if the node type is the input parameter node, analyzing the log to be analyzed;
if the node type is the output parameter node, outputting the list parameter after log analysis;
if the node type is the ending node, ending the analysis flow;
if the node type is an input parameter node, the analyzing the log to be analyzed includes:
identifying the format and field name of the log to be analyzed;
separating the log to be analyzed according to the log format to be analyzed to obtain a separator list;
analyzing the log to be analyzed according to the separator list and the field name to obtain a log analysis result;
the analyzing the log to be analyzed according to the separator list and the field name to obtain a log analysis result, which comprises the following steps:
associating the field name with the separator list to obtain a separator analysis result;
and analyzing the log to be analyzed according to the separator analysis result to obtain a log analysis result.
5. An electronic device comprising a memory and a processor, wherein,
the memory is used for storing programs;
the processor, coupled to the memory, is configured to execute the program stored in the memory to implement the steps in the dynamic log parsing method of any one of the preceding claims 1 to 3.
6. A computer readable storage medium storing a computer readable program or instructions which, when executed by a processor, is capable of carrying out the steps of the dynamic log parsing method of any one of claims 1 to 3.
CN202210461097.0A 2022-04-28 2022-04-28 Dynamic log analysis method, device, equipment and storage medium Active CN114785604B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210461097.0A CN114785604B (en) 2022-04-28 2022-04-28 Dynamic log analysis method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210461097.0A CN114785604B (en) 2022-04-28 2022-04-28 Dynamic log analysis method, device, equipment and storage medium

Publications (2)

Publication Number Publication Date
CN114785604A CN114785604A (en) 2022-07-22
CN114785604B true CN114785604B (en) 2023-11-07

Family

ID=82435061

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210461097.0A Active CN114785604B (en) 2022-04-28 2022-04-28 Dynamic log analysis method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114785604B (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106055450A (en) * 2016-05-20 2016-10-26 北京神州绿盟信息安全科技股份有限公司 Binary log analysis method and apparatus
CN106815306A (en) * 2016-12-16 2017-06-09 中铁程科技有限责任公司 Daily record analysis method and device
KR20180061891A (en) * 2016-11-30 2018-06-08 정준용 Log generator and big data analysis preprocessing system including the log generator
CN110826299A (en) * 2019-10-25 2020-02-21 上海工业自动化仪表研究院有限公司 General template log analysis method based on classification
CN111708860A (en) * 2020-06-15 2020-09-25 北京优特捷信息技术有限公司 Information extraction method, device, equipment and storage medium
CN112162965A (en) * 2020-10-12 2021-01-01 平安科技（深圳）有限公司 Log data processing method and device, computer equipment and storage medium
CN112632960A (en) * 2021-01-06 2021-04-09 北京启明星辰信息安全技术有限公司 Log analysis method and system based on dynamic field template

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11727025B2 (en) * 2015-04-03 2023-08-15 Oracle International Corporation Method and system for implementing a log parser in a log analytics system
US10592521B2 (en) * 2015-04-03 2020-03-17 Oracle International Corporation Method and system for implementing target model configuration metadata for a log analytics system

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106055450A (en) * 2016-05-20 2016-10-26 北京神州绿盟信息安全科技股份有限公司 Binary log analysis method and apparatus
KR20180061891A (en) * 2016-11-30 2018-06-08 정준용 Log generator and big data analysis preprocessing system including the log generator
CN106815306A (en) * 2016-12-16 2017-06-09 中铁程科技有限责任公司 Daily record analysis method and device
CN110826299A (en) * 2019-10-25 2020-02-21 上海工业自动化仪表研究院有限公司 General template log analysis method based on classification
CN111708860A (en) * 2020-06-15 2020-09-25 北京优特捷信息技术有限公司 Information extraction method, device, equipment and storage medium
CN112162965A (en) * 2020-10-12 2021-01-01 平安科技（深圳）有限公司 Log data processing method and device, computer equipment and storage medium
CN112632960A (en) * 2021-01-06 2021-04-09 北京启明星辰信息安全技术有限公司 Log analysis method and system based on dynamic field template

Also Published As

Publication number Publication date
CN114785604A (en) 2022-07-22

Similar Documents

Publication Publication Date Title
CN106919555B (en) System and method for field extraction of data contained within a log stream
US7984373B2 (en) EDI instance based transaction set definition
US10552293B2 (en) Logging as a service
CN111683066B (en) Heterogeneous system integration method, heterogeneous system integration device, computer equipment and storage medium
CN111131320A (en) Asset identification method, device, system, medium, and program product
CN111338693B (en) Model construction-based target file generation method, server and storage medium
CN112615758B (en) Application identification method, device, equipment and storage medium
CN112395843A (en) PHP code-based service processing method, device, equipment and medium
CN110990057A (en) Extraction method, device, equipment and medium of small program sub-chain information
CN111752820B (en) gRPC interface pressure test method, computer equipment and storage medium
CN113032256B (en) Automated testing method, apparatus, computer system, and readable storage medium
CN108959659B (en) Log access analysis method and system for big data platform
CN106909435B (en) Method and device for analyzing command line of network security equipment
CN114765559A (en) Method for verifying vulnerability of network equipment by using CVE (virtual container environment) entry
CN114785604B (en) Dynamic log analysis method, device, equipment and storage medium
CN111680288B (en) Container command execution method, device, equipment and storage medium
US9274910B2 (en) Automatic test map generation for system verification test
CN116662302A (en) Data processing method, device, electronic equipment and storage medium
CN116185393A (en) Method, device, equipment, medium and product for generating interface document
CN115150483A (en) Network data packet analysis method, system and readable storage medium
CN115033451A (en) Data generation method, data processing device, electronic device, and medium
CN114374745A (en) Protocol format processing method and system
CN114781404A (en) Translation method, translation system and storage medium
CN114116268A (en) Method and device for checking Flink SQL statement, computer equipment and storage medium
US20160188884A1 (en) Application Decomposition Using Data Obtained From External Tools For Use In Threat Modeling

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant