CN109284268B - Method, system and electronic equipment for rapidly analyzing logs - Google Patents

Method, system and electronic equipment for rapidly analyzing logs Download PDF

Info

Publication number
CN109284268B
CN109284268B CN201811272135.8A CN201811272135A CN109284268B CN 109284268 B CN109284268 B CN 109284268B CN 201811272135 A CN201811272135 A CN 201811272135A CN 109284268 B CN109284268 B CN 109284268B
Authority
CN
China
Prior art keywords
target
matching
log
rule
analyzed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201811272135.8A
Other languages
Chinese (zh)
Other versions
CN109284268A (en
Inventor
董云豪
范渊
刘博�
龙文洁
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
DBAPPSecurity Co Ltd
Original Assignee
Hangzhou Dbappsecurity Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Dbappsecurity Technology Co Ltd filed Critical Hangzhou Dbappsecurity Technology Co Ltd
Priority to CN201811272135.8A priority Critical patent/CN109284268B/en
Publication of CN109284268A publication Critical patent/CN109284268A/en
Application granted granted Critical
Publication of CN109284268B publication Critical patent/CN109284268B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Debugging And Monitoring (AREA)

Abstract

The invention provides a method, a system and electronic equipment for rapidly analyzing logs, in the method, after receiving a target log to be analyzed, sequentially matching the target log to be analyzed with a target analysis rule corresponding to asset information to be matched in a current optimized asset matching rule recording table, if the matching is unsuccessful, entering an analysis rule base for analysis rule matching, and if the asset information recorded in the current optimized asset matching rule recording table, analyzing rule ID and the corresponding relation among the matching times are arranged according to the descending order of the matching times corresponding to the same asset information, thus, when the sequence is matched, the matching times are reduced to a certain extent, the probability of successful matching is greatly improved, the speed of log analysis is accelerated, meanwhile, the analysis granularity is not required to be sacrificed, the reliability of log analysis is high, the technical problems of low analysis speed and poor reliability of the existing log analysis method are solved.

Description

Method, system and electronic equipment for rapidly analyzing logs
Technical Field
The present invention relates to the field of network security technologies, and in particular, to a method, a system, and an electronic device for quickly analyzing logs.
Background
With the increasing requirements of log auditing, people have higher and higher requirements on log auditing. The amount of log storage and resolution is also increasing. The analyzed log types also range from a single host, network equipment to application logs and various safety equipment logs. The complexity of log analysis is increasing day by day, the performance required by the analysis is also increasing gradually, and great pressure is generated on the high-speed processing of the logs.
In the existing log processing system, the logs are matched with the analysis rule base in a full quantity. The method can complete the analysis of the logs by occupying a small amount of performance under the condition of a small amount of logs, but can cause great performance waste under the condition of a large amount of logs. When the number of the analysis rule entries is only 10, the number of analysis matching times of each log may be less than ten, but when the number of the analysis rule entries reaches 10W, the number of analysis matching of the system to each log may reach tens of thousands of times or even 9W of times, and each log that enters needs to be matched through the flow, a great waste of resources is caused in the process, and the analysis speed is affected. Therefore, the problem of slow resolution speed or resolution granularity sacrifice to improve the resolution speed is generated.
In summary, the conventional log analysis method has the technical problems of low analysis speed and poor reliability.
Disclosure of Invention
In view of the above, an object of the present invention is to provide a method, a system and an electronic device for fast analyzing logs, so as to alleviate the technical problems of slow analysis speed and poor reliability of the existing log analysis method.
In a first aspect, an embodiment of the present invention provides a method for quickly parsing a log, including:
acquiring a target log to be analyzed, and acquiring asset information to be matched of the target log to be analyzed, wherein the asset information to be matched at least comprises an asset ID, an asset IP and a port number;
sequentially matching the target to-be-analyzed log with a target analysis rule corresponding to the asset information to be matched in a current optimized asset matching rule record table, wherein the current optimized asset matching rule record table comprises: the asset information at the current moment, the corresponding relation between the ID of the analysis rule and the matching times, and the corresponding relation is arranged according to the descending order of the matching times corresponding to the same asset information in the current optimized asset matching rule record table;
if the target to-be-analyzed log is matched with a sub-target analysis rule in the target analysis rule, analyzing the target to-be-analyzed log through the sub-target analysis rule;
and if the target to-be-analyzed log is not matched with all sub-target analysis rules in the target analysis rule, or no record of the asset information to be matched exists in the current optimized asset matching rule record table, matching the target to-be-analyzed log with the analysis rules in an analysis rule base one by one, and analyzing the target to-be-analyzed log according to the analysis rules obtained by matching, wherein the analysis rules represented by the analysis rule ID in the current optimized asset matching rule record table are subsets of the analysis rules in the analysis rule base.
With reference to the first aspect, an embodiment of the present invention provides a first possible implementation manner of the first aspect, where acquiring a target to-be-parsed log includes:
acquiring a log to be analyzed and acquiring asset information of the log to be analyzed;
judging whether the asset information of the log to be analyzed is matched with preset asset information or not;
and if the asset information of the log to be analyzed is matched with the preset asset information, determining the log to be analyzed as the target log to be analyzed.
With reference to the first aspect, an embodiment of the present invention provides a second possible implementation manner of the first aspect, where before sequentially matching the target to-be-analyzed log with a target analysis rule corresponding to the asset information to be matched in a current optimized asset matching rule record table, the method further includes:
judging whether the record of the asset information to be matched exists in the current optimized asset matching rule record table or not;
if yes, sequentially matching the target to-be-analyzed log with a target analysis rule corresponding to the asset information to be matched in the current optimized asset matching rule record table;
and if the target log to be analyzed does not exist, matching the target log to be analyzed with the analysis rules in the analysis rule base one by one.
With reference to the first aspect, an embodiment of the present invention provides a third possible implementation manner of the first aspect, where after the target log to be parsed is matched with a sub-target parsing rule in the target parsing rule, the method further includes:
updating the matching times corresponding to the asset information to be matched and the sub-target analysis rule ID in the current optimized asset matching rule record table to obtain the updated matching times, wherein the sub-target analysis rule ID is the ID corresponding to the sub-target analysis rule;
and updating the sequence of the corresponding relation in the current optimized asset matching rule record table based on the updated matching times to obtain an updated optimized asset matching rule record table, so that the updated optimized asset matching rule record table is used for log analysis at the next moment.
With reference to the first aspect, an embodiment of the present invention provides a fourth possible implementation manner of the first aspect, where after the target log to be analyzed is analyzed according to the analysis rule obtained by matching, the method further includes:
and taking the asset information to be matched, the analysis rule ID and the matching times corresponding to the analysis rule obtained by matching as a new corresponding relation, and adding the new corresponding relation to the current optimized asset matching rule record table according to the principle that the matching times corresponding to the same asset information to be matched are arranged in a descending order to obtain an updated optimized asset matching rule record table so as to use the updated optimized asset matching rule record table for log analysis at the next moment.
In a second aspect, an embodiment of the present invention further provides a system for quickly parsing a log, including:
the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring a target log to be analyzed and acquiring asset information to be matched of the target log to be analyzed, and the asset information to be matched at least comprises an asset ID, an asset IP and a port number;
a matching module, configured to sequentially match the target to-be-analyzed log with a target analysis rule corresponding to the asset information to be matched in a current optimized asset matching rule record table, where the current optimized asset matching rule record table includes: the asset information at the current moment, the corresponding relation between the ID of the analysis rule and the matching times, and the corresponding relation is arranged according to the descending order of the matching times corresponding to the same asset information in the current optimized asset matching rule record table;
the first analysis module is used for analyzing the target log to be analyzed according to the sub-target analysis rule if the target log to be analyzed is matched with the sub-target analysis rule in the target analysis rule;
and the second analysis module is used for matching the target log to be analyzed with the analysis rules in the analysis rule base one by one if the target log to be analyzed is not matched with all sub-target analysis rules in the target analysis rule or no record of the asset information to be matched exists in the current optimized asset matching rule record table, and analyzing the target log to be analyzed according to the analysis rules obtained by matching, wherein the analysis rules represented by the analysis rule ID in the current optimized asset matching rule record table are subsets of the analysis rules in the analysis rule base.
With reference to the second aspect, an embodiment of the present invention provides a first possible implementation manner of the second aspect, where the obtaining module includes:
the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a log to be analyzed and acquiring asset information of the log to be analyzed;
the judging unit is used for judging whether the asset information of the log to be analyzed is matched with preset asset information or not;
and the determining unit is used for determining the log to be analyzed as the target log to be analyzed if the asset information of the log to be analyzed is matched with the preset asset information.
With reference to the second aspect, an embodiment of the present invention provides a second possible implementation manner of the second aspect, where the system is further configured to:
judging whether the record of the asset information to be matched exists in the current optimized asset matching rule record table or not;
if yes, sequentially matching the target to-be-analyzed log with a target analysis rule corresponding to the asset information to be matched in the current optimized asset matching rule record table;
and if the target log to be analyzed does not exist, matching the target log to be analyzed with the analysis rules in the analysis rule base one by one.
With reference to the second aspect, an embodiment of the present invention provides a third possible implementation manner of the second aspect, where the system further includes:
a first updating module, configured to update the matching times corresponding to the asset information to be matched and the sub-target parsing rule ID in the current optimized asset matching rule record table to obtain updated matching times, where the sub-target parsing rule ID is an ID corresponding to the sub-target parsing rule;
and the second updating module is used for updating the sequence of the corresponding relation in the current optimized asset matching rule record table based on the updated matching times to obtain an updated optimized asset matching rule record table, so that the updated optimized asset matching rule record table is used for log analysis at the next moment.
In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory and a processor, where the memory stores a computer program operable on the processor, and the processor executes the computer program to implement the steps of the method in the first aspect.
The embodiment of the invention has the following beneficial effects:
in the embodiment, a target log to be analyzed is obtained first, and asset information to be matched of the target log to be analyzed is obtained; then, sequentially matching the target to-be-analyzed log with a target analysis rule corresponding to the asset information to be matched in the current optimized asset matching rule record table; if the target to-be-analyzed log is matched with the sub-target analysis rule in the target analysis rule, analyzing the target to-be-analyzed log through the sub-target analysis rule; and if the target to-be-analyzed log is not matched with all sub-target analysis rules in the target analysis rule, or no record of the asset information to be matched exists in the current optimized asset matching rule record table, matching the target to-be-analyzed log with the analysis rules in the analysis rule base one by one, and analyzing the target to-be-analyzed log according to the analysis rule obtained through matching. It can be known from the above description that, in this embodiment, after receiving a target log to be parsed, sequentially matching the target log to be parsed with a target parsing rule corresponding to asset information to be matched in a current optimized asset matching rule record table, if the matching is unsuccessful, entering a parsing rule library for parsing rule matching, and arranging the corresponding relationship between the asset information, the parsing rule ID, and the matching times recorded in the current optimized asset matching rule record table according to a descending order of the matching times corresponding to the same asset information, so that when the matching is performed sequentially, the matching times are reduced to a certain extent, the probability of successful matching is greatly improved, the log parsing speed is increased, and meanwhile, the parsing granularity does not need to be sacrificed, so that the reliability of log parsing is high, and the parsing speed of the existing log parsing method is reduced, the reliability is poor.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a method for fast log parsing according to an embodiment of the present invention;
fig. 2 is a flowchart of a method for obtaining a target to-be-analyzed log according to an embodiment of the present invention;
FIG. 3 is a flowchart of a method for determining a matching location according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a system for fast parsing a log according to an embodiment of the present invention;
fig. 5 is a schematic diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
To facilitate understanding of the embodiment, a method for fast log parsing disclosed in the embodiment of the present invention is first described in detail.
The first embodiment is as follows:
in accordance with an embodiment of the present invention, there is provided an embodiment of a method for fast resolving logs, it is noted that the steps illustrated in the flowchart of the figure may be performed in a computer system such as a set of computer executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
Fig. 1 is a flowchart of a method for fast log parsing according to an embodiment of the present invention, as shown in fig. 1, the method includes the following steps:
step S102, acquiring a target log to be analyzed, and acquiring asset information to be matched of the target log to be analyzed, wherein the asset information to be matched at least comprises an asset ID, an asset IP and a port number;
in the embodiment of the invention, the method for rapidly analyzing the logs can be applied to a log auditing system which is developed autonomously and is used for receiving, processing and analyzing the logs of the equipment to be audited (namely the target logs to be analyzed).
Step S104, sequentially matching the target to-be-analyzed log with a target analysis rule corresponding to the asset information to be matched in a current optimized asset matching rule record table, wherein the current optimized asset matching rule record table comprises: the asset information at the current moment, the corresponding relation between the ID of the analysis rule and the matching times, and the corresponding relation is arranged according to the descending order of the matching times corresponding to the same asset information in the current optimized asset matching rule record table;
and after the target log to be analyzed is obtained, sequentially matching the target log to be analyzed with a target analysis rule corresponding to the asset information to be matched in the current optimized asset matching rule record table.
Specifically, the current optimized asset matching rule record table is obtained by arranging the current asset information, the analysis rule ID and the corresponding relation between the matching times according to the descending order of the matching times corresponding to the same asset information. The form of the current optimized asset matching rule record table in the embodiment of the present invention is as follows (which is not particularly limited by the embodiment of the present invention):
asset ID Resolution rule ID Number of matches
id1 ID5 1200
id1 ID3 800
id2 ID5 1000
id2 ID4 600
id3 ID2 200
...
idn ID6 600
Step S106, if the target to-be-analyzed log is matched with the sub-target analysis rule in the target analysis rule, analyzing the target to-be-analyzed log through the sub-target analysis rule;
and S108, if the target to-be-analyzed log is not matched with all sub-target analysis rules in the target analysis rule, or no record of the asset information to be matched exists in the current optimized asset matching rule record table, matching the target to-be-analyzed log with the analysis rules in the analysis rule base one by one, and analyzing the target to-be-analyzed log according to the analysis rules obtained by matching, wherein the analysis rules represented by the analysis rule ID in the current optimized asset matching rule record table are subsets of the analysis rules in the analysis rule base.
Specifically, the parsing rule base is used for storing parsing rules for parsing the log, and the parsing rules stored in the parsing rule base are full-amount rules. The analysis rules are rule files which are independently developed by the inventor, when the log audit system is initialized, the rule files are automatically loaded to the analysis rule base, the analysis rules of the customized equipment in the rule files can be imported into the analysis rule base according to requirements, and each analysis rule corresponds to a unique ID. The following table is a parsing rule base of an embodiment of the invention:
serial number Resolution rule ID Parsing rules
1 ID1 Rule 1
2 ID2 Rule 2
3 ID3 Rule 3
4 ID4 Rule 4
5 ID5 Rule 5
n IDn Rule n
And the analysis rule represented by the analysis rule ID in the current optimized asset matching rule record table is a subset of the analysis rule in the analysis rule base.
In the embodiment, a target log to be analyzed is obtained first, and asset information to be matched of the target log to be analyzed is obtained; then, sequentially matching the target to-be-analyzed log with a target analysis rule corresponding to the asset information to be matched in the current optimized asset matching rule record table; if the target to-be-analyzed log is matched with the sub-target analysis rule in the target analysis rule, analyzing the target to-be-analyzed log through the sub-target analysis rule; and if the target to-be-analyzed log is not matched with all sub-target analysis rules in the target analysis rule, or no record of the asset information to be matched exists in the current optimized asset matching rule record table, matching the target to-be-analyzed log with the analysis rules in the analysis rule base one by one, and analyzing the target to-be-analyzed log according to the analysis rule obtained through matching. It can be known from the above description that, in this embodiment, after receiving a target log to be parsed, sequentially matching the target log to be parsed with a target parsing rule corresponding to asset information to be matched in a current optimized asset matching rule record table, if the matching is unsuccessful, entering a parsing rule library for parsing rule matching, and arranging the corresponding relationship between the asset information, the parsing rule ID, and the matching times recorded in the current optimized asset matching rule record table according to a descending order of the matching times corresponding to the same asset information, so that when the matching is performed sequentially, the matching times are reduced to a certain extent, the probability of successful matching is greatly improved, the log parsing speed is increased, and meanwhile, the parsing granularity does not need to be sacrificed, so that the reliability of log parsing is high, and the parsing speed of the existing log parsing method is reduced, the reliability is poor.
The above-mentioned contents briefly describe the process of the method for fast parsing a log according to the present invention, and other contents related thereto are described below.
In an optional embodiment of the present invention, referring to fig. 2, the obtaining of the target to-be-parsed log includes the following steps:
step S201, acquiring a log to be analyzed, and acquiring asset information of the log to be analyzed;
step S202, judging whether the asset information of the log to be analyzed is matched with preset asset information;
specifically, configuring the asset information in the log auditing system in advance includes: asset ID, asset IP, port number. After the asset information is configured, if the log to be analyzed is obtained, whether the asset information of the log to be analyzed is matched with preset asset information (namely the pre-configured asset information) is judged.
Step S203, if the asset information of the log to be analyzed is matched with the preset asset information, determining the log to be analyzed as the target log to be analyzed.
Step S204, if the asset information of the log to be analyzed is not matched with the preset asset information, discarding the log to be analyzed.
In an optional embodiment of the present invention, referring to fig. 3, before sequentially matching the target to-be-parsed log with the target parsing rule corresponding to the asset information to be matched in the current optimized asset matching rule record table, the method further includes the following steps:
step S301, judging whether a record of asset information to be matched exists in a current optimized asset matching rule record table;
step S302, if the log to be analyzed exists, the log to be analyzed and the target analysis rule corresponding to the asset information to be matched in the current optimization asset matching rule record table are sequentially matched;
step S303, if the log to be analyzed does not exist, the target log to be analyzed and the analysis rules in the analysis rule base are matched one by one.
In an optional embodiment of the present invention, after the target to-be-parsed log matches a sub-target parsing rule in the target parsing rule, the method further includes the following (1) and (2):
(1) updating the matching times corresponding to the asset information to be matched and the sub-target analysis rule ID in the current optimized asset matching rule recording table to obtain the updated matching times, wherein the sub-target analysis rule ID is the ID corresponding to the sub-target analysis rule;
(2) and updating the sequence of the corresponding relation in the current optimized asset matching rule record table based on the updated matching times to obtain an updated optimized asset matching rule record table, so that the updated optimized asset matching rule record table is used for log analysis at the next moment.
In an optional embodiment of the present invention, after parsing the target log to be parsed according to the parsing rule obtained by matching, the method further includes:
and taking the asset information to be matched, the analysis rule ID and the matching times corresponding to the analysis rule obtained by matching as a new corresponding relation, and adding the new corresponding relation to the current optimized asset matching rule record table according to the principle that the matching times corresponding to the same asset information to be matched are arranged in a descending order to obtain an updated optimized asset matching rule record table so as to use the updated optimized asset matching rule record table for log analysis at the next moment.
The following describes the log parsing process in a specific embodiment:
the initial state of the record table of the optimized asset matching rule is as follows:
asset ID Resolution rule ID Number of matches
1) When the received asset ID is ID1Matching the target log to be analyzed with the optimized asset matching rule record table, because no asset ID in the optimized asset matching rule record table is ID1The corresponding analysis rule ID is entered into the analysis rule base to match the analysis rules in the target log to be analyzed and the target log to be analyzed one by one, if the analysis rule 5 is obtained by matching, the analysis rule 5 is used for analyzing the target log to be analyzed, and the asset ID is ID1The resolution rule ID is ID5And adding the corresponding relation with the matching times of 1 into an optimized asset matching rule recording table, and sequencing the corresponding relations according to a principle that the matching times corresponding to the same asset information are arranged in a descending order to obtain a first optimized asset matching rule recording table.
Asset ID Resolution rule ID Number of matches
id1 ID5 1
2) When the asset ID is received again as ID1Matching the target log to be analyzed with the optimized asset matching rule record table, wherein the asset ID in the optimized asset matching rule record table is ID1The corresponding resolution rule ID is ID5Matching the target log to be analyzed with the analysis rule 5, if the target log to be analyzed is matched with the analysis rule 5, analyzing the target log to be analyzed by using the analysis rule 5, and updating the id1And ID5And (3) sequencing the corresponding matching times (namely adding 1 to the matching times) according to the principle that the matching times corresponding to the same asset information are arranged in a descending order to obtain a second optimized asset matching rule recording table.
Asset ID Resolution rule ID Number of matches
id1 ID5 2
3) When the re-received asset ID is id1Matching the target log to be analyzed with the optimized asset matching rule record table, wherein the asset ID in the optimized asset matching rule record table is ID1The corresponding resolution rule ID is ID5Matching the target log to be analyzed with the analysis rule 5, if the target log to be analyzed is not matched with the analysis rule 5, entering an analysis rule base to match the target log to be analyzed with the analysis rules in the target log to be analyzed one by one, if the target log to be analyzed is matched with the analysis rules 3, analyzing the target log to be analyzed by using the analysis rule 3, and enabling the asset ID to be ID1The resolution rule ID is ID3And adding the corresponding relation with the matching times of 1 into the optimized asset matching rule recording table, and sequencing the corresponding relations according to the principle that the matching times corresponding to the same asset information are arranged in a descending order to obtain a third optimized asset matching rule recording table.
Asset ID Resolution rule ID Number of matches
id1 ID5 2
id1 ID3 1
4) When receiving again the asset ID as ID2Matching the target log to be analyzed with the optimizationAsset matching rule record table because none of the asset IDs in the optimized asset matching rule record table are ID' s2The corresponding analysis rule ID is entered into the analysis rule base to match the analysis rules in the target log to be analyzed and the target log to be analyzed one by one, if the analysis rule 5 is obtained by matching, the analysis rule 5 is used for analyzing the target log to be analyzed, and the asset ID is ID2The resolution rule ID is ID5And adding the corresponding relation with the matching times of 1 into the optimized asset matching rule recording table, and sequencing the corresponding relations according to the principle that the matching times corresponding to the same asset information are arranged in a descending order to obtain a fourth optimized asset matching rule recording table.
Asset ID Resolution rule ID Number of matches
id1 ID5 2
id1 ID3 1
id2 ID5 1
….
Along with the increase of log analysis amount, the optimized asset matching rule record table is gradually enriched and gradually tends to be comprehensive and complete, so that when a new log is analyzed, the probability of successful matching of the analysis rule corresponding to the asset ID in the optimized asset matching rule record table approaches to one hundred percent, and the analysis rule base does not need to be matched one by one each time, thereby greatly improving the matching efficiency.
Example two:
the embodiment of the present invention further provides a system for fast analyzing logs, where the system for fast analyzing logs is mainly used to execute the method for fast analyzing logs provided in the foregoing content of the embodiment of the present invention, and the system for fast analyzing logs provided in the embodiment of the present invention is specifically described below.
Fig. 4 is a schematic diagram of a system for fast parsing a log according to an embodiment of the present invention, as shown in fig. 4, the system for fast parsing a log mainly includes an obtaining module 10, a matching module 20, a first parsing module 30 and a second parsing module 40, where:
the acquisition module is used for acquiring a target log to be analyzed and acquiring asset information to be matched of the target log to be analyzed, wherein the asset information to be matched at least comprises an asset ID, an asset IP and a port number;
the matching module is used for sequentially matching the target to-be-analyzed log with a target analysis rule corresponding to the asset information to be matched in a current optimized asset matching rule record table, wherein the current optimized asset matching rule record table comprises: the asset information at the current moment, the corresponding relation between the ID of the analysis rule and the matching times, and the corresponding relation is arranged according to the descending order of the matching times corresponding to the same asset information in the current optimized asset matching rule record table;
the first analysis module is used for analyzing the target log to be analyzed through the sub-target analysis rule if the target log to be analyzed is matched with the sub-target analysis rule in the target analysis rule;
and the second analysis module is used for matching the target log to be analyzed with the analysis rules in the analysis rule base one by one if the target log to be analyzed is not matched with all sub-target analysis rules in the target analysis rule or no record of the asset information to be matched exists in the current optimized asset matching rule record table, and analyzing the target log to be analyzed according to the analysis rules obtained by matching, wherein the analysis rules represented by the analysis rule ID in the current optimized asset matching rule record table are subsets of the analysis rules in the analysis rule base.
In the embodiment, a target log to be analyzed is obtained first, and asset information to be matched of the target log to be analyzed is obtained; then, sequentially matching the target to-be-analyzed log with a target analysis rule corresponding to the asset information to be matched in the current optimized asset matching rule record table; if the target to-be-analyzed log is matched with the sub-target analysis rule in the target analysis rule, analyzing the target to-be-analyzed log through the sub-target analysis rule; and if the target to-be-analyzed log is not matched with all sub-target analysis rules in the target analysis rule, or no record of the asset information to be matched exists in the current optimized asset matching rule record table, matching the target to-be-analyzed log with the analysis rules in the analysis rule base one by one, and analyzing the target to-be-analyzed log according to the analysis rule obtained through matching. It can be known from the above description that, in this embodiment, after receiving a target log to be parsed, sequentially matching the target log to be parsed with a target parsing rule corresponding to asset information to be matched in a current optimized asset matching rule record table, if the matching is unsuccessful, entering a parsing rule library for parsing rule matching, and arranging the corresponding relationship between the asset information, the parsing rule ID, and the matching times recorded in the current optimized asset matching rule record table according to a descending order of the matching times corresponding to the same asset information, so that when the matching is performed sequentially, the matching times are reduced to a certain extent, the probability of successful matching is greatly improved, the log parsing speed is increased, and meanwhile, the parsing granularity does not need to be sacrificed, so that the reliability of log parsing is high, and the parsing speed of the existing log parsing method is reduced, the reliability is poor.
Optionally, the obtaining module includes:
the acquisition unit is used for acquiring the log to be analyzed and acquiring the asset information of the log to be analyzed;
the judging unit is used for judging whether the asset information of the log to be analyzed is matched with the preset asset information;
and the determining unit is used for determining the log to be analyzed as the target log to be analyzed if the asset information of the log to be analyzed is matched with the preset asset information.
Optionally, the system is further configured to:
judging whether a record of asset information to be matched exists in a current optimized asset matching rule record table or not;
if yes, sequentially matching the target to-be-analyzed log with a target analysis rule corresponding to the asset information to be matched in the current optimized asset matching rule record table;
and if the target log to be analyzed does not exist, matching the target log to be analyzed with the analysis rules in the analysis rule base one by one.
Optionally, the system further comprises:
the first updating module is used for updating the asset information to be matched and the matching times corresponding to the sub-target analysis rule IDs in the current optimized asset matching rule recording table to obtain the updated matching times, wherein the sub-target analysis rule IDs are the IDs corresponding to the sub-target analysis rules;
and the second updating module is used for updating the sequence of the corresponding relation in the current optimized asset matching rule record table based on the updated matching times to obtain an updated optimized asset matching rule record table so as to use the updated optimized asset matching rule record table for log analysis at the next moment.
Optionally, the system further comprises:
and the adding module is used for taking the asset information to be matched, the analysis rule ID and the matching times corresponding to the analysis rule obtained by matching as a new corresponding relation, and adding the new corresponding relation to the current optimized asset matching rule record table according to the principle that the matching times corresponding to the same asset information to be matched are arranged in a descending order manner to obtain an updated optimized asset matching rule record table so as to use the updated optimized asset matching rule record table for log analysis at the next moment.
The system provided by the embodiment of the present invention has the same implementation principle and technical effect as the foregoing method embodiment, and for the sake of brief description, no mention is made in the system embodiment, and reference may be made to the corresponding contents in the foregoing method embodiment.
Example three:
an embodiment of the present invention provides an electronic device, and with reference to fig. 5, the electronic device includes: the processor 50, the memory 51, the bus 52 and the communication interface 53, wherein the processor 50, the communication interface 53 and the memory 51 are connected through the bus 52; the processor 50 is arranged to execute executable modules, such as computer programs, stored in the memory 51. The processor, when executing the program or the program, performs the steps of the method as described in the method embodiments.
The Memory 51 may include a high-speed Random Access Memory (RAM) and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 53 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used.
The bus 52 may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 5, but this does not indicate only one bus or one type of bus.
The memory 51 is used for storing a program, and the processor 50 executes the program after receiving an execution instruction, and the method executed by the system defined by the flow process disclosed in any of the foregoing embodiments of the present invention may be applied to the processor 50, or implemented by the processor 50.
The processor 50 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 50. The Processor 50 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 51, and the processor 50 reads the information in the memory 51 and completes the steps of the method in combination with the hardware thereof.
The method, the system and the computer program product of the electronic device for rapidly analyzing the log provided by the embodiment of the present invention include a computer readable storage medium storing a program code, where instructions included in the program code may be used to execute the method described in the foregoing method embodiment, and specific implementation may refer to the method embodiment, and will not be described herein again.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the system described above may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.
In addition, in the description of the embodiments of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (10)

1. A method for fast resolving logs, comprising:
acquiring a target log to be analyzed, and acquiring asset information to be matched of the target log to be analyzed, wherein the asset information to be matched at least comprises an asset ID, an asset IP and a port number;
sequentially matching the target to-be-analyzed log with a target analysis rule corresponding to the asset information to be matched in a current optimized asset matching rule record table, wherein the current optimized asset matching rule record table comprises: the asset information at the current moment, the corresponding relation between the ID of the analysis rule and the matching times, and the corresponding relation is arranged according to the descending order of the matching times corresponding to the same asset information in the current optimized asset matching rule record table;
if the target to-be-analyzed log is matched with a sub-target analysis rule in the target analysis rule, analyzing the target to-be-analyzed log through the sub-target analysis rule;
and if the target to-be-analyzed log is not matched with all sub-target analysis rules in the target analysis rule, or no record of the asset information to be matched exists in the current optimized asset matching rule record table, matching the target to-be-analyzed log with the analysis rules in an analysis rule base one by one, and analyzing the target to-be-analyzed log according to the analysis rules obtained by matching, wherein the analysis rules represented by the analysis rule ID in the current optimized asset matching rule record table are subsets of the analysis rules in the analysis rule base.
2. The method of claim 1, wherein obtaining the target to-be-parsed log comprises:
acquiring a log to be analyzed and acquiring asset information of the log to be analyzed;
judging whether the asset information of the log to be analyzed is matched with preset asset information or not;
and if the asset information of the log to be analyzed is matched with the preset asset information, determining the log to be analyzed as the target log to be analyzed.
3. The method according to claim 1, wherein before sequentially matching the target to-be-parsed log with the target parsing rule corresponding to the asset information to be matched in the current optimized asset matching rule record table, the method further comprises:
judging whether the record of the asset information to be matched exists in the current optimized asset matching rule record table or not;
if yes, sequentially matching the target to-be-analyzed log with a target analysis rule corresponding to the asset information to be matched in the current optimized asset matching rule record table;
and if the target log to be analyzed does not exist, matching the target log to be analyzed with the analysis rules in the analysis rule base one by one.
4. The method of claim 1, wherein after the target to-be-parsed log matches a sub-target parsing rule of the target parsing rules, the method further comprises:
updating the matching times corresponding to the asset information to be matched and the sub-target analysis rule ID in the current optimized asset matching rule record table to obtain the updated matching times, wherein the sub-target analysis rule ID is the ID corresponding to the sub-target analysis rule;
and updating the sequence of the corresponding relation in the current optimized asset matching rule record table based on the updated matching times to obtain an updated optimized asset matching rule record table, so that the updated optimized asset matching rule record table is used for log analysis at the next moment.
5. The method according to claim 1, wherein after parsing the target log to be parsed according to the matching parsing rule, the method further comprises:
and taking the asset information to be matched, the analysis rule ID and the matching times corresponding to the analysis rule obtained by matching as a new corresponding relation, and adding the new corresponding relation to the current optimized asset matching rule record table according to the principle that the matching times corresponding to the same asset information to be matched are arranged in a descending order to obtain an updated optimized asset matching rule record table so as to use the updated optimized asset matching rule record table for log analysis at the next moment.
6. A system for fast parsing a log, comprising:
the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring a target log to be analyzed and acquiring asset information to be matched of the target log to be analyzed, and the asset information to be matched at least comprises an asset ID, an asset IP and a port number;
a matching module, configured to sequentially match the target to-be-analyzed log with a target analysis rule corresponding to the asset information to be matched in a current optimized asset matching rule record table, where the current optimized asset matching rule record table includes: the asset information at the current moment, the corresponding relation between the ID of the analysis rule and the matching times, and the corresponding relation is arranged according to the descending order of the matching times corresponding to the same asset information in the current optimized asset matching rule record table;
the first analysis module is used for analyzing the target log to be analyzed according to the sub-target analysis rule if the target log to be analyzed is matched with the sub-target analysis rule in the target analysis rule;
and the second analysis module is used for matching the target log to be analyzed with the analysis rules in the analysis rule base one by one if the target log to be analyzed is not matched with all sub-target analysis rules in the target analysis rule or no record of the asset information to be matched exists in the current optimized asset matching rule record table, and analyzing the target log to be analyzed according to the analysis rules obtained by matching, wherein the analysis rules represented by the analysis rule ID in the current optimized asset matching rule record table are subsets of the analysis rules in the analysis rule base.
7. The system of claim 6, wherein the acquisition module comprises:
the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a log to be analyzed and acquiring asset information of the log to be analyzed;
the judging unit is used for judging whether the asset information of the log to be analyzed is matched with preset asset information or not;
and the determining unit is used for determining the log to be analyzed as the target log to be analyzed if the asset information of the log to be analyzed is matched with the preset asset information.
8. The system of claim 6, wherein the system is further configured to:
judging whether the record of the asset information to be matched exists in the current optimized asset matching rule record table or not;
if yes, sequentially matching the target to-be-analyzed log with a target analysis rule corresponding to the asset information to be matched in the current optimized asset matching rule record table;
and if the target log to be analyzed does not exist, matching the target log to be analyzed with the analysis rules in the analysis rule base one by one.
9. The system of claim 6, further comprising:
a first updating module, configured to update the matching times corresponding to the asset information to be matched and the sub-target parsing rule ID in the current optimized asset matching rule record table to obtain updated matching times, where the sub-target parsing rule ID is an ID corresponding to the sub-target parsing rule;
and the second updating module is used for updating the sequence of the corresponding relation in the current optimized asset matching rule record table based on the updated matching times to obtain an updated optimized asset matching rule record table, so that the updated optimized asset matching rule record table is used for log analysis at the next moment.
10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method of any of claims 1 to 5 when executing the computer program.
CN201811272135.8A 2018-10-29 2018-10-29 Method, system and electronic equipment for rapidly analyzing logs Active CN109284268B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201811272135.8A CN109284268B (en) 2018-10-29 2018-10-29 Method, system and electronic equipment for rapidly analyzing logs

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811272135.8A CN109284268B (en) 2018-10-29 2018-10-29 Method, system and electronic equipment for rapidly analyzing logs

Publications (2)

Publication Number Publication Date
CN109284268A CN109284268A (en) 2019-01-29
CN109284268B true CN109284268B (en) 2020-11-24

Family

ID=65174378

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811272135.8A Active CN109284268B (en) 2018-10-29 2018-10-29 Method, system and electronic equipment for rapidly analyzing logs

Country Status (1)

Country Link
CN (1) CN109284268B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113992364B (en) * 2021-10-15 2024-06-07 湖南恒茂高科股份有限公司 Network data packet blocking optimization method and system
CN115102848B (en) * 2022-07-13 2024-05-28 中广核数字科技有限公司 Log data extraction method, system, equipment and medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103618692A (en) * 2013-10-28 2014-03-05 中国航天科工集团第二研究院七〇六所 A method for constructing log fast matching
CN103873441A (en) * 2012-12-12 2014-06-18 中国电信股份有限公司 Firewall safety rule optimization method and device thereof
CN105760274A (en) * 2016-01-27 2016-07-13 杭州安恒信息技术有限公司 Dynamically activated and adjusted log analyzing method and system
CN108462717A (en) * 2018-03-21 2018-08-28 北京理工大学 The firewall rule sets under discrimination optimization method of rule-based match hit rate and distribution variance

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101237326B (en) * 2008-02-29 2011-09-14 成都市华为赛门铁克科技有限公司 Method, device and system for real time parsing of device log
WO2015141630A1 (en) * 2014-03-19 2015-09-24 日本電信電話株式会社 Analysis rule adjustment device, analysis rule adjustment system, analysis rule adjustment method, and analysis rule adjustment program

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103873441A (en) * 2012-12-12 2014-06-18 中国电信股份有限公司 Firewall safety rule optimization method and device thereof
CN103618692A (en) * 2013-10-28 2014-03-05 中国航天科工集团第二研究院七〇六所 A method for constructing log fast matching
CN105760274A (en) * 2016-01-27 2016-07-13 杭州安恒信息技术有限公司 Dynamically activated and adjusted log analyzing method and system
CN108462717A (en) * 2018-03-21 2018-08-28 北京理工大学 The firewall rule sets under discrimination optimization method of rule-based match hit rate and distribution variance

Also Published As

Publication number Publication date
CN109284268A (en) 2019-01-29

Similar Documents

Publication Publication Date Title
CN109246064B (en) Method, device and equipment for generating security access control and network access rule
US9313226B2 (en) Method and system for network validation of information
US11381599B2 (en) Cyber chaff using spatial voting
CN110275965B (en) False news detection method, electronic device and computer readable storage medium
CN113132267B (en) Distributed system, data aggregation method and computer readable storage medium
CN109284268B (en) Method, system and electronic equipment for rapidly analyzing logs
CN109670091B (en) Metadata intelligent maintenance method and device based on data standard
CN108241618B (en) Database migration method and device and service program migration method and device
Wu et al. Detect repackaged android application based on http traffic similarity
CN112181430A (en) Code change statistical method and device, electronic equipment and storage medium
CN113946566B (en) Web system fingerprint database construction method and device and electronic equipment
CN115390847A (en) Log processing method and device, computer readable storage medium and terminal
CN110599278B (en) Method, apparatus, and computer storage medium for aggregating device identifiers
CN110471966B (en) Information data verification method, device, computer equipment and storage medium
CN116866241A (en) Internet of things terminal detection method, system and storage medium based on DPI
CN114490673B (en) Data information processing method and device, electronic equipment and storage medium
US9235639B2 (en) Filter regular expression
CN109635567A (en) For the method for calibration of applications client, device and server platform
CN114398518A (en) Method and system for rapidly matching normalization strategy for log
CN114020772A (en) Query condition configuration method, system, electronic device and storage medium
CN113992364A (en) Network data packet blocking optimization method and system
US20200007499A1 (en) Big-data-based business logic learning method and protection method and apparatuses thereof
CN110801630A (en) Cheating program determining method, device, equipment and storage medium
CN111159162A (en) Database configuration method and device
CN111191234A (en) Method and device for detecting virus information

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information
CB02 Change of applicant information

Address after: 310052 188 Lianhui street, Xixing street, Binjiang District, Hangzhou, Zhejiang Province

Applicant after: Dbappsecurity Co.,Ltd.

Address before: Zhejiang Zhongcai Building No. 68 Binjiang District road Hangzhou City, Zhejiang Province, the 310000 and 15 layer

Applicant before: Dbappsecurity Co.,Ltd.

GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20220914

Address after: Room 709, 7th Floor, No. 188, Lianhui Street, Xixing Street, Binjiang District, Hangzhou City, Zhejiang Province 310000

Patentee after: Hangzhou Anheng Vehicle Network Security Technology Co.,Ltd.

Address before: No. 188, Lianhui street, Xixing street, Binjiang District, Hangzhou City, Zhejiang Province

Patentee before: Dbappsecurity Co.,Ltd.

TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20240709

Address after: No. 188, Lianhui street, Xixing street, Binjiang District, Hangzhou, Zhejiang Province, 310000

Patentee after: Dbappsecurity Co.,Ltd.

Country or region after: China

Address before: Room 709, 7th Floor, No. 188, Lianhui Street, Xixing Street, Binjiang District, Hangzhou City, Zhejiang Province 310000

Patentee before: Hangzhou Anheng Vehicle Network Security Technology Co.,Ltd.

Country or region before: China