CN105786927A

CN105786927A - Log processing method and device

Info

Publication number: CN105786927A
Application number: CN201410831840.2A
Authority: CN
Inventors: 张站朝; 何申; 张雯; 俞诗源
Original assignee: China Mobile Communications Group Co Ltd
Current assignee: China Mobile Communications Group Co Ltd
Priority date: 2014-12-26
Filing date: 2014-12-26
Publication date: 2016-07-20
Anticipated expiration: 2034-12-26
Also published as: CN105786927B

Abstract

The invention discloses a log processing method. The method comprises the following steps: obtaining a first log packet; analyzing each log in the first log packet to obtain field information satisfying a preset condition; when N is an integer larger than or equal to 2, obtaining a first log from the first log packet, wherein the first log is a log in the first log packet; judging whether the field information of the first log is the same as the field information of a first sub-log to obtain a first judgment result; when the first judgment result indicates that the field information of the first log is the same as the field information of the first sub-log, obtaining a first log source and a first sub-log source, wherein the first log source is a log source of the first log, and the first sub-log source is the log source of the first sub-log; and determining whether to discard the first log or not according to the priority of the first log source and the priority of the first sub-log source. The invention further discloses a log processing device.

Description

Log processing method and device

Technical Field

The present invention relates to information processing technologies, and in particular, to a log processing method and apparatus.

Background

In order to continuously respond to new security challenges, enterprises deploy firewalls, Unified Threat Management (UTM), intrusion detection and protection systems, vulnerability scanning systems, antivirus systems, terminal management systems and the like in sequence, and a security defense line is constructed. However, these lines of security defense only defend against security threats from a certain aspect, forming individual "islands of security defense" that do not produce synergistic effects. More seriously, a large amount of safety logs and events are continuously generated in the running process of the complex Information Technology (IT) resources and the safety defense facilities thereof, a large amount of information isolated islands are formed, limited safety management personnel face the large amount of safety information which is split mutually, control console interfaces and alarm windows of various products are operated, the restraint is unfair, the working efficiency is extremely low, and the real potential safety hazard is difficult to find.

The centralized security management platform (system) can acquire logs generated by normal service access on each network side device or user side device from a large-scale network, and provides an effective method for analyzing abnormal security events; specifically, the centralized security monitoring platform collects logs with various sources and structures in a unified manner, analyzes, filters, merges and performs correlation analysis on the logs, and forms a security alarm by monitoring and discovering abnormal security events. The multiple logs of varying origin and structure, comprising: mirror Flow logs for switch networks, router Flow (Flow) information logs, firewall logs, intrusion detection logs, intrusion prevention logs, Virtual Private Network (VPN) access logs, host operating system logs, database operation logs, middleware logs, service system logs, and user operation logs.

The centralized security management platform in the prior art has the following disadvantages: 1) redundancy of a plurality of pieces of log information from different levels in different network structures cannot be effectively filtered, so that inconsistency of safety information and low query and retrieval performance are caused; 2) the subjectivity of the artificial configuration of the filtering rules can cause excessive filtering and ineffective filtering of logs, resulting in loss of safety information and large redundancy of safety information.

Disclosure of Invention

In view of this, embodiments of the present invention provide a log processing method and apparatus for solving at least one problem in the prior art, so that a large number of redundant logs can be reduced, and the real effectiveness of filtered log information is improved.

The technical scheme of the embodiment of the invention is realized as follows:

in a first aspect, an embodiment of the present invention provides a log processing method, where the method includes:

acquiring a first log packet, wherein the first log packet comprises N logs, and the N logs are logs from different log sources;

analyzing each log in the first log packet to obtain field information meeting preset conditions;

when N is an integer greater than or equal to 2, acquiring a first log from the first log packet, wherein the first log is one log in the first log packet;

judging whether the field information of the first log is the same as the field information of the first sub-log or not to obtain a first judgment result;

wherein the first sub-log is any one of the first log packets except the first log;

when the first judgment result shows that the field information of the first log is the same as the field information of the first sub-log, acquiring a first log source and a first sub-log source;

and determining whether to discard the first log according to the priority of the first log source and the priority of the first sub-log source.

In a second aspect, an embodiment of the present invention provides a log processing apparatus, where the apparatus includes a first obtaining unit, an analyzing unit, a second obtaining unit, a determining unit, a third obtaining unit, and a determining unit, where:

the first obtaining unit is configured to obtain a first log packet, where the first log packet includes N logs, and the N logs are logs from different log sources;

the analysis unit is used for analyzing each log in the first log packet to obtain field information meeting preset conditions;

the second acquiring unit is used for acquiring a first log from the first log packet when N is an integer greater than or equal to 2, wherein the first log is one log in the first log packet;

the judging unit is used for judging whether the field information of the first log is the same as the field information of the first sub-log or not to obtain a first judgment result;

the third obtaining unit is configured to obtain the first log source and the first sub-log source when the first determination result indicates that the field information of the first log is the same as the field information of the first sub-log;

the determining unit is configured to determine whether to discard the first log according to the priority of the first log source and the priority of the first sub-log source.

According to the log processing method and device provided by the embodiment of the invention, a first log packet is acquired; analyzing each log in the first log packet to obtain field information meeting preset conditions; when N is an integer greater than or equal to 2, acquiring a first log from the first log packet, wherein the first log is one log in the first log packet; when the field information of the first log is determined to be the same as the field information of the first sub-log, acquiring a first log source and a first sub-log source; determining whether to discard the first log according to the priority of the first log source and the priority of the first sub-log source; therefore, a large number of redundant logs can be reduced, and the real effectiveness of the filtered log information is improved.

Drawings

FIG. 1-1 is a schematic diagram of a log at different layers of a network in a switch;

FIG. 1-2 is a schematic diagram of a relationship between a time window and a step size according to an embodiment of the present invention;

FIGS. 1-3 are a first flowchart illustrating a log processing method according to an embodiment of the present invention;

FIGS. 1-4 are schematic diagrams of a flow chart of a log processing method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of an implementation flow of a log processing method according to a second embodiment of the present invention;

FIG. 3 is a schematic diagram of an implementation flow of a third log processing method according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of an implementation flow of a quad log processing method according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of an implementation flow of a five-log processing method according to an embodiment of the present invention;

FIG. 6-1 is a first schematic diagram illustrating a composition structure of a log processing apparatus according to an embodiment of the present invention;

fig. 6-2 is a schematic diagram of a composition structure of a log processing apparatus according to an embodiment of the present invention.

Detailed Description

The log processing method provided by the following embodiments of the present invention may be applied to a centralized security management platform, for example, in a time window of | T2-T1| ═ T0, where T1 is an initial time, T2 is a time for obtaining a log, and when T2 is different from T1, T0 is not 0, and T0 is defined as the size of the time window, when a certain service application system is accessed through a network, a log established by connection may be obtained in a switch or router traffic of a network layer, an access log is generated on a firewall or a VPN, an alarm log is generated for abnormal access traffic, intrusion detection, or intrusion prevention, a web middleware or a database may also generate a log of service operation corresponding to service start stop or abnormal log information on a host, and finally, a service application system may also record a log. Therefore, a normal service application access generates related log information at a network layer, a system layer and an application layer, and the centralized security management platform monitors abnormal security events according to the related logs so as to form security alarms.

The log processing method provided by the embodiment of the invention filters logs with different sources and structures, such as a network mirror flow log of a switch, a flow information log of a router, a firewall log, an intrusion detection log, an intrusion prevention log, a VPN access log, a host operating system log, a database operating log, a middleware log, a service system log, a user operating log and the like, so that a large amount of redundant log information is reduced, and the real effectiveness of the filtered log information is improved.

In the following embodiments of the present invention, a method for determining the priority of a log source based on an advantageous-evidence method is provided, a log with the same field information but different sources in the log is filtered based on the priority of the log source, during the filtering process, the log of the advantageous evidence replaces the log of the relatively non-advantageous evidence, and meanwhile, the confidence of the log of the retained advantageous evidence is also verified. Wherein, the log source can be, for example, a switch, a firewall, a router, a database; as shown in fig. 1-1, the logs with different sources and structures are divided according to a network structure, and include a log of a network layer, a log of a system layer, and a log of an application layer, where the log of the network layer includes: the system comprises a switch network mirror image flow log or a router flow log, a firewall access log and a virtual private network access log, an intrusion detection log, an intrusion defense log and an intrusion alarm log; the logs of the system layer comprise host operating system logs, middleware logs and database operating logs; the logs of the application layer comprise service application system logs, user operation logs, middleware logs and database operation logs.

In the embodiment of the invention, the dominant evidence method is embodied in the priority of the log source, as shown in fig. 1-1, the evidentity of the service application system log is greater than that of the middleware and the database log; the evidentiality of the middleware and the database logs is greater than that of the host system logs; the evidentity of the system log of the host computer is greater than that of the system log of the intrusion detection and intrusion prevention; the evidentiality of the intrusion detection and intrusion prevention system logs is greater than that of the firewall and the VPN logs; the evidential property of the firewall and VPN access logs is larger than that of the flow logs of the switch and the router. Therefore, the advantageous evidence law provided by the embodiment of the invention is arranged according to the application layer, the system layer and the network layer of the log source.

It should be noted that the following embodiments of the present invention provide an advantageous evidence method in terms of the priority of the log source, and there is another priority in the prior art, and the priority in the prior art is mainly reflected in the priority of the problem described in the log, for example, in the same log or several logs, the log or several logs may describe two events, such as a first event and a second event, wherein the first event is more serious, and the severity of the second event is far less than that of the first event compared with the first event, so that the priority of the first event described in the log is higher than that of the second event described in the log.

The technical solution of the present invention is further elaborated below with reference to the drawings and the specific embodiments.

Example one

The embodiment of the invention provides a log processing method, which is applied to a centralized security management platform, wherein the centralized security management platform can be a common computer or a server or equipment such as an industrial control computer (industrial personal computer) in the specific implementation process, the functions implemented by the log processing method can be realized by calling a program code through a processor in the centralized security management platform, and the program code can be saved in a computer storage medium; it can be seen that the centralized security management platform comprises at least a processor and a storage medium.

In the process of processing the log, the log in a time window, for example, a time window of | T2-T1| ═ T0, as described above, T1 is the initial time, T2 is the time of obtaining the log, when T2 is different from T1, T0 is not 0, and T0 is defined as the time window; in the specific implementation process, the time window may be specified to be 10 seconds, 8 seconds, 5 seconds, 20 seconds, 60 seconds, 120 seconds, and the like, and when the time window is specified, there is a premise that all the network side devices and the user side device need to use a uniform time, for example, a Network Time Protocol (NTP) server may be used to provide the uniform time. The logs in a time window can be logs obtained from all log sources within the time window, and can also be logs obtained from partial log sources within the time window. For example, the centralized security management platform has 10 log sources, and the log obtained in one time window may be the log obtained in the time window from the 10 log sources, or the log obtained in the time window from the 1 st log source to the 4 th log source. It should be noted that, when the time window is used to obtain the log in the log source, another important parameter is the step size, and the time window is moved according to the step size. In general, the step size has the same units as the time window, and as the time window is calculated in seconds as described above, the step size can also be calculated in seconds, and the step size is generally smaller than or equal to the size of the time window. Assuming a time window of 20 seconds and a step size of 10 seconds, the time window is shifted as shown in fig. 1-2. In addition, there may be multiple obtaining manners for obtaining the logs in the log source, and the obtaining manner is not limited to the above description manner of the time window, and a person skilled in the art may obtain the logs in the log source by using various existing techniques according to specific situations, and therefore details are not described again.

In the embodiments shown in fig. 1-3, the example of processing a log within a time window is described. Fig. 1-3 are schematic diagrams illustrating a flow chart of a log processing method according to an embodiment of the present invention, as shown in fig. 1-3, the method includes:

step 101, acquiring a first log packet;

here, the first log packet includes N logs, which are logs originated from different log sources; the first log packet may be a log obtained in a time window manner, that is, the first log packet may be a log of a time window; of course, the first log packet may also be a log obtained in other manners.

Here, the acquiring the first log packet includes:

acquiring a first log packet from a source log library, wherein the source log library comprises logs from different log sources; or,

and directly acquiring N logs from different log sources, and forming a first log packet by the N logs.

102, analyzing each log in the first log packet to obtain field information meeting preset conditions;

here, the preset condition is field information of at least any one of an event number, a name, a log reception time, a log generation time, a type, a level, a source host name, a source asset ID, a source user name, a source process name, a source IP address, a source port, an operation or action behavior name, a destination host name, a destination asset ID, a destination user name, a destination process name, a destination IP address, a destination port, a protocol, an original log number, a log source IP, an acquisition engine IP, a destination data object, a destination object level, a trusted object, an object state, a special flag, and the like.

Step 103, when N is an integer greater than or equal to 2, acquiring a first log from the first log packet;

here, the first log is one log in the first log packet; when the first log is acquired in a time window manner as shown in fig. 1-2, the first log may refer to a most recent log in the first log packet, and the first sub-log in the opposite embodiment may be a log temporally adjacent to the first log.

104, judging whether the field information of the first log is the same as the field information of the first sub-log to obtain a first judgment result;

wherein the first sub-log is any one of the first log packets except the first log; it should be noted here that the field information of the first log only needs to be the same as the field information of the same field in the first sub-log, and all the field information does not need to be completely the same. This is because, in the process of implementing the method specifically, the number of fields and the names of fields after log parsing from the log sources of different advantageous evidence levels are not completely consistent, so that when determining whether two logs represent the same security event information, in the embodiment of the present invention, it is only necessary to determine that the values of the same fields are the same, which can provide the accuracy of filtering logs. It should be further noted that the number of log fields of adjacent dominant evidence hierarchies is the most in fig. 1-1.

Step 105, when the first judgment result shows that the field information of the first log is the same as the field information of the first sub-log, acquiring a first log source and a first sub-log source;

here, the first log source is a log source of a first log, and the first sub-log source is a log source of the first sub-log;

and 106, determining whether to discard the first log according to the priority of the first log source and the priority of the first sub-log source.

In the embodiments shown in fig. 1-3, the example of processing a log within a time window is described. In the embodiment shown in fig. 1-4, another log in the first log packet is processed based on the processing manner of processing one log as described in the embodiment shown in fig. 1-3. As shown in fig. 1-4 in detail, after step 104, the method further includes:

and step 107, when the first judgment result shows that the field information of the first log is different from the field information of the first sub-log, outputting the first log so as to form a safety alarm.

Step 108, deleting the first log from the first log packet to obtain a second log packet;

here, the second log packet includes N-1 logs.

Step 109, when N-1 is greater than 2, obtaining a second log from the second log packet;

here, the second log is one log in the second log packet;

step 110, judging whether the field information of the second log is the same as the field information of the second sub-log, and obtaining a first judgment result;

the second sub-log is any one of the second log packets except the second log;

step 111, when the first judgment result shows that the field information of the second log is the same as the field information of the second sub-log, acquiring a second log source and a second sub-log source;

here, the second log source is a log source of a second log, and the second sub-log source is a log source of the second sub-log;

and step 112, determining whether to reserve the second log according to the priority of the second log source and the priority of the second sub-log source.

Here, the first log may still be output on the centralized security management platform, in other words, the centralized security management platform may form a security alarm according to the first log; the first log is of course output to another computer, i.e. the first log may not be output on the centralized security management platform, i.e. the security alarm may be implemented on another computer.

In the embodiment of the invention, a first log packet is obtained; analyzing each log in the first log packet to obtain field information meeting preset conditions; when N is an integer greater than or equal to 2, acquiring a first log from the first log packet, wherein the first log is one log in the first log packet; when the field information of the first log is determined to be the same as the field information of the first sub-log, acquiring a first log source and a first sub-log source; and determining whether to discard the first log according to the priority of the first log source and the priority of the first sub-log source, so that a large number of redundant logs can be reduced, and the real effectiveness of the filtered log information is improved.

Example two

Based on the first embodiment, an embodiment of the present invention provides a log processing method, which is applied to a centralized security management platform, where the centralized security management platform may be a device such as a common computer, a server, or an industrial control computer (industrial personal computer) in a specific implementation process, and functions implemented by the log processing method may be implemented by invoking program codes by a processor in the centralized security management platform, and certainly, the program codes may be stored in a computer storage medium; it can be seen that the centralized security management platform comprises at least a processor and a storage medium.

Fig. 2 is a schematic flow chart of an implementation of a second log processing method according to an embodiment of the present invention, and as shown in fig. 2, the method includes:

step 201, acquiring a first log packet;

here, the first log packet includes N logs, which are logs originated from different log sources;

step 202, analyzing each log in the first log packet to obtain field information meeting preset conditions;

Step 203, when N is an integer greater than or equal to 2, acquiring a first log from the first log packet;

here, the first log is one log in the first log packet;

step 204, judging whether the field information of the first log is the same as the field information of the first sub-log, and obtaining a first judgment result;

step 205, when the first determination result indicates that the field information of the first log is the same as the field information of the first sub-log, acquiring a first log source and a first sub-log source;

step 206, judging whether the priority of the first log source is higher than that of the first sub-log source to obtain a second judgment result;

and step 207, when the second judgment result shows that the priority of the first log source is higher than or equal to the priority of the first sub-log source, outputting the first log.

Here, the first log is output in order to form a security alarm according to the first log.

And step 208, when the second judgment result shows that the priority of the first log source is smaller than that of the first sub-log source, discarding the first log and outputting the first sub-log.

Here, the first sub-log is output in order to form a security alarm from the first sub-log.

In the embodiment of the present invention, the method further includes: and when the first judgment result shows that the field information of the first log is different from the field information of the first sub-log, outputting the first log so as to form a safety alarm.

In the embodiment of the present invention, the first log and/or the first sub-log may still be output on the centralized security management platform, in other words, the centralized security management platform may form a security alarm according to the first sub-log and form a security alarm according to the first log. Of course, the first log and/or the first sub-log are output to another computer, i.e. the first log and/or the first sub-log may not be on a centralized security management platform, i.e. the security alarm may be implemented on another computer.

In the embodiment of the invention, a first log packet is obtained; analyzing each log in the first log packet to obtain field information meeting preset conditions; when N is an integer greater than or equal to 2, acquiring a first log from the first log packet, wherein the first log is one log in the first log packet; when the field information of the first log is determined to be the same as the field information of the first sub-log, acquiring a first log source and a first sub-log source; according to the priority of the first log source and the priority of the first sub-log source, the first log is output and/or the first sub-log is output, and therefore the technical scheme provided by the embodiment of the invention has the following advantages:

1) by filtering the logs based on the dominant evidence method, the security analysis amount of the logs of network layer equipment such as switches, firewalls, intrusion detection equipment and intrusion prevention equipment with large log amount can be reduced, and the security event analysis performance is improved;

2) the advantage evidence method provided by the invention can reduce a large number of redundant logs and improve the real effectiveness of the filtered log information.

EXAMPLE III

In the embodiments shown in fig. 1-3, the example of processing a log within a time window is described. In the embodiment shown in fig. 3, based on the processing manner of processing one log described in the foregoing embodiment shown in fig. 3, another log in the first log packet is processed, and according to the method shown in fig. 3, other logs in the first log packet may be processed. Fig. 3 is a schematic flow chart of an implementation of a three-log processing method according to an embodiment of the present invention, and as shown in fig. 3, the method includes:

step 301, acquiring a first log packet;

step 302, analyzing each log in the first log packet to obtain field information meeting preset conditions;

Step 303, when N is an integer greater than or equal to 2, obtaining a first log from the first log packet;

here, the first log is one log in the first log packet;

step 304, judging whether the field information of the first log is the same as the field information of the first sub-log, if so, entering step 305; if not, go to step 307;

step 305, acquiring a first log source and a first sub-log source;

step 306, judging whether the priority of the first log source is higher than or equal to the priority of the first sub-log source, if yes, entering step 307; if not, go to step 308;

in step 307, the first log is output, and the process proceeds to step 309.

Step 308, discarding the first journal and outputting the first sub-journal, and proceeding to step 310.

Here, steps 301 to 308 correspond to steps 201 to 208 in the first embodiment, respectively, and thus are not described again.

Step 309, deleting the first log from the first log packet to obtain a second log packet, where the second log packet includes N-1 logs, and when N-1 is greater than or equal to 2, entering step 311.

Step 310, deleting the first log and the first sub-log from the first log packet to obtain a second log packet, and entering step 311 when N-2 is greater than or equal to 2;

here, the second log packet includes N-2 logs.

Step 311, obtaining a second log from the second log packet, where the second log is one of the second log packets;

step 312, determining whether the field information of the second log is the same as the field information of the second sub-log, if yes, entering step 313; if not, go to step 315;

the second sub-log is any one of the second log packets except the second log;

step 313, acquiring a second log source and a second sub-log source;

step 314, determining whether the priority of the second log source is higher than or equal to the priority of the second sub-log source, if yes, entering step 315; if not, go to step 316;

step 315, outputting the second log.

Here, the second log is output in order to form a security alarm from the second log.

Step 316, discarding the second journal and outputting the second sub-journal.

Here, the second sub-log is output in order to form a security alarm from the second sub-log.

In the embodiment of fig. 3 of the present invention, a processing manner of consecutively processing two logs in a first log packet is shown, and it should be understood by those skilled in the art that if one log is detected, the log should be deleted from the log packet, and if the number of remaining log pieces of the log packet is greater than or equal to 2, the next log in the log packet may be processed according to the method described in fig. 3. The process described herein in which a log is checked is the process shown in steps 304 through 308.

Example four

In the embodiments shown in fig. 1-3, the example of processing a log within a time window is described. In the embodiment shown in fig. 4, the embodiment shown in fig. 4 describes how to process the logs in the next log packet while processing the remaining one log, based on the processing manner of processing one log described in the foregoing embodiments shown in fig. 1 to 3. Fig. 4 is a schematic flow chart of an implementation of a four-log processing method according to an embodiment of the present invention, and as shown in fig. 4, the method includes:

step 301, acquiring a first log packet;

here, the first log is one log in the first log packet;

step 305, acquiring a first log source and a first sub-log source;

in step 307, the first log is output, and the process proceeds to step 401.

Step 308, discarding the first journal and outputting the first sub-journal, and proceeding to step 402.

Step 401, deleting the first log from the first log packet to obtain a second log packet, where the second log packet includes N-1 logs, and when N-1 is greater than or equal to 1, entering step 403.

Step 402, deleting the first log and the first sub-log from the first log packet to obtain a second log packet, and entering step 403 when N-2 is greater than or equal to 1;

step 403, acquiring a first sub-log packet;

here, the first sub-log packet includes M logs, where M is an integer greater than or equal to 1, and the M logs are logs from different log sources; the first sub-log packet may be considered a log obtained within a next time window; for example, when a first log packet is a log obtained within a first time window, then the first sub-log packet may be considered a log obtained within a second time window, with the first time window being adjacent in time to the second time window, and the second time window occurring after the first time window.

Step 404, obtaining a second log from the second log packet;

here, the second log is the only log in the second log packet; there is no strict precedence between step 403 and step 404.

Step 405, judging whether the field information of the second log is the same as the field information of the second sub-log, if so, entering step 406; if not, go to step 407;

wherein the second sub-log is any one log in the first sub-log packet;

step 406, acquiring a second log source and a second sub-log source;

step 407, determining whether to keep the second log according to the priority of the second log source and the priority of the second sub-log source.

Step 408, outputting the first log to form a security alarm.

EXAMPLE five

In the embodiments shown in fig. 1-3, the example of processing a log within a time window is described. Based on the processing manner of processing one log described in the foregoing embodiments shown in fig. 1 to 3, when only one log is included in a first log packet, the embodiment shown in fig. 5 describes how to process the logs in the next log packet while processing only one log. Fig. 5 is a schematic flow chart of an implementation of a five-log processing method according to an embodiment of the present invention, and as shown in fig. 5, the method includes:

step 301, acquiring a first log packet;

here, the first log packet includes N logs, and the nth log is a log derived from a different log source;

here, the first log is one log in the first log packet;

step 305, acquiring a first log source and a first sub-log source;

step 307, outputting the first log.

Step 308, discarding the first journal and outputting the first sub-journal.

Step 501, when N is equal to 1, acquiring a first sub-log packet;

here, the first sub-log packet includes M logs, where M is an integer greater than or equal to 1, and the M logs are logs from different log sources;

step 502, obtaining a first log from the first log packet;

here, the first log is a unique log in the first log packet;

step 503, judging whether the field information of the first log is the same as the field information of the first sub-log, if yes, entering step 504; if not, go to step 506;

wherein the first sub-log is any one log in the first sub-log packet;

step 504, acquiring a first log source and a first sub-log source;

step 505, judging whether the priority of the first log source is higher than or equal to the priority of the first sub-log source, if yes, entering step 506; if not, go to step 507;

step 506, outputting the first log.

Step 507, discarding the first journal and outputting the first sub-journal.

In the first to fifth embodiments of the present invention, the analyzing each log in the first log packet to obtain field information meeting a preset condition includes: performing regular expression analysis on the logs in the first log packet to acquire field information meeting preset conditions;

here, the preset condition is field information of at least any one of generation time, event number, level, source IP address, source port, destination IP address, destination port, protocol, action, or operation.

It should be noted that, the method in the first to fifth embodiments may further include: when the field value after parsing has a mappable extension, extension mapping is also needed, for example, the first IP address is mapped to a second IP address through Network Address Translation (NAT), where the second IP address may include one IP address or two or more IP addresses; another example is that 80 ports are mapped to 808 ports by Port Address Translation (PAT).

In the above-mentioned embodiments one to five of the present invention, in the steps (such as step 104, step 204, step 304, step 312, etc.) related to comparing whether the field information of the two logs is the same, there may be a plurality of comparison manners. Taking step 104 as an example, it is determined whether the field information of the first log is the same as the field information of the first sub-log, and it may be determined whether at least any of the following field information in the two logs is the same.

Source Internet Protocol (IP) address, destination IP address, destination port, source port, protocol, operation.

EXAMPLE six

Based on the log processing method provided by the above embodiment, an embodiment of the present invention provides a log processing apparatus, which is applied to a centralized security management platform, where the centralized security management platform may be a common computer or server or an industrial control computer (industrial personal computer) or other device in a specific implementation process, and functions implemented by the log processing apparatus may be implemented by invoking program codes by a processor in the centralized security management platform, and certainly, the program codes may be stored in a computer storage medium; it can be seen that the centralized security management platform comprises at least a processor and a storage medium.

In the process of processing the log, the log in a time window, for example, a time window of | T2-T1| ═ T0, as described above, T1 is the initial time, T2 is the time of obtaining the log, when T2 is different from T1, T0 is not 0, and T0 is defined as the time window; in the specific implementation process, the time window may be specified as 10 seconds, 8 seconds, 5 seconds, 20 seconds, 60 seconds, 120 seconds, and the like, and when the time window is specified, there is a premise that all the network side devices and the user side devices need to use a uniform time, for example, a network time protocol server may be used to provide the uniform time. The logs in a time window can be obtained from all log sources within the time window, or can be obtained from partial time sources within the time window. For example, the centralized security management platform has 10 log sources, and the log obtained in one time window may be the log obtained in the time window from the 10 log sources, or the log obtained in the time window from the 1 st log source to the 4 th log source. It should be noted that when the time window is used to obtain the logs in the log source, another parameter is more important, namely the step size, and the time window is moved according to the step size. In general, the step size has the same units as the time window, and as the time window is calculated in seconds as described above, the step size can also be calculated in seconds, and the step size is generally smaller than or equal to the size of the time window. Assuming a time window of 20 seconds and a step size of 10 seconds, the time window is shifted as shown in fig. 1-2. In addition, there may be multiple obtaining manners for obtaining the logs in the log source, and the obtaining manner is not limited to the above description manner of the time window, and a person skilled in the art may obtain the logs in the log source by using various existing techniques according to specific situations, and therefore details are not described again.

Fig. 6-1 is a schematic diagram of a composition structure of a log processing apparatus according to an embodiment of the present invention, and as shown in fig. 6-1, the log processing apparatus 600 includes a first obtaining unit 601, an analyzing unit 602, a second obtaining unit 603, a determining unit 604, a third obtaining unit 605, and a determining unit 606, where:

the first obtaining unit 601 is configured to obtain a first log packet, where the first log packet includes N logs, and the nth log is a log derived from a different log source;

the analyzing unit 602 is configured to analyze each log in the first log packet to obtain field information meeting a preset condition;

the second obtaining unit 603 is configured to obtain a first log, which is one log in the first log packet, from the first log packet when N is an integer equal to or greater than 2;

the determining unit 604 is configured to determine whether the field information of the first log is the same as the field information of the first sub-log, so as to obtain a first determination result;

the third obtaining unit 605 is configured to obtain a first log source and a first sub-log source when the first determination result indicates that the field information of the first log is the same as the field information of the first sub-log, where the first log source is a log source of the first log, and the first sub-log source is a log source of the first sub-log;

the determining unit 606 is configured to determine whether to discard the first log according to the priority of the first log source and the priority of the first sub-log source.

In this embodiment of the present invention, as shown in fig. 6-2, the determining unit 606 includes a determining module 6061, an outputting module 6062, and a discarding module 6063, where:

the determining module 6061 is configured to determine whether the priority of the first log source is higher than the priority of the first sub-log source, so as to obtain a second determination result;

the output module 6062 is configured to output the first log source to form a security alarm when the second determination result indicates that the priority level of the first log source is higher than or equal to the priority level of the first sub-log source.

The discarding module 6063 is configured to discard the first log when the second determination result indicates that the priority of the first log source is smaller than the priority of the first sub-log source.

In this embodiment of the present invention, the output module 6062 is further configured to output the first sub-log, so as to form a security alarm.

In this embodiment of the present invention, the apparatus further includes an output unit, configured to output the first log when the first determination result indicates that the field information of the first log is different from the field information of the first sub-log, so as to form a security alarm.

In this embodiment of the present invention, the apparatus further includes a deleting unit, configured to delete the first log from the first log packet, so as to obtain a second log packet, where the second log packet includes N-1 logs.

In this embodiment of the present invention, the deleting unit is further configured to delete the first log and the first sub-log from the first log packet, so as to obtain a second log packet, where the second log packet includes N-2 logs.

In this embodiment of the present invention, the second obtaining unit is further configured to obtain a second log from the second log packet when N-2 is not equal to 1 and not equal to 0, where the second log is one of the second log packets;

correspondingly, the judging unit is further configured to judge whether the field information of the second log is the same as the field information of the second sub-log, so as to obtain a first judgment result;

the second sub-log is any one of the second log packets except the second log;

correspondingly, the third obtaining unit is further configured to obtain a second log source and a second sub-log source when the first determination result indicates that the field information of the second log is the same as the field information of the second sub-log, where the second log source is the log source of the second log, and the second sub-log source is the log source of the second sub-log;

correspondingly, the determining unit is further configured to determine whether to retain the second log according to the priority of the second log source and the priority of the second sub-log source.

In this embodiment of the present invention, the apparatus further includes a fourth obtaining unit, configured to obtain, when N-1 is equal to 1, a first sub-log packet, where the first sub-log packet includes M logs, where M is an integer greater than or equal to 1, and the M logs are logs from different log sources;

correspondingly, the second obtaining unit is further configured to obtain a second log from the second log packet, where the second log is the only log in the second log packet;

wherein the second sub-log is any one log in the first sub-log packet;

correspondingly, the third obtaining unit is configured to obtain a second log source and a second sub-log source when the first determination result indicates that the field information of the second log is the same as the field information of the second sub-log, where the second log source is the log source of the second log, and the second sub-log source is the log source of the second sub-log;

In this embodiment of the present invention, the apparatus further includes a fourth obtaining unit, configured to obtain, when N is equal to 1, a first sub-log packet, where the first sub-log packet includes M logs, where M is an integer greater than or equal to 1, and the M logs are logs from different log sources;

correspondingly, the second obtaining unit is further configured to obtain a first log from the first log packet, where the first log is a unique log in the first log packet;

correspondingly, the judging unit is further configured to judge whether the field information of the first log is the same as the field information of the first sub-log, so as to obtain a first judgment result;

wherein the first sub-log is any one log in the first sub-log packet;

correspondingly, the third obtaining unit is configured to obtain a first log source and a first sub-log source when the first determination result indicates that the field information of the first log is the same as the field information of the first sub-log, where the first log source is the log source of the first log, and the first sub-log source is the log source of the first sub-log;

correspondingly, the determining unit is further configured to determine whether to retain the first log according to the priority of the first log source and the priority of the first sub-log source.

In an embodiment of the present invention, the first obtaining unit is configured to obtain a first log packet from a source log library, where the source log library includes logs from different log sources; or,

the first obtaining unit is configured to directly obtain N logs from the different log sources, and form the N logs into a first log packet.

In this embodiment of the present invention, the preset condition is field information of at least any one of an event number, a name, log receiving time, log generating time, a type, a level, a source host name, a source asset ID, a source user name, a source process name, a source IP address, a source port, an operation or action name, a destination host name, a destination asset ID, a destination user name, a destination process name, a destination IP address, a destination port, a protocol, an original log number, a log source IP, an acquisition engine IP, a target data object, a target object level, a trusted object, an object state, a special mark, and the like.

Here, it should be noted that: the description of the log processing device item is similar to the description of the log processing method, and is not repeated as long as the beneficial effect description of the log processing method is described. For technical details that are not disclosed in the embodiment of the log processing apparatus of the present invention, please refer to the description of the embodiment of the log processing method of the present invention.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described device embodiments are merely illustrative, for example, the division of the unit is only a logical functional division, and there may be other division ways in actual implementation, such as: multiple units or components may be combined, or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the coupling, direct coupling or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between the units may be electrical, mechanical or other forms.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units; can be located in one place or distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, all the functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately regarded as one unit, or two or more units may be integrated into one unit; the integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.

Those of ordinary skill in the art will understand that: all or part of the steps for realizing the method embodiments can be completed by hardware related to program instructions, the program can be stored in a computer readable storage medium, and the program executes the steps comprising the method embodiments when executed; and the aforementioned storage medium includes: various media that can store program codes, such as a removable memory device, a Read Only Memory (ROM), a magnetic disk, or an optical disk.

Alternatively, the integrated unit of the present invention may be stored in a computer-readable storage medium if it is implemented in the form of a software functional module and sold or used as a separate product. Based on such understanding, the technical solutions of the embodiments of the present invention may be essentially implemented or a part contributing to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the methods described in the embodiments of the present invention. And the aforementioned storage medium includes: a removable storage device, a ROM, a magnetic or optical disk, or other various media that can store program code.

The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.

Claims

1. A method of log processing, the method comprising:

when N is an integer greater than or equal to 2, acquiring a first log from the first log packet;

judging whether the field information of the first log is the same as the field information of the first sub-log or not to obtain a first judgment result; wherein the first sub-log is any one of the first log packets except the first log;

2. The method of claim 1, wherein determining whether to discard the first journal based on the priority of the first journal source and the priority of the first sub-journal source comprises:

judging whether the priority of the first log source is higher than that of the first sub-log source or not to obtain a second judgment result;

and when the second judgment result shows that the priority level of the first log source is higher than or equal to that of the first sub-log source, outputting the first log so as to form a safety alarm.

3. The method of claim 1, wherein determining whether to discard the first log according to the priority of the first log source and the priority of the first sub-log source further comprises:

and when the second judgment result shows that the priority of the first log source is less than that of the first sub-log source, discarding the first log.

4. The method of claim 3, further comprising: outputting the first sub-log to form a security alarm.

5. The method according to any one of claims 1 to 4, further comprising: and when the first judgment result shows that the field information of the first log is different from the field information of the first sub-log, outputting the first log so as to form a safety alarm.

6. The method of claim 5, further comprising: and deleting the first log from the first log packet to obtain a second log packet, wherein the second log packet comprises N-1 logs.

7. The method according to any one of claims 2 to 4, further comprising: and deleting the first log and the first sub-log from the first log packet to obtain a second log packet, wherein the second log packet comprises N-2 logs.

8. The method of claim 7, further comprising: when N-2 is not equal to 1 and not equal to 0, acquiring a second log from the second log packet, wherein the second log is one log in the second log packet;

judging whether the field information of the second log is the same as the field information of the second sub-log or not to obtain a first judgment result;

the second sub-log is any one of the second log packets except the second log;

when the first judgment result shows that the field information of the second log is the same as the field information of the second sub-log, acquiring a second log source and a second sub-log source, wherein the second log source is the log source of the second log, and the second sub-log source is the log source of the second sub-log;

and determining whether to reserve the second log according to the priority of the second log source and the priority of the second sub-log source.

9. The method of claim 7, further comprising:

when N-1 is equal to 1, acquiring a first sub-log packet, wherein the first sub-log packet comprises M logs, M is an integer greater than or equal to 1, and the M logs are logs from different log sources;

acquiring a second log from the second log packet, wherein the second log is the only log in the second log packet;

wherein the second sub-log is any one log in the first sub-log packet;

10. The method of any one of claims 1 to 4, or 6, or 8, or 9, further comprising: when N is equal to 1, acquiring a first sub-log packet, wherein the first sub-log packet comprises M logs, M is an integer greater than or equal to 1, and the M logs are logs from different log sources;

acquiring a first log from the first log packet, wherein the first log is the only log in the first log packet;

wherein the first sub-log is any one log in the first sub-log packet;

when the first judgment result shows that the field information of the first log is the same as the field information of the first sub-log, acquiring a first log source and a first sub-log source, wherein the first log source is the log source of the first log, and the first sub-log source is the log source of the first sub-log;

and determining whether to reserve the first log according to the priority of the first log source and the priority of the first sub-log source.

11. The method of any one of claims 1 to 4, or 6, or 8, or 9, wherein said obtaining a first log package comprises:

12. The method according to any one of claims 1 to 4, 6, 8 or 9, wherein the preset condition is field information of at least any one of event number, name, log receiving time, log generating time, type, level, source host name, source asset identification ID, source user name, source process name, source internet protocol IP address, source port, operation or action behavior name, destination host name, destination asset ID, destination user name, destination process name, destination IP address, destination port, protocol, original log number, log source IP, collection engine IP, destination data object, destination object level, trusted object, object status, and special mark.

13. A log processing device is characterized by comprising a first acquisition unit, an analysis unit, a second acquisition unit, a judgment unit, a third acquisition unit and a determination unit, wherein:

the second acquiring unit is used for acquiring a first log from the first log packet when N is an integer greater than or equal to 2;

14. The apparatus of claim 13, wherein the determining unit comprises a determining module and an output module, wherein:

the judging module is configured to judge whether the priority of the first log source is higher than the priority of the first sub-log source, and obtain a second judgment result;

and the output module is used for outputting the first log to form a safety alarm when the second judgment result shows that the priority of the first log source is higher than or equal to that of the first sub-log source.

15. The apparatus of claim 14, wherein the determining unit further comprises a discarding module configured to discard the first log when the second determination result indicates that the priority of the first log source is less than the priority of the first sub-log source.

16. The apparatus of claim 15, wherein the output module is further configured to output the first sub-log to form a security alarm.

17. The apparatus according to any one of claims 13 to 16, further comprising an output unit configured to output the first log to form a security alarm when the first determination result indicates that the field information of the first log is not the same as the field information of the first sub-log.

18. The apparatus according to claim 17, further comprising a deleting unit configured to delete the first log from the first log packet, resulting in a second log packet, where the second log packet includes N-1 logs.

19. The apparatus according to any one of claims 13 to 16, wherein the deleting unit is further configured to delete the first log and the first sub-log from the first log packet, so as to obtain a second log packet, where the second log packet includes N-2 logs.

20. The apparatus according to claim 19, wherein the second obtaining unit is further configured to obtain a second log from the second log packet when N-2 is not equal to 1 and not equal to 0, where the second log is one of the second log packets;

the second sub-log is any one of the second log packets except the second log;

21. The apparatus according to claim 19, wherein the apparatus further comprises a fourth obtaining unit, configured to, when N-1 is equal to 1, obtain a first sub-log packet, where the first sub-log packet includes M logs, where M is an integer greater than or equal to 1, and the M logs are logs from different log sources;

wherein the second sub-log is any one log in the first sub-log packet;

22. The apparatus according to any one of claims 13 to 16, 18, 20 or 21, wherein the apparatus further comprises a fourth obtaining unit, configured to, when N is equal to 1, obtain a first sub-log packet, where the first sub-log packet includes M logs, where M is an integer greater than or equal to 1, and the M logs are logs from different log sources;

wherein the first sub-log is any one log in the first sub-log packet;

23. The apparatus according to any one of claims 13 to 16, 18, 20 or 21, wherein the first obtaining unit is configured to obtain a first log packet from a source log library, where the source log library includes logs derived from different log sources; or,